I would remove as much as you can without having to take out the landscape and put in fine grained sand instead. A crushed coral "bed" is only a problem, or has gained the reputation of being a problem because its large grains allows food to settle down between the grains and rot away into dissolved nutrients, the reputation comes from the habitat such large grains provide not being stocked with the life forms that use such habitats (amphipods, copepods and worms) and thus, the food that settles into it. Thats why I don't have a problem with you leaving some of the CC behind. I also purposely built an area within my aquarium to provide such a habitat, coral rubble and "CC" sitting on top of a few inches of live sand.
Charles