I totally agree with removing problematic species from a reef tank. However, from your description it doesn't sound like there's any problem.

I think getting some ID on these sponges is the next logical step. "Invertebrates: A quick reference guide: by Sprung has a whole chapter on sponge ID. I flipped through it and the only sponges listed in the book as potentially "harmful to corals" are Chrondrilla aka chicken liver sponge (brown, black, gray or dull orange) and Collospongia aka ear sponge (blue, gray or purple). Based on your description I'd venture to guess you have Clathrina aka leather latticework sponge... a harmless hitchhiker. Try a Google image search to see if any of these genus match.

HTH,
Tom