judging that sps peice has come from your, as you said, so/so fish store, I would guess the BN is brown from nutrient issues. A little nitrate/phosphate can ruin the color in your sps polyps. i would guess thats why theirs are brown. Yours took a trip and is very small. probly lost some color during the trip.
Give yours some time. I would figure, in your tank, with your tank maintanance, yours will have nice green tips on the polyps soon.
I've never had my birdsnest keep its polyps retracted for more than a few minutes that I've noticed, and always from something physically disturbing it.
I use an ATO so I never get a very large top off at once. How much did you need to top off? How often do you top off? I'd guess that if it was a sudden and large enough change in sg, that could do it.