Yeah I would feel super awkward if that happend to me. You would think there would be more woman then men (I'am not saying this in a sexist way) which is why I always order online now same with Gilly hicks ,even if they hadn't closed down . So I would advise M &S for bras and bra fitting the staff are lovely and you get a lot of good advise from the staff (that are woman ) so I would advise shop there if you want to go in a store :)
I am surprised on one level to hear that men worked there because it's obviously kind of awkward for your female customers when a man makes suggestions about underwear and lingerie, finds out what size you need to help you select items or is possibly (and very likely) picturing his customers in the items they are looking at. On the other hand it's not surprising because no company can discriminate in the hiring process based on things like gender. A company like Victoria's Secret could get themselves into a lot of legal trouble if a man decided to take them to court after a perceived "rejection" for employment based on his gender.
Legally, they can't refuse to hire men. That's gender discrimination.