Are they irked because the club exists, or because they weren't invited?
"Neighbors say an underground swingers club is bringing down the neighborhood"
The trouble with swingers, online casual encounters and such is the choice of party partner, more than likely, isn't the stuff of fantasies. Often, it’s the middle-age bored couples and tattooed young boys with little vocabulary skills, and fewer teefs, partaking. I’ve seen enough amateur swinger porn to define it as such. Trust.
Same for nude beaches and nudist resorts (the term "colonies", apparently, is no longer in fashion). Gawking is discouraged and given the abundance of an older clientele and naturally occurring drooping and flopping (do they let the older men do calisthenics?) I wouldn’t need reminded twice. However, given my…uhh…well-documented-in-this-blog interest in the male of the species and appreciation of…uhh…particular regions, I imagine I would stare inappropriately and if in a hypnotic trance. Like cartoons or episodes of “Ab Fab”; I could watch all day.
Same with lesbians. Most men envision women-on-women encounters as moaning, sexy, hot and steamy, perhaps including stilettos. Bet you don't picture Rosie frenching her girl.