I agree with the downvoted commenter, this isn’t much different that those who make fun of people for baggy clothes. These guys obviously have a curated fashion sense that does not align with your own. There is nothing wrong with what they are wearing. Women wear tight clothing all the time but you don’t see “lighthearted jokes” about their clothes being too tight because of the dryer. The implication is that these clothes are too tight for men to wear, which is judgey and weird, and ties into sexist and homophobic ideas about what men should wear.

“It’s just a joke” is the worst way to respond to this kind of criticism. This joke was funny because it drew upon your pre-existing ideas about how men should dress. And you won’t even bother to imagine that the basis for your joke might be even a tiny bit harmful.