I feel like you're being willfully obtuse here. The penis is commonly used for exactly two things. Sex is the only reason why anyone would consider it advantageous to have a large penis.

And not men?

Sure, men too. Let me repeat that: you are "allowed" to be offended about this if you are a man. People tend to say "creates a hostile environment for women" not because of assumption or stereotype but because of many personal accounts of specific women who say that this sort of joking makes them feel like they're in a hostile environment.