Yes, yes it is. Same as if someone referred to all women as a class. It's not fair to generalize about all women. It's not fair to generalize about all men. If you refer to men as a class, you're inherently being just as unfair as if you do the same to women.
You do realize that when people talk about men as a class, that isn't the same thing as a personal accusation, right?