Things like that have a tendency to become effectively mandatory, even if they never officially are. If you're always the odd one out on these events, how likely do you think it is that your ideas and proposals will be adopted, you'll be picked first for career advancement opportunities or important assignments, people will respond quickly and helpfully to your questions, etc?

In a perfect world, none of that would be affected by your social habits, but our world is an imperfect place.