Managing fine is different than it being insane to go into an office. And I don't know on what basis we can conclude that it wouldn't be better if core Linux devs were working together.

I don't know much about the nitty gritty of what kernel deving looks like, but it's possible that the work there looks more like longer stretches of heads down time than average. E.g., I can imagine you could be pretty impactful being heads down for weeks or months at a time micro-optimizing the performance of various components. I think this tends not to be true of most software.

One thing I do know about the social dynamics of Linux development is that Linus can be an asshole and flame at people a lot. You could worry that this would be more intense and thus worse in an office, but IME this kind of stuff tends to get better once people can see each other as actual humans and not just text on a screen.