If a sufficiently interested botnet discovers your SSH server, it'll hammer your server with 10's of connections a second, from many different IPs. That doesn't take up enough bandwidth to be noticeable, but it can fill up the disk, when you're least expecting, which can cause all sorts of trouble.

Of course there are ways to deal with it, but that's just extra work.