Earlier quoted context omitted.
>If the answer is no, then you've very little to worry about as long as you practice good security habits. This is very hard to define. I have all my servers on cloud locked down and can only ssh with keys (No passwords) and everytime I look at the access log, it just makes me sweat seeing all the "drive by" ssh access attempts using standard usernames (which I don't use) and even unstandard user names (like "mother"…
Have you tried/are you interested in setting a different port number on the server? Setting a different port number and using ssh_config on your clients (so you don't have to keep specifying the different port) helps a ton. I've done this with a Google 2FA solution[1] and honestly haven't seen a single rogue attempt. ----- [1] https://www.digitalocean.com/community/tutorials/how-to-set-...
Changing ssh port from standard (I literally forgot, is it 22?) to something else is one of those things you pick up in the field.
Usually only once you actually had to maintain production servers. And hopefully before some of the fancy things (e.g.: port-knockers) failed and locked you out :)
Changing port is super simple to configure and yet it cuts out 99.9% of noise (if not 100%) from brute-force attempts that otherwise might make you ignore alarms ...