If you're lucky enough to have a big ISP with a single big block of IP addresses that never changes you can disallow all other ranges on your VPS admin ports and only have to worry about VPNing through that ISP. I guess you could block the main country offenders but you'd have to pay an API to keep up with the IP allocations to be sure.
Or alternatively, block port 22 entirely on your firewall and use something like Tailscale to access the machine. Of course, now your attack surface includes Tailscale, which has had it's own vulns in the past, but I think blocking all public traffic ends up being much stronger than any weaknesses Tailscale may introduce.
Also, it's a bit tricky to set up but port knocking is a very effective solution, and you can keep the SSH on port 22 if you like.