Earlier quoted context omitted.
So that means you can't SSH into your remote machine from an arbitrary connection? What do you do when you're traveling?
You use a VPN and then connect to SSH
IE, Why is it important that your SSH port only be connected to by known IPs, but your VPN port is OK to be connected to from anywhere?