Earlier quoted context omitted.
Is there any specific thing you did apart from having a fiber and being close ? I tried to set up my home PC as a server with tincVPN and X2go, i couldn't get a usable performance. Well, to be fair i tried using KiCAD remotely but even the XFCE desktop interface felt sluggish. I also had fiber on the three nodes (Vultr server, home server and my local machine).
Did you use WiFi? An old Wifi router can easily add half a second of latency. Even high end Wifi endpoint still adds a few ms of latency which can be significant considering that short distances (20 miles) of fiber has sub ms latency. A software router or switch can also add up to a ms of latency. Could also try SSH (tunnel) instead of VPN as SSH encryption might be faster.
I suspected ssh over tinc might bring latency since there is an extra layer of encryption, i will try ssh tunneling. Thanks!