Jitter is one of the words you're looking for. 20 milliseconds does not sound like much, but when it's a delay in some (but not all) packets in a call getting to you, you can sure hear it! Jitter is why you could have a 1 GB/s connection at home, and still have the same occurrence of poor call quality. QoS (Quality of Service) can help, but only if it is applied at every router the traffic for a call goes through. Qo…
You really only need it on routers where a bottleneck is. If a router is forwarding 100% of packets immediately there is nothing for QoS to do.
In practice what this means is that the most important place to have it is at the border where you go from high bandwidth LAN to lower bandwidth internet.
In particular if you control a router there, make sure it knows to limit traffic to what the next hop link speed actually is. In will often see the link as gigabit ethernet even if it's slower than that and then forward all traffic when you really want it to be the device that chooses which traffic gets dropped.
Unfortunately if your ISP supports burst bandwidth it's not always easy to get it right without breaking that.