The Internet is optimized for throughput at lowest cost.No amount of good netcode that you write can compensate for this.
The problem is the internet itself.
The internet doesn’t care about your game.
The Internet did change, such that streaming movies became viable. Given that games are now bigger than movies, could it be argued that that the network lag we're seeing is the industry lagging behind the culture? (Which it generally does, at least a little.)
A part of this might well involve special deals between large companies involving specialized infrastructure. There are such things now involved with movie streaming. Google's plans involve the infrastructure part of that already. What happens to net neutrality?
Once game streaming becomes mature, then it's only a matter of time for App streaming to follow. We'll be back in the mainframe days, just with much larger companies with much larger multi-box cloud "mainframes", commanding the attention of a much larger portion of the population and the culture. "The computer is your friend. Trust the computer!" https://en.wikipedia.org/wiki/Paranoia_(role-playing_game)