Is there a way to have a discussion where we use words according to their ordinary meaning? You said "the government paid ISPs $400 billion dollars." As you acknowledge, money didn't change hands. So nothing was "paid" right?
The government deregulated prices in 1996 for the same reason we deregulated airline fares, truck tariffs, etc. The government wanted to get out of the business of price control. And it didn't just happen in the U.S., it happened all over the world.
Fiber was presented as a benefit that would result from increased capital investment following deregulation. And that happened to an extent--wireline capital expenditure almost doubled from 1996-2001. But after 2000, capital started moving to the fast-growing industry of cellular wireless, where the returns were better. So yeah, we didn't get as much investment in fiber as we expected. But we got huge investments in cellular. That's how free markets work--the government can't predict or direct where capital goes.
The 1996 telecom deregulation wasn't a bargain, where ISPs were allowed to raise prices in exchange for a "promise" to build fiber. It was a deregulation of the industry. It's fair to complain that maybe the benefits of deregulation were oversold. (I can also claim that projections about premiums reductions under Obamacare have been oversold!) I think the 1996 deregulation should have been paired with a preemption of state-and-local broadband construction regulation. (American municipalities regulate in a way that makes it expensive to build infrastructure, whether you're talking about broadband or subway tunnels.) In particular, the nearly universal requirement to agree to service an entire city in exchange for permission to service any part of a city vastly limits the competition that could have been enabled by deregulation.
Its fair to have a debate about the ultimate outcomes of the 1996 deregulation. But its deceitful to act like it was a planned government fiber-construction project where ISPs just "pocketed the money" and didn't build the infrastructure.