401(k)'s started to became popular in the 1970s because they allowed management to increase their tax-advantaged compensation relative to labor, whereas defined-benefit plans had stricter and more effective rules regarding management/labor compensation disparities.
Then during the 1980s as corporate accounting schemes became more sophisticated preference for 401(k)'s and other defined-contribution plans exploded because of how liabilities are calculated.
In every conceivable measure pensions are theoretically better for workers and society as a whole. I say theoretically because that's predicated on employers and pension funds (if employer managed) obeying good (translation: dead simple) accounting practices rather than spending all their time and effort figuring out how to subvert them in order to cook their balance sheets to increase their yearly bonuses. Many corporate merger waves since the 1980s have been driven by schemes to drain pension funds (cost "synergies"), and of course states are notorious for failing to fully fund pensions on a YtoY basis (states, unlike corporations, aren't actually required to fully fund pensions[1]).
All of these downsides to pensions can be easily remedied. But corporate interests have succeeded in selling the narrative that pensions are unreliable and inequitable, while 401(k)'s are more reliable and equitable. In fact pensions are categorically more equitable, and any less reliability (which is a dubious claim, notwithstanding the many high profile pension failures over the years--nobody reports on someone's 401(k) fund vanishing during a recession) is a consequence of lobbyists phenomenal success in killing legislation and enforcement efforts responsive to corporate financial accounting schemes. Overall reliability of private retirement systems has fallen over the past 40 years, but the decline for pensions was more precipitous simply because they were so damned good before CFO offices became profit centers.
Life insurance markets were once dens of fraud and outright thievery, the mortgage backed securities and CDOs of the late 19th and early 20th century. Relatively simple reforms restored the market to nearly unassailable reliability. Though, I suspect the necessary legislation only succeeded after life insurance stopped being the center of growth and profit for financial markets. Legislation regarding private retirement plans likely will only ever make substantial reform once corporations have finished exploiting all opportunities for subverting employee income disparity and asset protections, finally shifting their attention elsewhere.
[1] "Fully fund" does not mean paying in the entirety of a worker's expected retirement disbursements. Rather, it means ensuring that each and every year you pay the full fractional share of expected liabilities. Basically the same thing an individual is expected to do when managing their 401(k) contributions.