> My understanding is we underwent a reorientation in the 1980s, because it was realized that there was too little attention paid to the benefits that a relatively autonomous economy can produce, killing off bad ideas, where what's being offered doesn't match with what people need or want, and allowing better ones to replace them.
There was a rightward swing in the late 1970s that took root in our political system, then commentariat, and then culture. It has never reverted. The term "neoliberalism" gets thrown around (usually by dweebs like me) but it's the precise term to use. Wendy Brown's recent book is probably the best overview of the topic in recent years.
The cultural shift that was unleashed in that period is so insidious that you don't even notice it half the time. Think about dating apps/sites where users talk about their romantic lives using terms like "R.O.I." Or people discussing ways to "optimize" their lives by making them more efficient. It's nuts.
Steve Jobs' old "bicycle of the mind" chestnut is, in a way, emblematic of this way of thinking. He was talking about how the most "efficient" animal was a human with a bicycle. He wanted human thinking to be "more efficient." If you listen to Kay, on the other hand, he's talking about something entirely different. The transcendent effect of literacy on mankind created the very possibility of civilization, for good or ill. Computing as an aid to thinking in the way the written word was previously could take this to the next, higher stage – one we cannot really describe or talk about because we don't even have the language to do so.
But short term thinking, shareholder value, and the need for economic growth – these are and have been the pillars of our politics and culture for several decades now. No one says who that growth benefits, of course, which is why it's no coincidence that the maw of inequality has opened ever wider during the same period. If you're wondering where all the "good ideas" are, well, we don't have time for good ideas. We only have time for profitable ones, or at least ones that can be sold after a high valuation.
The culture also trickled into the university, and then to funding (not just science funding, but funding for most fields. We need more than science to do new science). I have been on the bitch end of writing NSF grants for pretty ambitious projects, and the requirements are straight out of Kafka. They want you to demonstrate that you'll be able to do the things you're saying you'll hope to be able to do. That's not how it used to work. But the angle is always the same: they want something "innovative" that can be useful as immediately as possible. Useful for the economy, that is. They don't understand this undeniable fact: if you want amazing developments, you have to let passionate and smart people screw around and you have to pay them for it. The university used to be the place to screw around with ideas and methods. Now it's career prep.
> I understand where the impulse to focus on that comes from, because globalization tends to produce a much more competitive economic landscape, where people feel much more uncertain about basic questions they have to answer
This kind of globalization is a choice, one made by powerful people with explicit interests. It was not inevitable. Right now I live in the wealthiest country that has ever existed on the planet. And right now many of its citizens are calling their elected leaders to beg them not to take away the sliver of health care that they have left. We serve the economy and not each other. When there's a big decision to make, our leaders wonder "how the market will react," rather than how people will be affected.
Last point: the idea of this thing called "the economy" as an object of policy is relatively new. Timothy Mitchell has an amazing chapter on it in his book Carbon Democracy. The 20th century was one where we allowed the field of economics to cannibalize all others. The 21st has not taken the chance to escape this.