Live data from Hacker News

Humanity isn't ready for the coming intelligence explosion

economist.com

211–220 of 524 posts

Re: Humanity isn't ready for the coming intelligence explosion

#211

We don't need to live "side by side" with AI. AI is not alive, it's a technology we use. This is like talking about living side by side with your toaster.

I'll use this opportunity to recommend The Selfish Gene, which puts forward a strong argument for how the actual competitive battlefield on Earth is between genes, while us humans and everything else we call "life" are their "survival machines" (read: technology that they use).

Re: Humanity isn't ready for the coming intelligence explosion

#212

We don't need to live "side by side" with AI. AI is not alive, it's a technology we use. This is like talking about living side by side with your toaster.

I would happily live side by side if it is the Red Dwarf toaster

Re: Humanity isn't ready for the coming intelligence explosion

#213

Earlier quoted context omitted.

Here is an issue I think about often, but I am not quite sure how to put it into words. We have many extremely smart people in various fields. Executives, politicians, and society generally ignore them and do whatever they want. I don't believe that lack of access to intelligence is our problem. How is "free" intelligence going to improve this? I don't just mean climate, but business planning, health, risk assessment…

The free intelligence may help to steer the society and politicians to one or the other side. The only question is who decides which.

No invention was able to steer the psychopaths away from waging wars.

But this time it's absolutely different, right? Riiight?..

Re: Humanity isn't ready for the coming intelligence explosion

#215
post #154

Earlier quoted context omitted.

Why is AI not alive

You might be trolling, but I like Schrödingers definition of life: living things avoid decay into equilibrium. Inference engines are trying to converge any input to a certain limit, and I presume that's close enough to an equilibrium to not count as living thing.

I am no troll, and I hate to look like one when, to the appeal of Schrodinger, I am forced to remember that announcement this month about Claude or something acting in self-preservation when threatened during testing.

Re: Humanity isn't ready for the coming intelligence explosion

#216
post #200

Earlier quoted context omitted.

"but nobody would argue that an asteroid is alive" Many animists would disagree: https://en.wikipedia.org/wiki/Animism

TIL, thanks. Holy fuck we humans are crazy sometimes.

To be honest, I don't believe it is a any more crazy believe than all of the other religions. But practitioners there might have a bit more respect for existing things and not mindlessly crush because they can, so I actually like the concept a bit. But I assume it can also set foundation for some exhausting debates .. "can we really move that random rock to build a house with?" but as far as I understood the practical answer is "we ask the spirits, maybe make a offering and if we feel it is fine, we can" (which can be a quick thing, or .. not).

Re: Humanity isn't ready for the coming intelligence explosion

#217
post #189

Earlier quoted context omitted.

"Alive" is irrelevant. Bullshit like "alive", "sentient" or "conscious" doesn't decide much, at the day's end. It just distracts from the thing that matters. And the thing that matters is: capability. Even today's AIs are capable of autonomous goal-oriented agentic behavior - and growing more and more so with every release. At sufficient capability, AI stops being "a technology we use", and becomes a force in itself…

DNNs/LLMs can only predict next tokens based on training data. They often make big direction mistakes as they are particularly bad at common sense. Kind of like the Paperclip Maximizer scenario. They need a human with deep knowledge to drive them and to catch them when they go off the rails.

"Next token prediction" isn't a system. It's an interface a system uses. Nothing precludes an arbitrarily simple or complex behavior from producing a token logit.

And with what we know of LLMs? Autoregressive transformers are Turing complete in theory, and we are yet to find anything that LLMs are "fundamentally incapable" of in practice. Even continuous learning is already approximated with in-context learning - both allow a system to learn from prior experience, both have practical limits on how far they go. That's what powers "trial and error" in today's agentic LLMs.

"LLMs can only predict next tokens based on training data" is comforting but misleading. It just isn't the saving grace you want it to be. It describes an interface, not a ceiling. And if there is some sort of fundamental "capability ceiling" that LLMs are heading towards, we are yet to see it. We know plenty of things LLMs are bad at, but they keep getting less bad at them release to release.

If there is none, then, simply improving over the current recipes iteratively might yield systems that only "need a human" in the same way you "need" to have a boss. Maybe less so.

Re: Humanity isn't ready for the coming intelligence explosion

#218
post #154

Earlier quoted context omitted.

Why is AI not alive

It has no body, metabolism or reproductive capability. It's a process, not an entity.

Being a process doesn't preclude something from being an entity. In particular, all of us living organisms have our constituent atoms constantly replaced, and we don't typically consider that a risk to our sense of self.

Re: Humanity isn't ready for the coming intelligence explosion

#219

Earlier quoted context omitted.

No, that’s not accurate. I’m not sure what to tell you. It’s like hearing that no serious programmer uses Python, it’s so far from my experience and that of everyone I know in the field that I don’t know how to engage.

Do you think that everyone in the Linux kernel team write their code using AI only? What about Emacs? And cURL?

Ok, how about Ladybird: https://ladybird.org/posts/adopting-rust/

> I used Claude Code and Codex for the translation. This was human-directed, not autonomous code generation. I decided what to port, in what order, and what the Rust code should look like. It was hundreds of small prompts, steering the agents where things needed to go. After the initial translation, I ran multiple passes of adversarial review, asking different models to analyze the code for mistakes and bad patterns.

Sounds to me like "wrote their code using AI only".

And from the creator of Redis, antirez: https://antirez.com/news/158

> It is simply impossible not to see the reality of what is happening. Writing code is no longer needed for the most part.

Re: Humanity isn't ready for the coming intelligence explosion

#220

I couldn't read the whole article, but just from the part I could read: > Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories. I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a m…

> an incentive to make their technology seem transformative and powerful I have no frame of reference to process this. Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans…

Agree with most of this, it's well articulated and captures how we react to change.

However - 'Humans species perhaps 300,000 years, we are essentially the same' is an enormous misconception. That fact that we lack gross anatomical changes during this period, ignores everything we now know about punctuated equilibrium and rapid evolution. It's highly probably we've had an enormous number of evolved psychological changes during the last few hundred, and even tens of thousands of years. Changes that relate to our capacity to live in large groups, adapt to urban environments, resist disease and so on. We know that's the case simply because acute pandemics become epidemics through herd immunity, and through the acquisition of lactose tolerance etc.

It seems highly unlikely that adaptations stop there. Altering the environment (in the last 10K years that means the built environment) alters the environment of evolutionary adaptiveness. It seems likely that we've essentially domesticated out much of our propensity for violence and increased our capacity for mood regulation.

Obviously it's incredibly tricky to pair these specific behavioural changes to genetic changes -> protein synthesis -> behaviour. Bearing in mind though we're only 20 years out from the first study to link allele variant to behaviour (the COMT Val/Met polymorphism), and the potential controversy around such research, this shouldn't be surprising.

Post reply on HN