Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I feel about AI

beza1e1.tuxen.de

Post reply on HN
Print view View original

131–140 of 262 posts

Re: How I feel about AI

#131

Post by Crabber »

cubefox wrote: post #120
Crabber wrote: post #18

>a superintelligent AI will inevitably destroy humanity Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? Perhaps an agent…

> >a superintelligent AI will inevitably destroy humanity > Can someone please explain to me how an LLM is going to "destroy humanity"? He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.

Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm.

And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.

Re: How I feel about AI

#132

Post by arctic-true »

bigstrat2003 wrote: post #60
Crabber wrote: post #18

>a superintelligent AI will inevitably destroy humanity Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? Perhaps an agent…

> how is an AI going to affect anything in the real world? Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but…

This is exactly correct. You see some discussion of a superintelligent AI becoming “superhumanly persuasive,” such that it can talk anyone into doing anything, and using this as a means to take power. But even with AI that was orders of magnitude weaker than superintelligent, people were getting talked into doing all kinds of stupid nonsense.

And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?

Re: How I feel about AI

#133

Post by mitthrowaway2 »

Crabber wrote: post #18

>a superintelligent AI will inevitably destroy humanity Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? Perhaps an agent…

The list of things that can be done digitally includes:

Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.

(And of course, building robots, but let's set that aside).

So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?

Re: How I feel about AI

#134

Post by dwaltrip »

ThrowawayR2 wrote: post #73
Crabber wrote: post #18

>a superintelligent AI will inevitably destroy humanity Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? Perhaps an agent…

A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in " amateurs study [military] tactics, professionals study logistics ". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon. The actual likely mechanism that AI would…

Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by researchers.

It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least.

To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios.

The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation.

Link: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

Re: How I feel about AI

#135

Post by add-sub-mul-div »

tomashubelbauer wrote: post #7

This and "Intellectual Fly Is Open" being at the top of the Hacker News front page right now and both having their titles auto-editorialized by HN really makes me wonder what benefit this mechanism is supposed to bring readers other than needless confusion.

The hubris that simple code can fix deceptively complex and ambiguous human problems is too baked into the community to separate.

Re: How I feel about AI

#136

Post by FabHK »

arctic-true wrote: post #119
Crabber wrote: post #18

>a superintelligent AI will inevitably destroy humanity Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? Perhaps an agent…

Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the…

Sure, the world was never safe for individuals, and even entire civilisations have gone extinct. For climate change and even nuclear weapons, most of humanity might perish, but there's the hope that some will survive. So, in the past humanity was not in a position to extinguish humanity as a whole.

Re: How I feel about AI

#137

Post by Crabber »

cush wrote: post #78
Crabber wrote: post #45

Earlier quoted context omitted.

For millennia evil people and dictators have been using manipulation, propaganda, threats of violence to get entire populations to try and do "things that they wouldn't do otherwise" and humanity has not been exterminated yet. Even in the modern world there's human scammers that try everything to coerce and manipulate people. Humans are resistant to this kind of thing because it's been part of human social society si…

The scale and personalization is unlike anything people have ever encountered. The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone. Also think of it on a 1000+ year timescale, which for an entire species isn’t even typically measurable. On that timescale AI can easily cause us to discover countless technologies to assist mov…

>The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.

You could say the same thing about social media algorithms though, or scammers who target people directly, foreign agents posting propaganda campaigns on social media sites. People have been claiming for years that there are armies of russian or chinese or whoever bots online trying to destabilise and destroy the west. Some of that stuff probably is effective and probably has influenced people's thoughts and values, changed their voting patterns, put decadent ideas in people's heads. Even on social media other humans are constantly trying to manipulate you into following them, buying stuff from them or certain brands, putting political ideas in your heads.

I'm sure an army of AI agents could also do all of this but what I'm saying is it's really nothing new. And none of this has lead to the destruction of humanity so far.

Re: How I feel about AI

#138

Post by mitthrowaway2 »

Crabber wrote: post #116
dist-epoch wrote: post #107

Earlier quoted context omitted.

> it is very easy to disrupt the operation of large language models so go on, disrupt the OpenAI data-center in Texas for 1 minute

We are talking about a humans vs AI scenario. There are probably thousands of humans in the world right now that could do exactly that right now if they wanted to (datacenter and OpenAI employees, power grid workers).

Are you assuming that the humans are all unanimously aligned against the AI in this scenario? That sounds like the hard part. How does that happen? Does the AI announce its evil plans like a movie villain?

Until then, it's the job of the power grid workers to keep the power running, and shutting down the datacenters would just get them fired.

Re: How I feel about AI

#139

Post by bobsmooth »

hn_throwaway_99 wrote: post #52

The thing that makes me pretty terrified about AI is that I think the "utopia" scenario is not much better than the "doom" scenario. We will very quickly get to a point where very few people will be able to contribute economically because they will be worse than AI (including robotics) at most domains. A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this…

Here's a great short story about how it could go either way

https://web.archive.org/web/20100211181147/https://marshallb...

Re: How I feel about AI

#140

Post by simianwords »

> This is even harder now that AI can generate some kinds of "content" cheaper and faster.

Its good that we have competition to so called "art". Art is not a jobs program. If people don't appreciate your art, don't force them to. The revealed preference says everything.

Post reply on HN