Live data from Hacker News

The first big AI disaster is yet to happen

seangoedecke.com

31–40 of 154 posts

Re: The first big AI disaster is yet to happen

#31

Maybe we should have models govern models to prevent models from going haywire and falling into a loop of a repetitive pattern or influencing society towards swarm like behavior patterns

Reminds me of Neal Stephenson's thoughts on the topic here: https://nealstephenson.substack.com/p/remarks-on-ai-from-nz

> By training AIs to fight and defeat other AIs we can perhaps preserve a healthy balance in the new ecosystem. If I had time to do it and if I knew more about how AIs work, I’d be putting my energies into building AIs whose sole purpose was to predate upon existing AI models by using every conceivable strategy to feed bogus data into them, interrupt their power supplies, discourage investors, and otherwise interfere with their operations. Not out of malicious intent per se but just from a general belief that everything should have to compete, and that competition within a diverse ecosystem produces a healthier result in the long run than raising a potential superpredator in a hermetically sealed petri dish where its every need is catered to.

Re: The first big AI disaster is yet to happen

#32
post #17

I feel like the first big disaster already happened. In the job market. Anyway, where there is public safety risk involved, I can't imagine a scenario where AI will directly cause a catastrophe. If anything, I feel overall safety will increase. However, I can't help but wonder what the over reliance would do to general population over the next few decades. Would people essentially become dumber and less skilled?

I don’t think AI is meaningfully taking any jobs. Just like return to office mandates, it is just more cover to do layoffs in less than prosperous economic conditions.

Yep I agree. It does make senior developers more efficient at building new features but it's maybe like 20% more efficient in the grand scheme of things... But this efficiency saving is a joke considering that companies spend most of their development money creating and maintaining unnecessary complexity. People will just produce unnecessary code at a faster rate. I think the net benefit for a typical corporation will be negative and they will need to hire more people to maintain the increasingly large body of code, not fewer.

It's an intractable problem because even if some really astute senior engineer joins a company and notices the unnecessary complexity and knows how to fix it, there is no way they will do anything about it. Do you think thousands of existing engineers will support a plan to refactor and simplify the complexity? The same complexity which guarantees their job security? It's so easy to discredit the 'new dev' who is advocating for change. Existing devs will destroy the new dev with complicated-but-deeply-flawed arguments which management will not fully comprehend. At the end of the day, management will do what the majority of existing devs tell them. Devs will nod in unison for the argument which sounds more complex and benefits them the most.

Nobody ever listens to the new hire, even the new hire knows they are powerless to change the existing structure. The best they can do is add more complexity to integrate themselves into the scheme.

Ironically, the new hire can provide more value doing nothing but that will not provide them with job security. The winning strategy is to trade-off deep long-term value for superficial short-term value by implementing lots of small features loaded with technical debt.

It's like gears in a big machine, you could always add 3 gears to do the exact same job as 1 gear but once those 3 gears are in place, remove any of them, the machine stops working... So each one seems demonstrably essential to anyone who doesn't understand the tech. That's the principle. When pressed, a merchant of complexity can always remove a gear and say "Look, the machine doesn't work without it, just like I told you."

Re: The first big AI disaster is yet to happen

#33
post #10

The problem with AI is that we might get an artificial general superintelligence before we get the first big disaster, and when a big disaster happens at that point, it could be so big that it’s catastrophic for humanity. Lots has been written about this topic, https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-annou... is one of the more pessimistic takes. I wrote a more neutral introduction a while back at http…

Interesting how this has been downvoted to grey. Is it just me or have AI doomers become unfashionable over the last 3-6 months or so, as in rather quickly? OP I'm curious if you've noticed people's responses to you change recently.

I think it’s more because we aren’t getting AGI anytime soon.

Re: The first big AI disaster is yet to happen

#34
post #7

> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would…

Cooking has proven its utility a long time ago. AI girlfriends? Let's say, it's still on the fence.

Re: The first big AI disaster is yet to happen

#35
post #17

I feel like the first big disaster already happened. In the job market. Anyway, where there is public safety risk involved, I can't imagine a scenario where AI will directly cause a catastrophe. If anything, I feel overall safety will increase. However, I can't help but wonder what the over reliance would do to general population over the next few decades. Would people essentially become dumber and less skilled?

I don’t think AI is meaningfully taking any jobs. Just like return to office mandates, it is just more cover to do layoffs in less than prosperous economic conditions.

AI may prevent lost jobs from coming back, even if it didn't cause the loss. Companies will be able to do more with less.

Re: The first big AI disaster is yet to happen

#36
Covid response / lockdown would be the first epistemological catastrophe due to AI . Not strictly due to LLM, but broadly due to poorly configured and likely tampered-with diagnostics signals, wryly inference models that led to global triggering of failsafe's and lockdown response protocols.

In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were amplified on social media, while critical voices were muffled.

If you think about AI as an end to end process, with diagnostic signals on one end and martial authority on the other -- covid was a stress-test & failure of the intervening systems (and protocols).

With LLM, you'll have more fog-of-war on the diagnostics side, vastly more complicated inference, infinite amounts of fabricated narratives, and broader scope of controls on the authority side.

Re: The first big AI disaster is yet to happen

#37
post #11

The first big AI disaster has already happened and it is the thousand little cuts from people putting faith into systems that aren't built to be trustworthy. It is the software dev missing a critical safety check because "ai coding tools" at work have promised 10x delivery results and now they must use them or be seen as falling behind. It is the engineering manager throwing concrete flexural strength questions into…

> The first big AI disaster has already happened and it is the thousand little cuts

I agree with your point, but per your own words that isn’t a big disaster but multiple little ones. Each person might have learned about some of them but only in passing and without too much information.

“The big one” will be something which everyone will be made aware of from the news, like the CrowdStrike outage.

Re: The first big AI disaster is yet to happen

#38

the first 'ai disaster' will be a corp shirking responsibility by blaming their unchecked automatic bureaucratic processes on ai. Look no further at what HERTZ did sending out automated arrest warrants for erroneous causes, leading to dangerous confrontations with armed police. Luckily no one was killed, but innocent law abiding people were very traumatized. This was without 'ai' but the blame was nearly close, "oops…

B is for Bureaucracy.

Re: The first big AI disaster is yet to happen

#39
post #2

While I agree, it won't be a purely "AI" disaster the way a plane crash is a direct consequence of the kinetic and potential energy associated with flying. s/AI/automation/ it's always bad to hook an unchecked algorithm up to something dangerous, it doesn't matter if it's an if statement or a neural net. It will be the hooking up that's responsible more than the AI which will just be what fools someone into thinking…

[deleted]

Re: The first big AI disaster is yet to happen

#40
post #10

The problem with AI is that we might get an artificial general superintelligence before we get the first big disaster, and when a big disaster happens at that point, it could be so big that it’s catastrophic for humanity. Lots has been written about this topic, https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-annou... is one of the more pessimistic takes. I wrote a more neutral introduction a while back at http…

Interesting how this has been downvoted to grey. Is it just me or have AI doomers become unfashionable over the last 3-6 months or so, as in rather quickly? OP I'm curious if you've noticed people's responses to you change recently.

It's been longer than that, more than 2 years since Time ran EY's moron piece which put the whole field in disrepute and back on its heels basically. That the community spins out literal murder cults and assassins doesn't help.
Post reply on HN