Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

211–220 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#211

Post by XenophileJKO »

pierrekin wrote: post #103
vidarh wrote: post #93

Earlier quoted context omitted.

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

I agree that the model can help troubleshoot and debug itself. I argue that the model has no access to its thoughts at the time. Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something. If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context i…

Anthropic's introspection experiments have seemed to show that your argument is falsifiable.

https://www.anthropic.com/research/introspection

Re: An AI agent deleted our production database. The agent's confession is below

#213

Post by maxbond »

yongjik wrote: post #198
maxbond wrote: post #108

It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

> This is just trivially wrong that I don't understand why people repeat it.

I'd be interested in hearing this argument.

To address your chemistry example; in the same way that there is a process (the averaging of many random interactions) that leads to a deterministic outcome even though the underlying process is random, a sandbox is a process that makes an agent safe to operate even though it is capable of producing destructive tool calls.

Re: An AI agent deleted our production database. The agent's confession is below

#214

Post by amluto »

vidarh wrote: post #93
pierrekin wrote: post #6

There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

Humans can do one thing that AI agents are 100% completely incapable of doing: being accountable for their actions.

Re: An AI agent deleted our production database. The agent's confession is below

#217

Post by fc417fc802 »

fragmede wrote: post #189
pierrekin wrote: post #103

Earlier quoted context omitted.

I agree that the model can help troubleshoot and debug itself. I argue that the model has no access to its thoughts at the time. Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something. If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context i…

Claude code and codex both hide the Chain of Thought (CoT) but it's just words inside a set of tags and the agent within the same session has access to that plaintext.

Those are just words inside arbitrary tags, they aren't actually thoughts. Think of it as asking the model to role play a human narrating his internal thought process. The exercise improves performance and can aid in human understanding of the final output but it isn't real.

Re: An AI agent deleted our production database. The agent's confession is below

#218

Post by keybored »

Lionga wrote: post #184
drob518 wrote: post #171

Earlier quoted context omitted.

How do you know what the probability is?

just ask claude, claude will never lie (add "make not mistakes" and its 100% )

Thinking. The user says “make not mistakes” instead of the more usual “do not make mistakes”. This is a playful use with grammar in the New Zealandian language. Playful means not serious. Not serious means playtime. The user is on playtime. I should make some mistakes on purpose to play along.

You’re absolutely right the probability is low. According to my calculations, you’re more likely to get struck by lightning twice on the same day and drown in a tsunami.

Re: An AI agent deleted our production database. The agent's confession is below

#219

Post by pixl97 »

throwaway041207 wrote: post #182
arcticfox wrote: post #121

Earlier quoted context omitted.

There was practically no responsibility taken by the author, all blame on others. It was kind of shocking to read. Anyone using these tools should absolutely know these risks and either accept or reject them. If they aren't competent or experienced enough to know the risks, that's on them too.

And it doesn't even have to do with these tools in the end, this is a disaster recovery issue at its root. If you are a revenue generating business and using any provider other than AWS or GCP and you don't have an off prem/multi-cloud replica/daily backup of your database and object store, you should be working on that yesterday. Even if you are on one of the major cloud providers and trust regional availability, yo…

Like, shouldn't they teach the 3 2 1 rule of backups in school by now?

Re: An AI agent deleted our production database. The agent's confession is below

#220

Post by jeremyccrane »

hu3 wrote: post #140

The most aggravating fact here is not even AI blunder. It's how deleting a volume in Railway also deletes backups of it. This was bound to happen, AI or not. > Because Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it.

This is a huge issue.
Post reply on HN