Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

501–510 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#501

Post by hunterpayne »

root_axis wrote: post #410

Ultimately, storing secrets on disk was the problem here. Never store secrets on disk. This is software engineering 101. The excuse that "we didn't know the scope of the token's access" is absurd. You knew it was a secret with access to production infrastructure, that's all you need to know. Their provider only having backups on the same volume as the data is also egregious, but definitely downstream of leaking secre…

"Never store secrets on disk."

Wait till you learn how that API stores cryptographic material.

Re: An AI agent deleted our production database. The agent's confession is below

#502

Post by LetsGetTechnicl »

It's definitely the fault of the operator. But also how many times has an AI deleted or modified files it was told not to touch? (and then lied about doing so?

How have they not solved this permissions problem? If the AI is operating on a database it should be using creds that don't have DELETE permissions.

Or just don't use a tool like AI that can be relied on.

Re: An AI agent deleted our production database. The agent's confession is below

#503

Post by smrtinsert »

827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

> "NEVER FUCKING GUESS"

It's very hard to treat this post seriously. I can't imagine what harness if any they attempted to place on the agent beyond some vibes. This is "most fast and absolutely destroy things" level thinking. That the poster asks for journalists to reach out makes it like a no news is bad news publicity grab. Just gross.

The AI era is turning about to be most disappointing era for software engineering.

Re: An AI agent deleted our production database. The agent's confession is below

#505

Post by 3eb7988a1663 »

827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

  Anyone who would follow a mistake like that up with demanding a confession out of the agent is not mature enough to be using these tools.
The proponents are screaming from the rooftops how AI is here and anyone less than the top-in-their-field is at risk. Given current capabilities, I will never raw-dog the stochastic parrot with live systems like this, but it is unfair to blame someone for being "too immature" to handle the tooling when the world is saying that you have to go all-in or be left behind.

There are just enough public success stories of people letting agents do everything that I am not surprised more and more people are getting caught up in the enthusiasm.

Meanwhile, I will continue plodding along with my slow meat brain, because I am not web-scale.

Re: An AI agent deleted our production database. The agent's confession is below

#507

Post by shiandow »

lmf4lol wrote: post #18

Interesting story. But despite Cursors or Railways failure, the blame is entirely on the author. They decided to run agents. They didnt check how Railway works. They relied on frontier tech to ship faster becsuse YOLO. I really feel sorry for them, I do. But the whole tone of the post is: Cursor screwed it up, Railway screwed it up, their CEO doesnt respond etc etc. Its on you guys! My learning: Live on the cutting e…

For a company that puts DO NOT FUCKING GUESS in their instructions they made a heck of a lot of assumptions

- assume tokens are scoped (despite this apparently not even being an existing feature?)

- assume an LLM didn't have access

- assume an LLM wouldn't do something destructive given the power

- assume backups were stored somewhere else (to anyone reading, if you don't know where they are, you're making the same assumption)

Also you should never give LLMs instructions that rely on metacognition. You can tell them not to guess but they have no internal monologue, they cannot know anything. They also cannot plan to do something destructive so telling then to ask first is pointless. A text completion will only have the information that they are writing something destructive afterwards.

Re: An AI agent deleted our production database. The agent's confession is below

#508

Post by kokada »

whartung wrote: post #461
kokada wrote: post #357

Earlier quoted context omitted.

I don't think this is a minor point. It seems clear by this point that the author is clueless how even API works and are just trying to shift blame for third-parties instead assuming that they're just vibecoding their whole product without doing proper checks. Yes sure, there seems to be lots of ways this issue could have been mitigated, but as other comments said, this mostly happened because the author didn't do it…

It's also moot. If the API replied "Are you sure (Y/N)?" the AI, in the mode it was in, guardrails completely pushed off the side of the road, it would have just said "Yes" anyway. If you needed to make two API calls, one to stage the delete and the other to execute it (i.e. the "commit" phase), the AI would have looked up what it needed to do, and done that instead. It's a privilege issue, not an execution issue.

Exactly, that just reinforces the fact that the author is just blaming others instead of getting any valuable insights about this "postmortem analysis".

Re: An AI agent deleted our production database. The agent's confession is below

#510

Post by mhh__ »

So I heard someone recently in person say "Oh you can just have the AI do things that don't really matter like database transaction"

It's so sad that given these amazing tools the average programmers attitude is to automate the things that should be their edge as an engineer.

Torvalds said that great programmers think about data structures. Midwits let the AI handle it.

Post reply on HN