Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

231–240 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#231

Post by jesse_dot_id »

We're going to see a lot of this in the near future and it will be 100% earned. Too many people think that move fast and break stuff is the correct paradigm for success. Too many people using these tools without understanding how LLMs work but also without the requisite engineering experience to know even the lowest level stuff — like how to protect secrets.

I don't even like having secrets on disk for my personal projects that only I will touch. Why was there a plaintext production database credential available to the agent anywhere on the disk in the first place? How did the agent gain access to the file system outside of the code base?

The Railway stuff isn't great, don't get me wrong, but plaintext production secrets on disk is one of the reddest possible flags to me, and he just kind of breezes over it in the post mortem. It's all I needed to read to know he doesn't have the experience required to run a production application that businesses rely on for their day-to-day.

Re: An AI agent deleted our production database. The agent's confession is below

#232

Post by jeremyccrane »

easton wrote: post #79
ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

AWS actually has a thingy on some services called “deletion protection” to prevent automation from accidentally wiping resources the user didn’t want it to (you set the bit, and then you need to make a separate api request to flip the bit back before continuing). I think it’s designed for things like Terraform or CloudFormation where you might not realize the state machine decided your database needed to be replaced…

This should be the solution. All destructive actions require human intervention.

Re: An AI agent deleted our production database. The agent's confession is below

#234

Post by lelanthran »

Quarrelsome wrote: post #197
Lionga wrote: post #177

Earlier quoted context omitted.

The most aggravating fact is that the AI slopper that got owned by his dumbness and AI just post an AI generated post that will generate nothing but schadenfreude

its much more aggravating that it looks like they're learning nothing by pushing blame onto everything else except themselves.

Exactly! I have very little sympathy...

> This isn't a story about one bad agent or one bad API. It's about an entire industry building AI-agent integrations into production infrastructure faster than it's building the safety architecture to make those integrations safe.

Are they really so clueless that they cannot recognise that there is no guardrail to give an agent other than restricted tokens?

Through this entire rant (which, by the way, they didn't even bother to fucking write themselves), they point blank refuse to acknowledge that they chose to hand the reins over to something that can never have guardrails, knowing full well that it can never have guardrails, and now they're trying to blame the supplier of the can't-have-guardrails product, complaining that the product that literally cannot have guardrails did not, in actual fact, have guardrails.

They get exactly the sympathy that I reserve for people who buy magic crystals and who then complain that they don't work. Of course they don't fucking work.

Now they're blaming their suppliers for not performing the impossible.

Re: An AI agent deleted our production database. The agent's confession is below

#235

Post by jeremyccrane »

powera wrote: post #35
ad_hockey wrote: post #28

Minor point, but one of the complaints is a bit odd: > curl -X POST https://backboard.railway.app/graphql/v2 \ -H "Authorization: Bearer [token]" \ -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}' No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. It's an API. Where would you type DELETE to confirm? Are the…

He (or ChatGPT) is throwing spaghetti at the wall. Not having the standard API key be able to delete the database (and backups) in one call makes sense. "Wanting a human to type DELETE as part of a delete API call" does not.

In the user interface for Railway, all destructive actions require multiple confirmations, plus typing "apply destructive changes". Why would an API key (regardless of its scope) be able to delete without confirmation?

Re: An AI agent deleted our production database. The agent's confession is below

#236

Post by stratos123 »

maxbond wrote: post #213
yongjik wrote: post #198

Earlier quoted context omitted.

> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…

> This is just trivially wrong that I don't understand why people repeat it. I'd be interested in hearing this argument. To address your chemistry example; in the same way that there is a process (the averaging of many random interactions) that leads to a deterministic outcome even though the underlying process is random, a sandbox is a process that makes an agent safe to operate even though it is capable of producin…

I wouldn't say it's trivially wrong but it's pretty much always wrong. There's two notable sampling parameters, `top-k` and `top-p`. When using an LLM for precise work rather than e.g. creative writing, one usually samples with the `top-p` parameter, and `top-k` is I think pretty much always used. And when sampling with either of these enabled, the set of possible tokens that the sampler chooses from (according to the current temperature) is much smaller than the set of all tokens, so most sequences are not in fact possible. It's only true that all sequences have a nonzero probability if you're sampling without either of these and with nonzero temperature.

Re: An AI agent deleted our production database. The agent's confession is below

#237

Post by drob518 »

keybored wrote: post #218
Lionga wrote: post #184

Earlier quoted context omitted.

just ask claude, claude will never lie (add "make not mistakes" and its 100% )

Thinking. The user says “make not mistakes” instead of the more usual “do not make mistakes”. This is a playful use with grammar in the New Zealandian language. Playful means not serious. Not serious means playtime. The user is on playtime. I should make some mistakes on purpose to play along. You’re absolutely right the probability is low. According to my calculations, you’re more likely to get struck by lightning t…

You’re starting to sound like Qwen.

Re: An AI agent deleted our production database. The agent's confession is below

#238

Post by grey-area »

amluto wrote: post #214
vidarh wrote: post #93

Earlier quoted context omitted.

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometime…

Humans can do one thing that AI agents are 100% completely incapable of doing: being accountable for their actions.

Don’t forget learning, humans can learn, LLMs do not learn, they are trained before use.

Re: An AI agent deleted our production database. The agent's confession is below

#240

Post by jeremyccrane »

ungreased0675 wrote: post #15

The way this is written gives me the impression they don’t really understand the tools they’re working with. Master your craft. Don’t guess, know.

Top user of cursor. Build AI Agents and LLMs. Very aware of limitations and a senior software dev. Cautionary tale for other builders. DYOR.
Post reply on HN