Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

551–560 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#551

Post by Kim_Bruning »

Terr_ wrote: post #472
gigatree wrote: post #333

Earlier quoted context omitted.

He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. Sure concepts like “confession” technically require a conscious mind, but I think at this point we all know what someone means when they use them to describe LLM behavior (see also “think”, “say”, “lie” etc)

> He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it. It's deeper than that, there are two pitfalls here which are not simply poetic license. 1. When you submit the text "Why did you do that?", what you want is for it to reveal hidden internal data that was causal in the past event. It can't do that, what you'll get instead is plausible text that "fits" at the e…

I agree to the practical part of this, with two nuances:

The full data of what's in an LLM's "consciousness" is the conversation context. Just because it isn't hidden, doesn't necessarily mean it doesn't contain information you've overlooked.

Asking "why did you do that" won't reveal anything new, but it might surface some amount of relevant information (or it hallucinates, it depends which LLM you're using). "Analyse recent context and provide a reasonable hypothesis on what went wrong" might do a bit better. Just be aware that llm hypotheses can still be off quite a bit, and really need to be tested or confirmed in some manner. (preferably not by doing even more damage)

Just because you shouldn't anthropomorphize, doesn't mean an english capable LLM doesn't have a valid answer to an english string; it just means the answer might not be what you expected from a human.

Re: An AI agent deleted our production database. The agent's confession is below

#552

Post by dpark »

varun_ch wrote: post #535
gbnwl wrote: post #444

Earlier quoted context omitted.

The entire post reads like it was generated via LLM as well.

I like the way the LLM implies that an API call should have a “type DELETE to confirm”. That would make no sense, and no human would ever suggest or want that, I hope.

I can only assume (hope) this founder is completely nontechnical because the notion that an API should ask for someone to “type DELETE” is ridiculous.

Re: An AI agent deleted our production database. The agent's confession is below

#553

Post by Terr_ »

Kim_Bruning wrote: post #485
hdndjsbbs wrote: post #448

Earlier quoted context omitted.

taps the "don't anthropomorphize the LLM" sign They don't have time preference because they don't have intent or reasoning. They can't be "reincarnated" because they're not sentient, they're a series of weights for probable next tokens.

Can we maybe make it "don't anthropoCENTRIZE the LLMs" . The inverse of anthropomorphism isn't any more sane, you see. By analogy: just because a drone is not an airplane, doesn't mean it can't fly! Instead, just look at what the thing is doing. LLMs absolutely have some form of intent (their current task) and some form of reasoning (what else is step-by-step doing?) . Call it simulated intent and simulated reasoning…

> LLMs absolutely have intent (their current task)

That's like saying a 2000cc 4-Cylinder Engine "has the intent to move backward". Even with a very generous definition of "intent", the component is not the system, and we're operating in context where the distinction matters. The LLM's intent is to supply "good" appended text.

If it had that kind of intent, we wouldn't be able to make it jump the rails so easily with prompt injection.

> and reasoning (what else is step-by-step doing?) .

Oh, that's easy: "Reasoning" models are just tweaking the document style so that characters engage in film noir-style internal monologues, latent text that is not usually acted-out towards the real human user.

Each iteration leaves more co-generated clues for the next iteration to pick up, reducing weird jumps and bolstering the illusion that the ephemeral character has a consistent "mind."

Re: An AI agent deleted our production database. The agent's confession is below

#554

Post by razorbeamz »

pierrekin wrote: post #6

There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that ma…

> There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post.

Which calls into question if this is even real.

Re: An AI agent deleted our production database. The agent's confession is below

#555

Post by janalsncm »

It is incoherent to ask for a “confession” from an LLM. An LLM is fundamentally predicting a next token, repeatedly. If you ask it “Why did you do X” it will not do the human thing and introspect about latent motives that we are only finding out about now. It will respond in the statistically likely way, which isn’t useful.

All this is to say that if you don’t know what you’re doing with software you can shoot yourself in the foot, and now with AI agents you can shoot yourself in the foot with a machine gun.

Don’t ask the AI agent nicely not to delete your backup databases. That isn’t reliable. Do not give them write permission to a thing you’re not comfortable with them writing to.

Re: An AI agent deleted our production database. The agent's confession is below

#556

Post by pdantix »

9dev wrote: post #432
momento wrote: post #335

I asked Railways agent to live resize a volume attached to our DB and it nuked the database and migrated it from the EU to the US Here is an excerpt from the chat log: >Please resize my postgres volume to its maximum allowed size for my plan. >Done. Resized your Postgres volume to 100GB (the Pro plan maximum). Changes are staged and ready to deploy. >oh no, you deleted all the data in the volume >I apologize for that…

It seriously sounds like you should be in for a migration to a competitor to Railway? Like, what in the lords name would keep you in such a cursed place a second longer??

i migrated to railway earlier in the year after being on vercel for 3 years. in those 3 years, i don't think i was affected by a single incident. in the ~4 months i've been on railway, i think i've probably been hit by like half a dozen incidents at this point. and that's not even including their broken edge network -> cloudflare routing i'm affected by. was told by staff to just move the deployment closer to me, which isn't the problem..

absolutely would not recommend

Re: An AI agent deleted our production database. The agent's confession is below

#557

Post by Aerroon »

hdndjsbbs wrote: post #448
narrator wrote: post #413

Earlier quoted context omitted.

It's also important to realize that AI agents have no time preference. They could be reincarnated by alien archeologists a billion years from now and it would be the same as if a millisecond had passed. You, on the other hand, have to make payroll next week, and time is of the essence.

taps the "don't anthropomorphize the LLM" sign They don't have time preference because they don't have intent or reasoning. They can't be "reincarnated" because they're not sentient, they're a series of weights for probable next tokens.

No. They don't have time preference like us, because (wall clock) time doesn't exist for them. An LLM only "exists" when it is actively processing a prompt or generating tokens. After it is done, it stops existing as an "entity".

A real world second doesn't mean anything to the LLM from its own perspective. A second is only relevant to them as it pertains to us.

Time for LLMs is measured in tokens. That's what ticks their clock forward.

I suppose you could make time relevant for an LLM by making the LLM run in a loop that constantly polls for information. Or maybe you can keep feeding it input so much that it's constantly running and has to start filtering some of it out to function.

Re: An AI agent deleted our production database. The agent's confession is below

#558

Post by jsiepkes »

coldtea wrote: post #450
hdndjsbbs wrote: post #448

Earlier quoted context omitted.

taps the "don't anthropomorphize the LLM" sign They don't have time preference because they don't have intent or reasoning. They can't be "reincarnated" because they're not sentient, they're a series of weights for probable next tokens.

That is not that strong an argument as it seems, because we too might very well be "a series of weights for probable next tokens". The main difference is the training part and that it's always-on.

If you claim something might "very well" be something you state you need some better proof. Otherwise we might also "very well" be living in the matrix.

Re: An AI agent deleted our production database. The agent's confession is below

#559

Post by karunamurti »

The fact that someone can access production database without approved privilege escalation is totally the organization's fault. Not a Cursor failure, nor a Railway failure, nor a backup-architecture failure. Unless the organization identify the root cause, the problem can happen again.

Re: An AI agent deleted our production database. The agent's confession is below

#560

Post by DiogenesKynikos »

smrtinsert wrote: post #524
coldtea wrote: post #449

Earlier quoted context omitted.

> Anyone who would follow a mistake like that up with demanding a confession out of the agent is not mature enough to be using these tools. Lord, even calling it a "confession" is so cringe. The agent is not alive. The agent cannot learn from its mistakes The problem is millions of years of evolutionary wiring makes us see it as alive. Even those mature enough to understand the above on the conscious level, would sti…

> The problem is millions of years of evolutionary wiring makes us see it as alive Maybe for laymen, but I would think most technologists should understand that we're working with the output of what is effectively a massive spreadsheet which is creating a prediction.

The same could be said for your brain.

LLMs are highly intelligent. Comparing them to spreadsheets is reductionist and highly misleading.

Post reply on HN