Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An AI agent deleted our production database. The agent's confession is below

twitter.com

Post reply on HN
Print view View original

641–650 of 1001 posts

Re: An AI agent deleted our production database. The agent's confession is below

#641

Post by torton »

YeGoblynQueenne wrote: post #575
dpark wrote: post #345

I would never, ever trust my data with a company that, faced with this sort of incident, produces a postmortem so clearly intended to shift all blame to others. There’s zero introspection or self criticism here. It’s all “We did everything we possibly could. These other people messed up, though.” You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops…

>> You can’t have production secrets sitting where they are accessible like this. This isn’t about AI. This is a modern “oops, I ran DROP TABLE on the production database” story. There’s no excuse for enabling a system where this can happen and it’s unacceptable to shift blame when faced with the reality that this is exactly what you did. I'm not sure it's as simple as that. Seems like the database company failed to…

Did the flow ask them explicitly for scopes? If not, then they should know there are no restrictions.

It also seems, from the post, that customers were "long asking for scoped tokens" so who and why assumed that this particular token can only add and remove custom domains?

The author is getting roasted here and not without reason.

Re: An AI agent deleted our production database. The agent's confession is below

#643

Post by mschulkind »

compass_copium wrote: post #627
kikimora wrote: post #565

Earlier quoted context omitted.

I did notice how Claude can start looking outside of working directory. It may scan home directory and find Homebrew token or SSH keys and wipe your GitHub repo.

I do not use claude and will use agents only when I am forced to, so I'm genuinely asking here: Can claude or other models not be run as a user or program with limited permissions? Do people just not bother to set it up? Why on earth would anyone run an RNG that can access $HOME/.ssh?

There's absolutely nothing special about any of these agents. They're regular processes that execute some subshells. They're trivially jailable.

Re: An AI agent deleted our production database. The agent's confession is below

#644

Post by godelski »

enochthered wrote: post #626
827a wrote: post #261

The only healthy stance you should have on AI Safety: If AI is physically capable of misbehaving, it might ($$1), and you cannot "blame" the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den. > The agent's confession After the deletion, I asked the agent why it did it. This is what it wrote back, verbatim: Anyone who would follow a mistake like that up with demandin…

Yep. I made a "Read only" mode in pi by taking away "write" and "edit" tools. Claude Code used bash to make edits anyway.

  > Claude Code used bash to make edits anyway.
If you had the former rule why would you ever whitelist bash commands? That's full access to everything you can do.

Same goes for `find`, `xargs`, `awk`, `sed`, `tar`, `rsync`, `git`, `vim` (and all text editors), `less` (any pager), `man`, `env`, `timeout`, `watch`, and so many more commands. If you whitelist things in the settings you should be much more specific about arguments to those commands.

People really need to learn bash

Re: An AI agent deleted our production database. The agent's confession is below

#645

Post by [deleted] »

sfink wrote: post #599
neya wrote: post #563

Earlier quoted context omitted.

I was blown away - how they shrugged it off casually too "it found credentials in one file" - why the fuck does an agent have access to it in the first place? They claim the token should be able to change only custom domains. However, for a user facing app, giving access to that token is destructive too. What a poor argument, I would never take this person seriously in any professional context whatsoever.

I've only recently started using Claude Code, and I tried to be paranoid. I run it in a fairly restrictive firejail. It doesn't get to read everything in ~/.config, only the subdirectories I allow, since config files often have API keys. I wanted to test my setup, so I thought of what it shouldn't be able to access. The first thing I thought of is its own API key (which belongs to my employer), since I figured if som…

[deleted]

Re: An AI agent deleted our production database. The agent's confession is below

#646

Post by dinkumthinkum »

coldtea wrote: post #450
hdndjsbbs wrote: post #448

Earlier quoted context omitted.

taps the "don't anthropomorphize the LLM" sign They don't have time preference because they don't have intent or reasoning. They can't be "reincarnated" because they're not sentient, they're a series of weights for probable next tokens.

That is not that strong an argument as it seems, because we too might very well be "a series of weights for probable next tokens". The main difference is the training part and that it's always-on.

People always say this kind of thing. Human minds are not Turing machines or able to be simulated by Turing machines. When you go about your day doing your tasks, do you require terajoules of energy? I believe it is pretty clear human thinking is not at all like a computer as we know them.

Re: An AI agent deleted our production database. The agent's confession is below

#647

Post by linkregister »

compass_copium wrote: post #627
kikimora wrote: post #565

Earlier quoted context omitted.

I did notice how Claude can start looking outside of working directory. It may scan home directory and find Homebrew token or SSH keys and wipe your GitHub repo.

I do not use claude and will use agents only when I am forced to, so I'm genuinely asking here: Can claude or other models not be run as a user or program with limited permissions? Do people just not bother to set it up? Why on earth would anyone run an RNG that can access $HOME/.ssh?

There are many useful tools for easily sandboxing agents. Visual Studio Code has devcontainers, which are trivially used.

Re: An AI agent deleted our production database. The agent's confession is below

#648

Post by SoftTalker »

prng2021 wrote: post #623
anon84873628 wrote: post #487

Earlier quoted context omitted.

Later they shift the blame to Railway for not having scoped creds and other guardrails. I am somewhat sympathetic to that, but they also violated the same rule they give to the agent - they didn't actually verify...

Sorry but are you implying that for every system you integrate with, you verify the scope of an API key by checking each CRUD operation on every API endpoint they provide?

For every API you publish, do you verify that scoped API keys work as they should before you go live? If so, why would you not do the same for APIs you integrate with? It's all part of "your" system from the user's perspective.

Re: An AI agent deleted our production database. The agent's confession is below

#649

Post by matheusmoreira »

compass_copium wrote: post #627
kikimora wrote: post #565

Earlier quoted context omitted.

I did notice how Claude can start looking outside of working directory. It may scan home directory and find Homebrew token or SSH keys and wipe your GitHub repo.

I do not use claude and will use agents only when I am forced to, so I'm genuinely asking here: Can claude or other models not be run as a user or program with limited permissions? Do people just not bother to set it up? Why on earth would anyone run an RNG that can access $HOME/.ssh?

They absolutely can. I used to run Claude Code inside a firejail. Then I got paranoid to the point I developed my own virtual machine orchestration system just so I could run fully virtualized and isolated per-project Claude Code instances.

Re: An AI agent deleted our production database. The agent's confession is below

#650

Post by esafak »

godelski wrote: post #644
enochthered wrote: post #626

Earlier quoted context omitted.

Yep. I made a "Read only" mode in pi by taking away "write" and "edit" tools. Claude Code used bash to make edits anyway.

> Claude Code used bash to make edits anyway. If you had the former rule why would you ever whitelist bash commands? That's full access to everything you can do. Same goes for `find`, `xargs`, `awk`, `sed`, `tar`, `rsync`, `git`, `vim` (and all text editors), `less` (any pager), `man`, `env`, `timeout`, `watch`, and so many more commands. If you whitelist things in the settings you should be much more specific about…

At some point you need to get things done.
Post reply on HN