Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code wiped our production database with a Terraform command

twitter.com

Post reply on HN
Print view View original

21–30 of 167 posts

Re: Claude Code wiped our production database with a Terraform command

#21

Post by levkk »

SunshineTheCat wrote: post #3

Putting yourself in a situation where this could happen is kinda insane, right? Could be something I'm missing. I can't think of any specific example where I would let any agent touch a production environment, the least of which, data. AI aside, doing any major changes makes sense to do in a dev/staging/preview environment first. Not really sure what the lesson would be here. Don't punch yourself in the face repeated…

As the tool gets better, people trust it more. It's like Tesla's self-driving: "almost" works, and that's good enough for people to take their hands off the wheel, for better or for worse.

The "almost" part of automation is the issue + the marketing attached to it of course, to make it a product people want to buy. This is the expected outcome and is already priced in.

Re: Claude Code wiped our production database with a Terraform command

#22

Post by jeanlucas »

SunshineTheCat wrote: post #3

Putting yourself in a situation where this could happen is kinda insane, right? Could be something I'm missing. I can't think of any specific example where I would let any agent touch a production environment, the least of which, data. AI aside, doing any major changes makes sense to do in a dev/staging/preview environment first. Not really sure what the lesson would be here. Don't punch yourself in the face repeated…

Yep, you're not insane, they were amateur.

Re: Claude Code wiped our production database with a Terraform command

#24

Post by forgotaccount3 »

tomcatfish wrote: post #11

Despite multiple comments blaming the AI agent, I think it's the backups that are the problem here, right? With backups, almost any destructive action can be rolled back, whether it's from a dumb robot, a mistaken junior, or a sleep-deprived senior. Without, you're sort of running the clock waiting for disaster.

Yes, backups are great but a 'dumb robot' or a 'mistaken junior' shouldn't have access to prod.

And a sleep-deprived senior? Even then. They shouldn't have access to destructive effects on prod.

Maybe the senior can get broader access in a time-limited scope if senior management temporarily escalates the developers access to address a pressing production issue, but at that point the person addressing the issue shouldn't be fighting to stay awake nor lulled into a false sense of security as during day to day operations.

Otherwise it's only the release pipeline that should have permissions to take destructive actions on production and those actions should be released as part of a peer reviewed set of changes through the pipeline.

Re: Claude Code wiped our production database with a Terraform command

#25

Post by happytoexplain »

tomcatfish wrote: post #11

Despite multiple comments blaming the AI agent, I think it's the backups that are the problem here, right? With backups, almost any destructive action can be rolled back, whether it's from a dumb robot, a mistaken junior, or a sleep-deprived senior. Without, you're sort of running the clock waiting for disaster.

They are two orthogonal issues. One doesn't make the other irrelevant.

Re: Claude Code wiped our production database with a Terraform command

#27

Post by renewiltord »

I don’t use Terraform much anymore because don’t need it but that’s not how you use it.

Always forward evolve infra. Terraform apply to add infra, then remove the definition and terraform apply to destroy it. There’s no use in running terraform destroy directly on a routine basis.

Also, I assume you defined RDS snapshots also in the same state? This is clearly erroneous. It means a malformed apply human or agent results in snapshot deletion.

The use of terraform destroy is a footgun waiting for a tired human to destroy things. The lesson has nothing to do with agent.

Re: Claude Code wiped our production database with a Terraform command

#28

Post by sofixa »

levkk wrote: post #21
SunshineTheCat wrote: post #3

Putting yourself in a situation where this could happen is kinda insane, right? Could be something I'm missing. I can't think of any specific example where I would let any agent touch a production environment, the least of which, data. AI aside, doing any major changes makes sense to do in a dev/staging/preview environment first. Not really sure what the lesson would be here. Don't punch yourself in the face repeated…

As the tool gets better, people trust it more. It's like Tesla's self-driving: "almost" works, and that's good enough for people to take their hands off the wheel, for better or for worse. The "almost" part of automation is the issue + the marketing attached to it of course, to make it a product people want to buy. This is the expected outcome and is already priced in.

Exactly, Waymo were talking about this a few year back, they found that building it up gradually will not work, because people would stop paying attention when it's "almost" there, until it isn't and it crashes. So they set out on having their automation good enough to operate on its own without a human driver before starting to deploy it.

Re: Claude Code wiped our production database with a Terraform command

#29

Post by samuelknight »

One of Terraform's most powerful features that it will tell exactly which resources change before it makes the changes. The hard part is writing Terraform, not reviewing and running one command. In my workflows I am the one who runs "terraform apply", NOT the agent.

Re: Claude Code wiped our production database with a Terraform command

#30

Post by semiquaver »

I’m not going to “defend” the LLM here but this:

  > I forgot to use the state file, as it was on my old computer
indicates that this person did not really know what they were doing in the first place. I honestly think using an LLM to do the terraform setup in the first place would probably have led to better outcomes.
Post reply on HN