Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code wiped our production database with a Terraform command

twitter.com

Post reply on HN
Print view View original

91–100 of 167 posts

Re: Claude Code wiped our production database with a Terraform command

#91

Post by fragmede »

JoBrad wrote: post #48
forgotaccount3 wrote: post #24

Earlier quoted context omitted.

Yes, backups are great but a 'dumb robot' or a 'mistaken junior' shouldn't have access to prod. And a sleep-deprived senior? Even then. They shouldn't have access to destructive effects on prod. Maybe the senior can get broader access in a time-limited scope if senior management temporarily escalates the developers access to address a pressing production issue, but at that point the person addressing the issue should…

If a sleep-deprived senior shouldn’t have access to prod, I think we have big problems, frankly.

Which, if you're Google-sized, you have follow-the-sun rotations, in order to avoid that problem. But what about the rest of the class?

Re: Claude Code wiped our production database with a Terraform command

#92

Post by tomcatfish »

happytoexplain wrote: post #25
tomcatfish wrote: post #11

Despite multiple comments blaming the AI agent, I think it's the backups that are the problem here, right? With backups, almost any destructive action can be rolled back, whether it's from a dumb robot, a mistaken junior, or a sleep-deprived senior. Without, you're sort of running the clock waiting for disaster.

They are two orthogonal issues. One doesn't make the other irrelevant.

I agree that a second issue doesn't erase the first, but also I've got enough work experience to know that a system which can be brought down by 1 person no matter the tooling they use is a system not destined to last for long.

Re: Claude Code wiped our production database with a Terraform command

#95

Post by jinko-niwashi »

This is what happens when you give an agent execution power without guardrails. The tool isn't the problem — the absence of governance is. In my setup I treat the AI as a junior dev with root access: every destructive operation requires explicit human approval, and the session context includes hard constraints on what it can and can't touch.

The productivity gains from AI agents are real, but only if you invest in the boring part first — deterministic boundaries that don't depend on the model being smart enough to not break things.

Re: Claude Code wiped our production database with a Terraform command

#96

Post by BitsAndObjects »

petcat wrote: post #41
xmodem wrote: post #16

An engineer recklessly ran untrusted code directly in a production environment. And then told on himself on Twitter.

From the article, it sounds like that engineer did a lot of other reckless things even before handing the tasks over to the AI agent to continue the recklessness with even more abandon. This is a case study in "if you don't know what you're doing, the answer is not just to hand it over to some AI bot to do it for you." The answer is to hire a professional. That is if you care about your data, or even just your reputa…

"To err is human; To really foul things up requires a computer."

Extended with: "To really foul things up quickly, requires an AI tool."

Re: Claude Code wiped our production database with a Terraform command

#97

Post by yearolinuxdsktp »

Your AWS backup snapshots must go one-way (append-only) to a separate AWS account, to which access is extremely limited and never has any automated tools connecting to with anything other than read access. I don’t think it costs more to do that—-but it takes your backups out of the blast radius of a root or admin account compromise OR a tool malfunction. With AWS DLM, you can safely configure your backup retention in the separate AWS account and not risk any tools deleting them.

Terraform is a ticking time bomb. All it takes is for a new field to show up in AWS or a new state in an existing field, and now your resource is not modified, but is destroyed and re-created.

I will never trust any process, AI or a CD pipeline, execute `terraform apply` automatically on anything production. Maybe if you examine the plan for a very narrow set of changes and then execute apply from that plan only, maybe then you can automate it. I think it’s much rarer for Terraform to deviate from a plan.

Regardless, you must always turn on Delete Protection on all your important resources. It is wild to me that AWS didn’t ship EKS with delete protection out of the gate—-they only added this feature in August 2025! Not long before that, I’ve witnessed a production database get deleted because Terraform decided that an AWS EKS cluster could not be modified, so it decided to delete it and re-create it, while the team was trying to upgrade the version of EKS. The same exact pipeline worked fine in the staging environment. Turns out production had a slight difference due to AWS API changes, and Terraform decided it could not modify.

The use of a state file with Terraform is a constant source of trouble and footguns:

- you must never use a local Terraform state file for production that’s not committed to source control - you must use a remote S3 state file with Terraform for any production system that’s worth anything - ideally, the only state file in source control is for a separate Terraform stack to bootstrap the S3 bucket for all other Terraform stacks

If you’re running only on AWS, and are using agents to write your IaaC anyway, use AWS CloudFormation, because it doesn’t use state files, and you don’t need your IaaC code to be readable or comprehensible.

Re: Claude Code wiped our production database with a Terraform command

#98

Post by fragmede »

QuantumGood wrote: post #76
BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

Marketing "protect your business from harm by Claude internally" seems to be a growth industry.

And they said Claude was gonna take our jobs.

Re: Claude Code wiped our production database with a Terraform command

#99

Post by rkuodys »

andrewstuart wrote: post #65

No. YOU wiped you production database. YOU failed to have adequate backups. YOU put Claude Code forward as responsible but it’s just a tool. YOU are responsible, not “the AI did it!”

My feeling was the same reading. If you give a task to a junior and he wipes out a database in production - it is not a fault of a Junior, it is your own fault that he was able to do it.

Re: Claude Code wiped our production database with a Terraform command

#100

Post by QuercusMax »

charcircuit wrote: post #42
forgotaccount3 wrote: post #24

Earlier quoted context omitted.

Yes, backups are great but a 'dumb robot' or a 'mistaken junior' shouldn't have access to prod. And a sleep-deprived senior? Even then. They shouldn't have access to destructive effects on prod. Maybe the senior can get broader access in a time-limited scope if senior management temporarily escalates the developers access to address a pressing production issue, but at that point the person addressing the issue should…

But smart robots like Claude should and will have access to production. There has to be something figured out on how to make sure operation remains smooth. The argument of don't do that will not be a viable position to hold long term. Keeping a human in the loop is not necessary.

Nobody, not even a "smart robot" should have unfettered read-write production access without guardrails. Read-only? Sure - that's a totally different story.

Read-write production access without even the equivalent of "sudo" is just insane and asking for trouble.

Post reply on HN