At the most basic level, even just not letting Claude run terraform apply would've solved this issue. Review the god damn plan first! This is like engineering 101
Claude Code wiped our production database with a Terraform command
101–110 of 167 posts
Re: Claude Code wiped our production database with a Terraform command
#102I suspect we need to build MCP servers that prevent destructive commands.
For example, we need a "bash" tool doesn't invoke /usr/bin executables directly.
The agent should think it is invoking a unix command but those commands are proxies that prevent destructive operations with no ability for an agent to circumvent the restrictions. If there isn't a MCP server for your specific setup/need, building one just for your need should be your first step.
Re: Claude Code wiped our production database with a Terraform command
#103These stories make me feel better for pushing a bug into production "that one time".
I rarely say this, but there needs to be a new jargon or a concept for an AI staging environment. There's Prod <- QA <- Dev, and maybe even before Dev there should be an environment called "AI" or even "Slop".
Re: Claude Code wiped our production database with a Terraform command
#104No. YOU wiped you production database. YOU failed to have adequate backups. YOU put Claude Code forward as responsible but it’s just a tool. YOU are responsible, not “the AI did it!”
"AI did it" is the new "Dog ate my homework". You are blaming someone/something else for your own failures.
Re: Claude Code wiped our production database with a Terraform command
#105So even if you delete everything and make sure to keep no backups, amazon can still recover the db. What am i missing here?
They have their own backups to protect user data if they fuck something up. I'm guessing those backups were useful in this case. Hopefully they're being pruned after some time, but I don't know.
Re: Claude Code wiped our production database with a Terraform command
#106> Founder @DataTalksClub | Teaching engineers to build production AI systems | AI agents, LLMs, ML, data engineering | 100,000+ learners
Dear lord, imagine this guy teaching you how to build anything in production...
Re: Claude Code wiped our production database with a Terraform command
#107Earlier quoted context omitted.
Considering engineers have made similar mistakes I’m not so sure that’s a great razor, haha
Usually engineers who have not recently been trained on well documented examples of what to do and what not to do and the consequences ;) (Yes, I chose the word "trained" intentionally)
So what I hear is after this makes the training set, Claude Code might get a promotion from junior to level 1?
Re: Claude Code wiped our production database with a Terraform command
#108s/Claude Code/unsupervised intern/ and it's the same story, except people might have more sympathy (for the intern).
Do you really think people would have more sympathy for an org that gave the keys to the kingdom to an intern? I think it would be the same "How did you think this was going to go?" conversation.
Sympathy for the intern. Like the HBO test email incident.
Re: Claude Code wiped our production database with a Terraform command
#109Re: Claude Code wiped our production database with a Terraform command
#110No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…
All of those things are reasonable questions. I've also watched videos talking on using Claude's built in hooks to do everything from "never git push, only prompt me that -I- should", and beyond, "if environment variable x = y (perhaps a la DEPLOYMENT_TARGET=prod) then do not execute any command that does not have a "dry run" mode" (or do not execute any commands, only tell me what to execute)."
I've also trashed production by "hand" in my previous time as an SRE.
> If the latter, then you cannot defend Claude's failure to act like a senior engineer in this situation.
This is rather black and white. Is it acceptable? No. Is it to be expected of a senior engineer? Yes, at times. If you have any length of career as an engineer or ops person and you tell me that you've never executed problematic commands whether or not caught by security nets, bluntly, you're lying.