Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code wiped our production database with a Terraform command

twitter.com

Post reply on HN
Print view View original

141–150 of 167 posts

Re: Claude Code wiped our production database with a Terraform command

#141

Post by tapoxi »

BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

The user's bio is literally "Teaching engineers to build production AI systems"

It would be funny if these LinkedIn/Twitter influencers weren't so widespread.

Re: Claude Code wiped our production database with a Terraform command

#142

Post by akouri »

tapoxi wrote: post #141
BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

The user's bio is literally "Teaching engineers to build production AI systems" It would be funny if these LinkedIn/Twitter influencers weren't so widespread.

Well, he definitely taught me what not to do

Re: Claude Code wiped our production database with a Terraform command

#143

Post by zamalek »

Friendly reminder: most cloud providers have deletion locks. Go and enable them on your prod dbs right now.

Sure, Claude could just remove the lock - but it's one more gate.

Edit: these existed long before agents, and for good reason: mistakes happen. Last week I removed tf destroy from a GitHub workflow, because it was 16px away from apply in the dropdown. Lock your dbs, irrespective of your take on agents.

Re: Claude Code wiped our production database with a Terraform command

#144

Post by tayo42 »

BitsAndObjects wrote: post #139
tayo42 wrote: post #120

Earlier quoted context omitted.

Do you really want the Ai to not do the things you tell it? It only knows what you tell it, if you tell it risky operations are OK, what do you expect?

That depends. As per my root comment, if you ignore a lot of the marketing of AI and view it as just a tool, then I agree with your point about it doing what you tell it but I still want the tool to help me avoid making mistakes (and I’d like it to work quite hard at that - much harder, it seems, than it currently does). And probably to the extent that it refuses to run dangerous commands for me and tells me to copy/…

An experienced engineer still gets decisions overridden all of the time and has to suck it up or get fired.

Re: Claude Code wiped our production database with a Terraform command

#146

Post by abustamam »

charcircuit wrote: post #126
abustamam wrote: post #124

Earlier quoted context omitted.

> AI can much faster troubleshoot and resolve production issues than humans Can, yes, with proper guardrails. The problem is that it seems like every team is learning this the hard way. It'd be great to have a magical robot that could magically solve all our problems without the risk of it wrecking everything. But most teams aren't there yet and to suggest that it's THE way to go without the nuances of "btw it could…

It didn't delete the prod db on its own a human introduced such error, and if there were backups it could fix such a mistake.

There were backups. The AI deleted them.

Re: Claude Code wiped our production database with a Terraform command

#147

Post by BitsAndObjects »

tayo42 wrote: post #144
BitsAndObjects wrote: post #139

Earlier quoted context omitted.

That depends. As per my root comment, if you ignore a lot of the marketing of AI and view it as just a tool, then I agree with your point about it doing what you tell it but I still want the tool to help me avoid making mistakes (and I’d like it to work quite hard at that - much harder, it seems, than it currently does). And probably to the extent that it refuses to run dangerous commands for me and tells me to copy/…

An experienced engineer still gets decisions overridden all of the time and has to suck it up or get fired.

True.. though an experienced engineer would also risk getting fired for doing all the other stuff the OP did too. Especially if they made minimal attempts to highlight consequences/outcomes to management in advance..

Re: Claude Code wiped our production database with a Terraform command

#148

Post by semiquaver »

BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

The narrative incudes this:

  > Claude was trying to talk me out of [reusing an existing AWS account for an unrelated project], saying I should keep it separate, but I wanted to save a bit
So in a very real sense the LLM did object to this and OP insisted. If Claude had objected to the more specific step that deleted the DB, it seems likely OP would also have pushed past the objection.

Re: Claude Code wiped our production database with a Terraform command

#149

Post by BitsAndObjects »

semiquaver wrote: post #148
BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

The narrative incudes this: > Claude was trying to talk me out of [reusing an existing AWS account for an unrelated project], saying I should keep it separate, but I wanted to save a bit So in a very real sense the LLM did object to this and OP insisted. If Claude had objected to the more specific step that deleted the DB, it seems likely OP would also have pushed past the objection.

An expert would’ve at least taken a backup or checked existing backups weren’t going to be destroyed. Silently - without asking their manager - they just do defensive engineering as a good practice. Or they would’ve, at minimum, highlighted the suggestion, which doesn’t seem to have happened in this case. As someone who recently did a short term contract to capture manually created AWS infrastructure into CDK, I can tell you this was one of my first moves!

So, Claude as a tool: sure, this is user error. Claude could be improved by making it suggest defensive steps and making it push harder for the user to do them first, but it’s still down to the user. I’ve repeatedly encountered this issue that Claude doesn’t plan for engineering - it just plans to code - even with Claude.md and skills and such.

Claude as a replacement for engineers? Well, yeah, the marketing is just that: marketing.

Re: Claude Code wiped our production database with a Terraform command

#150

Post by charcircuit »

abustamam wrote: post #146
charcircuit wrote: post #126

Earlier quoted context omitted.

It didn't delete the prod db on its own a human introduced such error, and if there were backups it could fix such a mistake.

There were backups. The AI deleted them.

When people talk about backups they typically mean located somewhere else. If one terraform command can take out the db and the backups then those backups aren't really separate. It's like using RAID as a backup. Sure it may help, but there are cases where you can lose everything.
Post reply on HN