Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code wiped our production database with a Terraform command

twitter.com

Post reply on HN
Print view View original

111–120 of 167 posts

Re: Claude Code wiped our production database with a Terraform command

#111

Post by jkubicek »

BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

I’m going to defend Claude. If you give a robot the ability to delete production it’s going to delete production. This is 100% the user’s fault.

This problem has not yet been solved and will never be solved.

Re: Claude Code wiped our production database with a Terraform command

#112

Post by inhumantsar »

stavarotti wrote: post #45

If you found this post helpful, follow me for more content like this. I publish a weekly newsletter where I share practical insights on data and AI. It focuses on projects I'm working on + interesting tools and resources I've recently tried: https://alexeyondata.substack.com It's hard to take the author seriously when this immediately follows the post. I can only conclude that this post was for the views not anything…

what's truly incredible is that this person is selling bootcamps.

the things they "didn't realize" or "didn't know" are basics. they're things you would know if you spent any time at all with terraform or AWS.

all the remediations are table stakes. things you should at least know about before using terraform. things you would learn by skimming the docs (or at least asking Claude about best practices).

even ignoring the technical aspects, a tiny amount of consideration at any point in that process would have made it clear to any competent person that they should stop and question their assumptions.

I mean, shit happens. good engineers take down prod all the time. but damn man, to miss those basics entirely while selling courses on engineering is just astounding.

the grifter mentality is probably so deeply engrained that I'm willing to bet that they never once thought "I'm totally qualified to sell courses", let alone question the thought.

Re: Claude Code wiped our production database with a Terraform command

#113

Post by overfeed »

gos9 wrote: post #80
BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

Considering engineers have made similar mistakes I’m not so sure that’s a great razor, haha

Usually junior engineers accidentally drop dbs.

Lacking backups and staging/test environments is organizational failure: everyone who is between senior and the CTO is to blame for not fixing it post-haste.

Re: Claude Code wiped our production database with a Terraform command

#114

Post by LelouBil »

fragmede wrote: post #55
stavarotti wrote: post #45

If you found this post helpful, follow me for more content like this. I publish a weekly newsletter where I share practical insights on data and AI. It focuses on projects I'm working on + interesting tools and resources I've recently tried: https://alexeyondata.substack.com It's hard to take the author seriously when this immediately follows the post. I can only conclude that this post was for the views not anything…

It's hard to take you take seriously. His blog has a generic "read more" footer and that's a demerit worth mentioning? What do serious people in your world that write blogs do? Not want people to read their other content? In what world do you live in that writers (serious or not) don't want you to read their other work?

It's not a generic footer, it's their reply directly to their tweet about the incident.

I agree with the person you are replying to, writing a tweet like :

"How I misused AI and caused an outage"

and replying to this very tweet saying

"Here's a blog where I write insights about AI"

Obviously do not make me want to read the blog.

Re: Claude Code wiped our production database with a Terraform command

#116

Post by BitsAndObjects »

FireBeyond wrote: post #110
BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

All of those things are reasonable questions. I've also watched videos talking on using Claude's built in hooks to do everything from "never git push, only prompt me that -I- should", and beyond, "if environment variable x = y (perhaps a la DEPLOYMENT_TARGET=prod) then do not execute any command that does not have a "dry run" mode" (or do not execute any commands, only tell me what to execute)." I've also trashed pro…

Yeah the videos hyping up Claude and other such AI tools don't help matters.

For sure I've made mistakes. But I also don't write the following on my CV:

"PhD-level expert in infrastructure and trained on the entire internet of examples of what to do and what not to do; I can replace your entire infrastructure team and do everything else in your codebase too, without any review."

And yet that's how Claude is marketed. AI tools in general have been repeatedly marketed as PhD-level experts in _every_ area of information-era work, especially code. They encourage hands-off (or consent-fatigued) usage.

[Just to be clear, in case anyone wants to hire me in future: I've never accidentally deleted a production database. I've never even irrecoverably destroyed production data - nor had to rely on AWS (or another provider) to recover the data for me. I've made mistakes, mostly in sandbox environments, sometimes stressful ones in production, but nothing even close to what the OP did.]

Re: Claude Code wiped our production database with a Terraform command

#117

Post by elashri »

jkubicek wrote: post #111
BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

I’m going to defend Claude. If you give a robot the ability to delete production it’s going to delete production. This is 100% the user’s fault. This problem has not yet been solved and will never be solved .

> If you give a robot the ability to delete production it’s going to delete production

If you give an intern the ability to delete production, it's going to delete production. But to be honest you can as well replace "intern" or "robot" by human in general. Deletion in production should have safety layers that anyone cannot accidentally do it, specially without the ability of rolling back.

Re: Claude Code wiped our production database with a Terraform command

#118

Post by BitsAndObjects »

jkubicek wrote: post #111
BitsAndObjects wrote: post #67

No staging environment? No prior attempt to follow best practices (e.g. deletion protection in production)? Nor manual gating of production changes? No attempt to review Claude's actions before performing them? No management of Terraform state file? No offline backups? And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? No, I'm not working on that." - Clearly Claude wasn't p…

I’m going to defend Claude. If you give a robot the ability to delete production it’s going to delete production. This is 100% the user’s fault. This problem has not yet been solved and will never be solved .

"Anything that can go wrong, will go wrong. Including deleting production."

I'm also waiting for the day we see a "Claude sold my production database on the darkweb" post haha.

Re: Claude Code wiped our production database with a Terraform command

#119

Post by throwawaypath »

BoredPositron wrote: post #4

You wiped your production database. You actively ignored the warnings of your tooling and your backup strategy was bad. Incompetence as content is surging in the last few weeks.

>backup strategy was bad.

Backup strategy was nonexistent. There was no backup strategy.

Re: Claude Code wiped our production database with a Terraform command

#120

Post by tayo42 »

BitsAndObjects wrote: post #82
tayo42 wrote: post #78

Earlier quoted context omitted.

> And to top it off, Claude (the supposed expert tool) didn't repeatedly output "Are you insane? It did though acoording to the article and he ignored it. The Ai can only work with what you tell it.

The difference is, an expert engineer would flat-out refuse to do these things and would keep pushing back. Claude may sometimes attempt _one time_ to warn someone, and then (after consent fatigue means they're just blindly clicking "yes"), it ploughs right ahead without further complaint.

Do you really want the Ai to not do the things you tell it?

It only knows what you tell it, if you tell it risky operations are OK, what do you expect?

Post reply on HN