Live data from Hacker News

I Dropped the Production Database on a Friday Night

vince.beehiiv.com

41–50 of 67 posts

Re: I Dropped the Production Database on a Friday Night

#41

Who is this guy? He seems like a poser. I wouldn't be surprised if these articles are AI-generated.

To be fair this was the norm 10 years ago. Just seems like he is stuck in the past. Really no excuse to provision an ec2 volume and dump all backups there. I’m not even in prod yet and have full backups to LTO to be ready for launch next month

This was never the norm for successful companies. This is only the norm for cowboys who have more pizza than good sense.

Re: I Dropped the Production Database on a Friday Night

#42
Echoing the other comments about just how bad the setup here is. Setting up staging/dev environments does not take so much time as to put you behind your competition. There's a vast, VAST chasm between "We're testing on the prod DB with no backups" and the dreaded guardrails and checkboxes.

That being said, I would love to see more resources about incident management for small teams and how to strike this balance. I'm the only developer working on a (small, but somehow super political/knives-out) company's big platform with large (F500) clients and a mandate-from-heaven to rapidly add features -- and it's by far the most stressed out I've ever been in my career if not life. Every incident, whether it be the big GCP outage from last week or a database crash this week, leads to a huge mental burden that I have no idea how to relieve, and a huge passive-aggressive political shitstorm I have no idea how to navigate.

Re: I Dropped the Production Database on a Friday Night

#43

Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.

Right?! This whole post is kinda absurd. It has the feel of a kid putting a fork into an outlet, getting the shock of a lifetime and then going “and thanks to this, everyone in my household now knows not to put a fork into an outlet.” You didn’t have to go through all this to figure out that you need backups. The fluff is the cherry on top

Re: I Dropped the Production Database on a Friday Night

#44
this is exactly how you earn your prod stripes. dropped the db on day 3? good. now you’re officially a backend engineer.

no backups? perfect. now you'll never forget to set one up again. friday night? even better. you got the full rite of passage.

people act like this's rare. it’s not. half of us have nuked prod, the other half are lying or haven't been given prod access yet.

you’re fine. just make the checklist longer next time. and maybe alias `drop` to `echo "no"` for a while

Re: I Dropped the Production Database on a Friday Night

#45

This is a good story and something everyone should experience in their career even just for the lesson in humility. That said: > Here's the technical takeaway: Never use CASCADE deletes on critical foreign keys. Set them to NULL or use soft deletes instead. It's fine for UPDATE operations, but it's too dangerous for DELETE ones. The convenience of automatic cleanup isn't worth the existential risk of chain reactions.…

> The point of cascading foreign keys is referential integrity.

Not quite. Databases can enforce referential integrity through foreign keys, without cascading deletes being enabled.

“On delete restrict” vs “on delete cascade” still enforces referential integrity, and is typically a better way to avoid the OP’s issue.

Re: I Dropped the Production Database on a Friday Night

#46

This is such a poorly written post, and im sure there are on-going disasters waiting to happen -- I've built 3 startups and sold 2 of them and never ever developed on production. ?? What level of crazy is this?

supabase kiinda pushes you in that direction though.

Re: I Dropped the Production Database on a Friday Night

#47

Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.

Right?! This whole post is kinda absurd. It has the feel of a kid putting a fork into an outlet, getting the shock of a lifetime and then going “and thanks to this, everyone in my household now knows not to put a fork into an outlet.” You didn’t have to go through all this to figure out that you need backups. The fluff is the cherry on top

Maybe the post is an attempt to save face in front of his colleagues. Owning up to the mistake and listing lessons learned.

Re: I Dropped the Production Database on a Friday Night

#49

This is a good story and something everyone should experience in their career even just for the lesson in humility. That said: > Here's the technical takeaway: Never use CASCADE deletes on critical foreign keys. Set them to NULL or use soft deletes instead. It's fine for UPDATE operations, but it's too dangerous for DELETE ones. The convenience of automatic cleanup isn't worth the existential risk of chain reactions.…

Thanks for your takeaway. Yes the dev environment is definitely a must as soon as you start growing!

Re: I Dropped the Production Database on a Friday Night

#50

Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.

While I agree with everything said here about making backups etc. and which I have done in my career at later stage companies, when you are just starting out and building MVPs, I'd argue (as I do in the newsletter) that losing 2 weeks to setup CI/CDs pipelines and backups before you can pay the rent is a waste of time! I was a Supabase noob back then so I had not explored their features for local development, which is the learning I try to share in this post.
Post reply on HN