Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Dropped the Production Database on a Friday Night

vince.beehiiv.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: I Dropped the Production Database on a Friday Night

#31

Post by RVuRnvbM2e »

This is a good story and something everyone should experience in their career even just for the lesson in humility. That said:

> Here's the technical takeaway: Never use CASCADE deletes on critical foreign keys. Set them to NULL or use soft deletes instead. It's fine for UPDATE operations, but it's too dangerous for DELETE ones. The convenience of automatic cleanup isn't worth the existential risk of chain reactions.

What? The point of cascading foreign keys is referential integrity. If you just leave dangling references everywhere your data will either be horribly dirty or require inconsistent manual cleanup.

As I'm sure others have said: just use a test/staging environment. It isn't hard to set up even if you are in startup mode.

Re: I Dropped the Production Database on a Friday Night

#33

Post by physix »

000ooo000 wrote: post #17
munchler wrote: post #10

Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.

I cut my dev teeth in a financial institution so I'll concede I'm biased away from risk, but devving directly on the prod DB, not having a local enviroment to test changes against, and worse: literally no backups.. it screams wreckless, stupid, cheap, arrogant, and immature (in the tech sense). Nothing I'd like my name against publicly.

A colleague upgraded the production database for a securities financing settlement system on a Friday evening by accident 20 years ago.

We were devs with root access to production and no network segregation. He wanted to upgrade his dev environment, but chose the wrong resource file.

He was lucky it was a Friday, because it took us the whole weekend working round the clock to get the system and the data to a consistent state by start of trading.

We called him The Dark Destroyer thereafter.

So I would add network segregation to the mix of good ideas for production ops.

Re: I Dropped the Production Database on a Friday Night

#34

Post by grepfru_it »

000ooo000 wrote: post #17
munchler wrote: post #10

Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.

I cut my dev teeth in a financial institution so I'll concede I'm biased away from risk, but devving directly on the prod DB, not having a local enviroment to test changes against, and worse: literally no backups.. it screams wreckless, stupid, cheap, arrogant, and immature (in the tech sense). Nothing I'd like my name against publicly.

I’m building my toy project and I have an LTO drive taking backups every night. Here I am complaining that having 2Tb of backups is too much.

lol good luck op

Re: I Dropped the Production Database on a Friday Night

#35

Post by [deleted] »

wombatpm wrote: post #19
bloudermilk wrote: post #6

I dropped the production database at the first startup I worked at, three days after we went live. We were scrappy™ and didn’t have backups yet, so we lost all the data permanently. I learned that day that running automated tests on a production database isn’t a good idea!

Here is another one: Don't trust ops when they say they have backups. I asked and was told there are weekly full backups, with daily incrementals. The time came when I needed a production DB restored due to an upgrade bug in our application. That was bad - thank $DEIITY we have backups. OPS: Huh, it appears we can't find your incremental. ME: Well just restore the weekly, its only Tuesday. Two Days later. OPS:About t…

[deleted]

Re: I Dropped the Production Database on a Friday Night

#36

Post by grepfru_it »

geuis wrote: post #23

Owww. The first or second paragraph of this made me cringe "I had just finished what I thought was a clean migration: moving our entire database from our old setup to PostgreSQL with Supabase" ... on a Friday. Never do prod deploys on a Friday unless you have at least 2 people available through the weekend to resolve issues. The rest of this post isn't much better. And come one. Don't do major changes to a prod db wh…

No. Never release/upgrade on a Friday. Had too many late night weekends when I should be happily drinking beer. Never release at eod Friday. Never.

Re: I Dropped the Production Database on a Friday Night

#37

Post by grepfru_it »

booleandilemma wrote: post #32

Who is this guy? He seems like a poser. I wouldn't be surprised if these articles are AI-generated.

To be fair this was the norm 10 years ago. Just seems like he is stuck in the past. Really no excuse to provision an ec2 volume and dump all backups there. I’m not even in prod yet and have full backups to LTO to be ready for launch next month

Re: I Dropped the Production Database on a Friday Night

#40

Post by danesparza »

Arainach wrote: post #27

>Here's the technical takeaway: Never use CASCADE deletes on critical foreign keys. The technical takeaway, as others have said, is to do prod deployment during business hours when there are people around to monitor and to help recover if anything goes wrong, and where it will be working hours for quite a while in the future. Fridays are not that.

Also: don't brag about doing the opposite of what this guy says.
Post reply on HN