Live data from Hacker News

I Dropped the Production Database on a Friday Night

vince.beehiiv.com

11–20 of 67 posts

Re: I Dropped the Production Database on a Friday Night

#11
Uhh, no, the answer is not to avoid cascading deletes. The answer is to not develop directly on a production database and to have even the most basic of backup strategies in place. It is not hard.

Also, “on delete restrict” isn’t a bad policy either for some keys. Make deleting data difficult.

Re: I Dropped the Production Database on a Friday Night

#12
post #3

I hope the poster will learn about transactions at some point. Postgres even lets you alter the schema within a transaction. What I learned, once upon a time, is that with a database, you shouldn't delete data you want to keep. If you want to keep something, you use SQL's fine UPDATE to update it, you don't delete it. Databases work best if you tell them to do what you want them to do, as a single transaction.

I mean

UPDATE users SET name='test'

is still effectively a delete...

Re: I Dropped the Production Database on a Friday Night

#13

I dropped the production database at the first startup I worked at, three days after we went live. We were scrappy™ and didn’t have backups yet, so we lost all the data permanently. I learned that day that running automated tests on a production database isn’t a good idea!

> I learned that day that running automated tests on a production database isn’t a good idea!

There's novel lessons to be learned in tech all the time.

This is not one of them.

Re: I Dropped the Production Database on a Friday Night

#14

Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.

Yeah. Imagine everything else that's completely wrong in that app.

Re: I Dropped the Production Database on a Friday Night

#15
post #7

I dunno. The effort needed to ensure you have backups is tiny compared to the work done to create the product. And to pull a backup before deleting stuff in production only needs a smidgen of experience. They were extremely lucky. Imagine what the boss would have said if they hadn't managed to recover the data.

This _was_ one of the bosses.

Re: I Dropped the Production Database on a Friday Night

#16
post #13

I dropped the production database at the first startup I worked at, three days after we went live. We were scrappy™ and didn’t have backups yet, so we lost all the data permanently. I learned that day that running automated tests on a production database isn’t a good idea!

> I learned that day that running automated tests on a production database isn’t a good idea! There's novel lessons to be learned in tech all the time. This is not one of them.

Learn lessons from other people. You can't learn them all yourself.

Re: I Dropped the Production Database on a Friday Night

#17

Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.

I cut my dev teeth in a financial institution so I'll concede I'm biased away from risk, but devving directly on the prod DB, not having a local enviroment to test changes against, and worse: literally no backups.. it screams wreckless, stupid, cheap, arrogant, and immature (in the tech sense). Nothing I'd like my name against publicly.

Re: I Dropped the Production Database on a Friday Night

#18

The “and honestly?” phrase smells like AI writing to the point I stopped there and closed the post. Don’t fuck your database up and do have point-in-time rollbacks. No excuses it’s not hard. Not something to be proud of.

Yeah, the whole thing is full of AI-isms. Started skimming and every other sentence has one.

"Picture this: Panic mode activated. You heard that right. But here's what surprised me the most" and so on. Ugh.

Re: I Dropped the Production Database on a Friday Night

#19

I dropped the production database at the first startup I worked at, three days after we went live. We were scrappy™ and didn’t have backups yet, so we lost all the data permanently. I learned that day that running automated tests on a production database isn’t a good idea!

Here is another one: Don't trust ops when they say they have backups. I asked and was told there are weekly full backups, with daily incrementals. The time came when I needed a production DB restored due to an upgrade bug in our application. That was bad - thank $DEIITY we have backups.

OPS: Huh, it appears we can't find your incremental.

ME: Well just restore the weekly, its only Tuesday.

Two Days later.

OPS:About that backup. Turns out it's a backup of the servers, not the database. We'll have to restore to new VM's in order to get at the data.

ME: How did this happen?

OPS: Well the backups work for MSSQL Server.

ME: This is PostgreSQl.

OPS: Yeah, apparently we started setting that up but never finished.

ME: You realize we have about 20 applications using that database?

OPS: Now we do.

Lesson: Until you personally have seen a successful restore from backup, you do not have backups. You have hopes and prayers that you have backups. I am forever in the Trust but Verify camp.

Re: I Dropped the Production Database on a Friday Night

#20

Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.

I cut my dev teeth in a financial institution so I'll concede I'm biased away from risk, but devving directly on the prod DB, not having a local enviroment to test changes against, and worse: literally no backups.. it screams wreckless, stupid, cheap, arrogant, and immature (in the tech sense). Nothing I'd like my name against publicly.

[deleted]
Post reply on HN