Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Dropped the Production Database on a Friday Night

vince.beehiiv.com

Post reply on HN
Print view View original

61–67 of 67 posts

Re: I Dropped the Production Database on a Friday Night

#61

Post by vincejos »

pton_xd wrote: post #18
peterldowns wrote: post #8

The “and honestly?” phrase smells like AI writing to the point I stopped there and closed the post. Don’t fuck your database up and do have point-in-time rollbacks. No excuses it’s not hard. Not something to be proud of.

Yeah, the whole thing is full of AI-isms. Started skimming and every other sentence has one. "Picture this: Panic mode activated. You heard that right. But here's what surprised me the most" and so on. Ugh.

I tried to have a conversational, story-telling style, maybe that's why you think there are lots of "AI-isms". But I take this as a feedback for the next editions: less fluff, more straight-to-the point writing. Thanks!

Re: I Dropped the Production Database on a Friday Night

#62

Post by kiwicopple »

vincejos wrote: post #52
codesnik wrote: post #46

Earlier quoted context omitted.

supabase kiinda pushes you in that direction though.

I agree. They also push you not to git migrations at first, which is definitely not the best practice.

(Supabase ceo)

We do not push devs not to do migrations - we would strongly prefer if everyone used migrations and declarative schemas.

Especially at the scale that OP is at (see maturity model: https://supabase.com/docs/guides/deployment/maturity-model)

Re: I Dropped the Production Database on a Friday Night

#63

Post by vincejos »

kiwicopple wrote: post #62
vincejos wrote: post #52

Earlier quoted context omitted.

I agree. They also push you not to git migrations at first, which is definitely not the best practice.

(Supabase ceo) We do not push devs not to do migrations - we would strongly prefer if everyone used migrations and declarative schemas. Especially at the scale that OP is at (see maturity model: https://supabase.com/docs/guides/deployment/maturity-model )

While I don’t question the maturity model in itself (which I read after the incident and that’s why I started gitting migrations just after), I realized it was harder than other Supabase features for it to work well, especially when you start working with other features than just authentication and Postgres.

In particular, webhooks and triggers don’t work out of the box. So maybe it’s not pushing in a particular direction but at least I’d argue it’s not nudging you to do it because it entails some hours of custom setup and debugging before the CLI commands like supabase db diff actually work as intended in my experience. But I know the Supabase team is improving it every release so I’m thankful for this work!

Re: I Dropped the Production Database on a Friday Night

#64

Post by Arnt »

XorNot wrote: post #12
Arnt wrote: post #3

I hope the poster will learn about transactions at some point. Postgres even lets you alter the schema within a transaction. What I learned, once upon a time, is that with a database, you shouldn't delete data you want to keep. If you want to keep something, you use SQL's fine UPDATE to update it, you don't delete it. Databases work best if you tell them to do what you want them to do, as a single transaction.

I mean UPDATE users SET name='test' is still effectively a delete...

Only as a matter of low level storage. It won't trigger ON DELETE CASCADE and that kind of thing.

This is a kind of misunderstanding I've heard from others who were first exposed to hacky things like early mysql. Databases are something else. A different kind of beast. If you use a database, and Postgres is the best of the DBMSes, then you can say things like "a lead shouldn't be deleted before three months have passed, no matter what" or "a lead can't be deleted until its state column says it's been handled" and the DBMS will make sure of it. If you have a bug that would involve leads being deleted prematurely, the DBMS will reject your change. Your change just won't break the database.

Re: I Dropped the Production Database on a Friday Night

#65

Post by wombatpm »

citizenpaul wrote: post #29
wombatpm wrote: post #19

Earlier quoted context omitted.

Here is another one: Don't trust ops when they say they have backups. I asked and was told there are weekly full backups, with daily incrementals. The time came when I needed a production DB restored due to an upgrade bug in our application. That was bad - thank $DEIITY we have backups. OPS: Huh, it appears we can't find your incremental. ME: Well just restore the weekly, its only Tuesday. Two Days later. OPS:About t…

If your company is big enough to have dedicated ops then it should be running regular tests on backups. A disaster recovery process if you will. At some point though its not your problem when the company is big enough. Are you gonna do everyone's job? You tell em what you need in writing and if they drop the ball its their head.

The majority of our apps were Java, running Tomcat on windows server using MSSQL or oracle. That was tested as part of DR. Our Linux servers running Python and Postgres were not as high a priority apparently.

The lack of working backups made it a problem because if assurances and certifications we were required to maintain.

When starting a new project I now request a dev database with a dump from prod more than 30 days ago just to see the process work. Does it waste their time? Maybe. In which case it encourages more automation. Do I care no? But I am not getting burned again.

Re: I Dropped the Production Database on a Friday Night

#66

Post by ghushn3 »

doubled112 wrote: post #16
ghushn3 wrote: post #13

Earlier quoted context omitted.

> I learned that day that running automated tests on a production database isn’t a good idea! There's novel lessons to be learned in tech all the time. This is not one of them.

Learn lessons from other people. You can't learn them all yourself.

Yes, this is what I'm saying.

Re: I Dropped the Production Database on a Friday Night

#67

Post by cranberryturkey »

grepfru_it wrote: post #38
cranberryturkey wrote: post #2

i dropped the dev database once at PayPal back in 2006

I once remailed emails to IEEE and ACM. I was ready to quit and take the L for such a bad mistake. Not write a blog post for Friday evening consumption

Haha yeah I wasn’t gonna blog about my f up
Post reply on HN