This is such a poorly written post, and im sure there are on-going disasters waiting to happen -- I've built 3 startups and sold 2 of them and never ever developed on production. ?? What level of crazy is this?
I Dropped the Production Database on a Friday Night
21–30 of 67 posts
Re: I Dropped the Production Database on a Friday Night
#22I'm sorry, but there's "move fast and break things" and then there's a group of junior devs not even bothering to google a checklist of development or moving to production best practices.
Your Joe AI customers should be worried. Anyone actually using the RankBid you did a Show HackerNews on 8 months ago should be worried (particularly by the "Secure by design: We partner with Stripe to ensure your data is secure." line.
If you don't want to get toasted by some future failure where you won't be accidentally saved by a vendor, then maybe start learning more on the technical side instead of researching and writing blogspam like "I Read 10 Business Books So You Don't Have To".
This might sound harsh, but it's intended as sound advice that clearly nobody else is giving you.
Re: I Dropped the Production Database on a Friday Night
#23Owww. The first or second paragraph of this made me cringe
"I had just finished what I thought was a clean migration: moving our entire database from our old setup to PostgreSQL with Supabase" ... on a Friday.
Never do prod deploys on a Friday unless you have at least 2 people available through the weekend to resolve issues.
The rest of this post isn't much better.
And come one. Don't do major changes to a prod db when critical team members have signed off for a weekend or holiday.
I'm actually quite happy OP posted their experiences. But it really needs to be a learning experience. We've all done something like this and I bet a lot of us old timers have posted similar stories.
Re: I Dropped the Production Database on a Friday Night
#24I dunno. The effort needed to ensure you have backups is tiny compared to the work done to create the product. And to pull a backup before deleting stuff in production only needs a smidgen of experience. They were extremely lucky. Imagine what the boss would have said if they hadn't managed to recover the data.
This _was_ one of the bosses.
Ah, yes.
> I immediately messaged my co-founders.
Re: I Dropped the Production Database on a Friday Night
#25The “and honestly?” phrase smells like AI writing to the point I stopped there and closed the post. Don’t fuck your database up and do have point-in-time rollbacks. No excuses it’s not hard. Not something to be proud of.
Let he who is without sin cast the first DELETE CASCADE.
Re: I Dropped the Production Database on a Friday Night
#26Developing directly on the production database with no known backups. Saved from total disaster by pure luck. Then a bunch of happy talk about it being a "small price to pay for the lessons we gained" and how such failures "unleash true creativity". It's amazing what people will self-disclose on the internet.
That's the first thing I took away. The author ignores every sane software engineering practice, is saved by pure luck and then dives into what commands not to use in supabase. Why do this? Why not spend a week or two before you launch to setup a decent ci/cd pipeline? That's the real lesson here.
Re: I Dropped the Production Database on a Friday Night
#27>Here's the technical takeaway: Never use CASCADE deletes on critical foreign keys.
The technical takeaway, as others have said, is to do prod deployment during business hours when there are people around to monitor and to help recover if anything goes wrong, and where it will be working hours for quite a while in the future. Fridays are not that.
Re: I Dropped the Production Database on a Friday Night
#28Your website title is "Profitable Programming" with a blog post "How I Dropped the Production Database on a Friday Night"
Thats not very profitable
Re: I Dropped the Production Database on a Friday Night
#29I dropped the production database at the first startup I worked at, three days after we went live. We were scrappy™ and didn’t have backups yet, so we lost all the data permanently. I learned that day that running automated tests on a production database isn’t a good idea!
Here is another one: Don't trust ops when they say they have backups. I asked and was told there are weekly full backups, with daily incrementals. The time came when I needed a production DB restored due to an upgrade bug in our application. That was bad - thank $DEIITY we have backups. OPS: Huh, it appears we can't find your incremental. ME: Well just restore the weekly, its only Tuesday. Two Days later. OPS:About t…
If your company is big enough to have dedicated ops then it should be running regular tests on backups. A disaster recovery process if you will.
At some point though its not your problem when the company is big enough. Are you gonna do everyone's job? You tell em what you need in writing and if they drop the ball its their head.
Re: I Dropped the Production Database on a Friday Night
#30Uhh, no, the answer is not to avoid cascading deletes. The answer is to not develop directly on a production database and to have even the most basic of backup strategies in place. It is not hard. Also, “on delete restrict” isn’t a bad policy either for some keys. Make deleting data difficult.
> Here's the technical takeaway: Never use CASCADE deletes on critical foreign keys. Set them to NULL or use soft deletes instead. It's fine for UPDATE operations, but it's too dangerous for DELETE ones. The convenience of automatic cleanup isn't worth the existential risk of chain reactions.
I actually agreed 100% with this learning, especially the last sentence. The younger me would write a long email to push for ON DELETE CASCADE everywhere. The older me doesn't even want to touch terraform, where an innocent looking update can end up destroying everything. I will rather live with some orphaned records and some infra drifts.
And still I got burnt few months ago, when I inadvertently triggered some internal ON DELETE CASCADE logic of Consul ACL.
(I do agree with your other points)