Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Dropped the Production Database on a Friday Night

vince.beehiiv.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: I Dropped the Production Database on a Friday Night

#51

Post by vincejos »

geuis wrote: post #23

Owww. The first or second paragraph of this made me cringe "I had just finished what I thought was a clean migration: moving our entire database from our old setup to PostgreSQL with Supabase" ... on a Friday. Never do prod deploys on a Friday unless you have at least 2 people available through the weekend to resolve issues. The rest of this post isn't much better. And come one. Don't do major changes to a prod db wh…

It's hard to have 2 people available when you have a 2 people tech team. We were very early back then, MVP stage.

Re: I Dropped the Production Database on a Friday Night

#52

Post by vincejos »

codesnik wrote: post #46
oulu2006 wrote: post #21

This is such a poorly written post, and im sure there are on-going disasters waiting to happen -- I've built 3 startups and sold 2 of them and never ever developed on production. ?? What level of crazy is this?

supabase kiinda pushes you in that direction though.

I agree. They also push you not to git migrations at first, which is definitely not the best practice.

Re: I Dropped the Production Database on a Friday Night

#53

Post by vincejos »

Arnt wrote: post #3

I hope the poster will learn about transactions at some point. Postgres even lets you alter the schema within a transaction. What I learned, once upon a time, is that with a database, you shouldn't delete data you want to keep. If you want to keep something, you use SQL's fine UPDATE to update it, you don't delete it. Databases work best if you tell them to do what you want them to do, as a single transaction.

I use transactions all the time for my other projects and I've read the great Designing Data Intensive Applications which cover the topic of linearization in depth.

Re: I Dropped the Production Database on a Friday Night

#55

Post by scott_w »

citizenpaul wrote: post #29
wombatpm wrote: post #19

Earlier quoted context omitted.

Here is another one: Don't trust ops when they say they have backups. I asked and was told there are weekly full backups, with daily incrementals. The time came when I needed a production DB restored due to an upgrade bug in our application. That was bad - thank $DEIITY we have backups. OPS: Huh, it appears we can't find your incremental. ME: Well just restore the weekly, its only Tuesday. Two Days later. OPS:About t…

If your company is big enough to have dedicated ops then it should be running regular tests on backups. A disaster recovery process if you will. At some point though its not your problem when the company is big enough. Are you gonna do everyone's job? You tell em what you need in writing and if they drop the ball its their head.

It’s relative. No, I’m not sitting on the shoulder of the team that manages that (nor should I, there’d be 40 EMs bothering them!) but I fully expect my CTO has done it. And if not? Well, one day it’ll blow up and I’m looking for another job but that’s no different to any other possible major issues.

Re: I Dropped the Production Database on a Friday Night

#56

Post by scott_w »

vincejos wrote: post #54
lawgimenez wrote: post #39

Did I read that correctly? They’re on Supabase’ free plan in production? We’re just getting started and we’re even in Supabase’ paid plan.

Why do you take the paid plan when getting started?

Once you’re at a point where some of your business depends on it, you probably want the things like backups they provide…

Re: I Dropped the Production Database on a Friday Night

#57

Post by scott_w »

b0a04gl wrote: post #44

this is exactly how you earn your prod stripes. dropped the db on day 3? good. now you’re officially a backend engineer. no backups? perfect. now you'll never forget to set one up again. friday night? even better. you got the full rite of passage. people act like this's rare. it’s not. half of us have nuked prod, the other half are lying or haven't been given prod access yet. you’re fine. just make the checklist long…

Dropping DB on day 3 of your business? Probably fine. Dropping it on your day 3 but on day 300 of your business when you have paying customers? Seriously?

Re: I Dropped the Production Database on a Friday Night

#58

Post by vincejos »

scott_w wrote: post #56
vincejos wrote: post #54

Earlier quoted context omitted.

Why do you take the paid plan when getting started?

Once you’re at a point where some of your business depends on it, you probably want the things like backups they provide…

Definitely! I had just finished the migration back then so that's why we were still on the free plan, but we had planned on enabling even PITR

Re: I Dropped the Production Database on a Friday Night

#59

Post by vincejos »

Arainach wrote: post #27

>Here's the technical takeaway: Never use CASCADE deletes on critical foreign keys. The technical takeaway, as others have said, is to do prod deployment during business hours when there are people around to monitor and to help recover if anything goes wrong, and where it will be working hours for quite a while in the future. Fridays are not that.

When you are a 3 people startup, I'd argue there is no such thing as "business hours". I worked every day back then. I'll concede that the "Friday Night" part in the title might be a bit clickbait to that regard.

Re: I Dropped the Production Database on a Friday Night

#60

Post by vincejos »

heroprotagonist wrote: post #22

I'm sorry, but there's "move fast and break things" and then there's a group of junior devs not even bothering to google a checklist of development or moving to production best practices. Your Joe AI customers should be worried. Anyone actually using the RankBid you did a Show HackerNews on 8 months ago should be worried (particularly by the "Secure by design: We partner with Stripe to ensure your data is secure." li…

Thanks for the feedback, I really appreciate it. Rankbid and other projects I've made, I built from scratch myself. They have strong, solid, technical foundations. Try them for yourself, even try to hack them if you want if it proves my point.

This was not the case of Joe AI. I joined later in the project, and the foundations where even weaker than what is shown in this newsletter (no API endpoint authentication whatsoever, open bar, for example) and so I had to secure and migrate everything myself when I joined them. This was what the Supabase migration was trying to accomplish. Before I joined, they didn't even have a database but I won't get into the details here.

Before Rankbid, and the other products I've built, I've worked at a B2C startup with millions of users and never caused a big outage there, I've been programming for more than ten years, and I have a double degree in computer science, and while I agree with what "should be done" in theory for production level apps, sometimes, you need to move very fast to build great startups. I've read many technical books in my life such as Designing Data Intensive Applications, High Performance Browser Networking. I know the theory, but sometimes you just don't have the time to do everything perfectly. That's what I try to expose in this blog post. I also wanted to share a humbling experience. Everyone makes mistakes, and I'm not ashamed of making some, even after years of software engineering.

My newsletter is about the intersection of programming and business. You might not find the "business" part interesting which is fine, but I think what you call blogspam has real value for engineers who have never sold before in their life and want to learn the ropes. I spend a lot of time writing each edition, because I try to respect the time of my readers as much as possible to deliver some actual insights (even if there is a bit of fluff or story telling sometimes).

And for Joe AI: it has since become much more secure, and is progressively implementing engineering best practices, so customers don't have to worry.

Post reply on HN