Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero downtime Postgres migration, done right

engineering.theblueground.com

Post reply on HN
Print view View original

31–40 of 135 posts

Re: Zero downtime Postgres migration, done right

#31

Post by yupper32 »

endisneigh wrote: post #26
yupper32 wrote: post #24

Earlier quoted context omitted.

I find it very hard to come up with a use case where a weekly 6 hours of downtime at night EST would be acceptable. US government websites already often do this, and I find that completely unacceptable since these services need to be available to anyone, regardless of work and life schedules. Any website that's international would also suffer greatly for an EST centralized scheduled downtime. Maybe a very localized w…

Why wouldn’t it be acceptable? People aren’t available all the time either, nor are stores. In fact software is unique in its availability.

Frankly, because it can be. Weekly scheduled downtime is arbitrary, manufactured, and lazy.

Of course, having to schedule the occasional downtime for a database migration is fine. There's probably a few times a year that you'd need to do it if you don't have the bandwidth to do fancy zero-downtime solutions. It's the weekly arbitrary downtime that I'm firmly against.

Re: Zero downtime Postgres migration, done right

#32

Post by endisneigh »

yupper32 wrote: post #31
endisneigh wrote: post #26

Earlier quoted context omitted.

Why wouldn’t it be acceptable? People aren’t available all the time either, nor are stores. In fact software is unique in its availability.

Frankly, because it can be. Weekly scheduled downtime is arbitrary, manufactured, and lazy. Of course, having to schedule the occasional downtime for a database migration is fine. There's probably a few times a year that you'd need to do it if you don't have the bandwidth to do fancy zero-downtime solutions. It's the weekly arbitrary downtime that I'm firmly against.

Why is it lazy? When things are up people have to work. Do you believe people should be working all of the time?

I think regular downtime is only natural. If you had to choose between 95% availability or 100% availability other than the before mentioned downtime which would you choose?

Re: Zero downtime Postgres migration, done right

#33

Post by yifanl »

maccard wrote: post #27
endisneigh wrote: post #22

I wonder how much easier software engineering would be if there were a period where things are simply not available. What problems are currently very difficult would be made trivial if 6 hours of downtime every Sunday were acceptable? 10PM-4AM EST

Your team is now working 10pm to 4am est every Sunday, and anyone who uses your services in any time zone around the world is without your service. 10pm est is lunch time in Australia. 4am est is working hours in many parts of western Europe. You don't have to be huge to have customers all around the world.

I think the idea is that this world is one where the customer accepts that there may be an outage every Sunday because feature work can be done far more cheaply.

Re: Zero downtime Postgres migration, done right

#34

Post by ikiris »

fullstop wrote: post #21
chefkoch wrote: post #15

Earlier quoted context omitted.

Then hopefully everything is HA in your company.

It absolutely is, BGP with geographically diverse paths, databases, app servers, etc, are all redundant. It's hosted in-house, so there is a cold standby database in AWS which would only be used if, say, an aircraft crashed into our server rooms.

It always amazes me when someone says they can't ever be down, and then says they only serve from one physical location actively.

One of these things is not like the other.

Re: Zero downtime Postgres migration, done right

#35

Post by endisneigh »

maccard wrote: post #27
endisneigh wrote: post #22

I wonder how much easier software engineering would be if there were a period where things are simply not available. What problems are currently very difficult would be made trivial if 6 hours of downtime every Sunday were acceptable? 10PM-4AM EST

Your team is now working 10pm to 4am est every Sunday, and anyone who uses your services in any time zone around the world is without your service. 10pm est is lunch time in Australia. 4am est is working hours in many parts of western Europe. You don't have to be huge to have customers all around the world.

You’re correct, but if there happened to be an outage during that time now you’d have to have your team working then anyway.

The existence of a planned downtime doesn’t necessarily mean deployments, or work, have to happen during that time.

Re: Zero downtime Postgres migration, done right

#36

Post by yupper32 »

endisneigh wrote: post #32
yupper32 wrote: post #31

Earlier quoted context omitted.

Frankly, because it can be. Weekly scheduled downtime is arbitrary, manufactured, and lazy. Of course, having to schedule the occasional downtime for a database migration is fine. There's probably a few times a year that you'd need to do it if you don't have the bandwidth to do fancy zero-downtime solutions. It's the weekly arbitrary downtime that I'm firmly against.

Why is it lazy? When things are up people have to work. Do you believe people should be working all of the time? I think regular downtime is only natural. If you had to choose between 95% availability or 100% availability other than the before mentioned downtime which would you choose?

Sorry I'm a bit confused by your argument.

> When things are up people have to work.

That's not true. Monitoring software and on-call rotations are well established things, and wouldn't even go away in your world. Believe it or not, people who run websites already sleep at night.

> Do you believe people should be working all of the time?

I'm not sure where you got that. Besides, you're already having people in the office in the middle of the night every Sunday already. Or are you saying that the downtime is when your team gets to sleep?

> If you had to choose between 95% availability or 100% availability other than the before mentioned downtime which would you choose?

Again, not sure what you're getting at. There's only a few reasons why you'd need to schedule a downtime. Database migrations are a good one. All of the reasons should only happen a few times a year, likely. Way better than 52 times a year.

Re: Zero downtime Postgres migration, done right

#37

Post by yupper32 »

endisneigh wrote: post #35
maccard wrote: post #27

Earlier quoted context omitted.

Your team is now working 10pm to 4am est every Sunday, and anyone who uses your services in any time zone around the world is without your service. 10pm est is lunch time in Australia. 4am est is working hours in many parts of western Europe. You don't have to be huge to have customers all around the world.

You’re correct, but if there happened to be an outage during that time now you’d have to have your team working then anyway. The existence of a planned downtime doesn’t necessarily mean deployments, or work, have to happen during that time.

> but if there happened to be an outage during that time now you’d have to have your team working then anyway.

So you're having them work late 52 times a year AND when there's an outage vs only when there's an outage. They're working late WAY more in your plan.

Re: Zero downtime Postgres migration, done right

#38

Post by svaha1728 »

jbverschoor wrote: post #16

> Blueground is a real estate tech company offering flexible and move-in ready furnished apartments across three continents and 12 of the world’s top cities. We search high and low for the best properties in the best cities, then our in-house design team transforms these spaces into turnkey spaces for 30 days or longer. Seriously, how big can that db be, and how bad would a 1hr reduced availability / downtime be? See…

Agree that it's a lot of wasted engineering effort for companies that can have planned downtimes.

My guess the conversation was 'To be agile, we need to be 100% CICD'. Next thing you know everything needs to get pushed straight to prod continuously with no downtime.

Re: Zero downtime Postgres migration, done right

#40

Post by maccard »

silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

I think it's always worth questioning both sides. Why is downtime acceptable? People on this site routinely complain about windows needing a restart for system updates while boasting about their Linux servers uptime.

People talk about how kubernetes is overkill for many people, but it gives you rolling deployments for your applications out of the box.

There's also the "slippery slope" argument. A 0 downtime migration means 0 downtime. A 5 minute migration creeps up to a 30 minutes migration occasionally, and then regression to the mean happens, causing standard migrations to be 30 minutes.

Post reply on HN