Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero downtime Postgres migration, done right

engineering.theblueground.com

Post reply on HN
Print view View original

51–60 of 135 posts

Re: Zero downtime Postgres migration, done right

#51

Post by munk-a »

maccard wrote: post #40
silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

I think it's always worth questioning both sides. Why is downtime acceptable? People on this site routinely complain about windows needing a restart for system updates while boasting about their Linux servers uptime. People talk about how kubernetes is overkill for many people, but it gives you rolling deployments for your applications out of the box. There's also the "slippery slope" argument. A 0 downtime migration…

> Why is downtime acceptable?

At least where I'm working downtime is acceptable because our customer base is composed of professionals in north america that work 9-5 thus taking the DB offline for five minutes at midnight PST has essentially no cost to our client base. To contrast that, spending three months developing a rollout plan costs our company dearly in competitive advantage.

I agree that it's always worth examining from both sides but I also think that 0 downtime migration is both a shiny interesting problem and a point of pride for developers that can cause a company to vastly over invest in a complex technical solution to a problem they never had.

Re: Zero downtime Postgres migration, done right

#52

Post by xcambar »

jbverschoor wrote: post #12
fullstop wrote: post #8

Earlier quoted context omitted.

I work for a small company and it would be devastating if our database was down for a few hours.

I doubt that.. aws is down sometimes too. Card payments are sometimes down, even 112(911) is sometimes down. I hope you have everything redundant, including the CEO

> I hope you have everything redundant, including the CEO

Made me laugh, I'll sure reuse it!

Re: Zero downtime Postgres migration, done right

#53

Post by xcambar »

fullstop wrote: post #45
ikiris wrote: post #34

Earlier quoted context omitted.

It always amazes me when someone says they can't ever be down, and then says they only serve from one physical location actively. One of these things is not like the other.

We have everything in place to run from AWS if needed but do not operate from there because of cost. This goes both ways -- there have been many AWS outages which have not affected us. I hear what you're saying, but we've had only one instance of extended (hours) downtime in the last 20 years.

The more experienced I become, the more such down to earth solutions seem OMG SO MUCH MORE reasonable than all the bells and whistles of "modern" engineering practices.

Re: Zero downtime Postgres migration, done right

#54

Post by coolspot »

I just don’t get all the “just shutdown the db for an hour, you’re not Netflix” comments.

If you can do things properly, as an engineer, you absolutely should, even if your company serves “just” hundreds of thousands instead of hundreds of millions users.

It is not like they wrote their own database for that, they just used an open source tool.

Re: Zero downtime Postgres migration, done right

#55

Post by xyst »

luhn wrote: post #6

Braintree (IIRC) had a really clever migration strategy, although I can't seem to find the blog post now. They paused all traffic at the load balancer, cut over to the new DB, and then resumed traffic. No requests failed, just a slight bump in latency while the LBs were paused. This app apparently had robust enough retry mechanisms that they were able just eat the errors and not have customer issues—Color me impresse…

probably had a good circuit breaker design. would love to read the article though.

Re: Zero downtime Postgres migration, done right

#56

Post by mswtk »

coolspot wrote: post #54

I just don’t get all the “just shutdown the db for an hour, you’re not Netflix” comments. If you can do things properly, as an engineer, you absolutely should, even if your company serves “just” hundreds of thousands instead of hundreds of millions users. It is not like they wrote their own database for that, they just used an open source tool.

My understanding of doing things "properly" as an engineer, is picking the solution with the right tradeoffs for my use case. If the cost to the business of having some amount of scheduled downtime occasionally is significantly less than the engineering cost of maintaining several 9s worth of availability over major migrations, then I consider the former to be "done right".

Re: Zero downtime Postgres migration, done right

#57

Post by endisneigh »

yupper32 wrote: post #37
endisneigh wrote: post #35

Earlier quoted context omitted.

You’re correct, but if there happened to be an outage during that time now you’d have to have your team working then anyway. The existence of a planned downtime doesn’t necessarily mean deployments, or work, have to happen during that time.

> but if there happened to be an outage during that time now you’d have to have your team working then anyway. So you're having them work late 52 times a year AND when there's an outage vs only when there's an outage. They're working late WAY more in your plan.

The existence of planned downtime doesn’t mean they have to work during said time. I’m not following your reasoning

Re: Zero downtime Postgres migration, done right

#58

Post by hinkley »

Daishiman wrote: post #47
maccard wrote: post #40

Earlier quoted context omitted.

I think it's always worth questioning both sides. Why is downtime acceptable? People on this site routinely complain about windows needing a restart for system updates while boasting about their Linux servers uptime. People talk about how kubernetes is overkill for many people, but it gives you rolling deployments for your applications out of the box. There's also the "slippery slope" argument. A 0 downtime migration…

A lot of 0-downtime migrations end up turning into a months-long ordeal and can still fail. Allowing maintenance windows means you can do things in a much simple manner. Need to take a snapshot of a DB without running into issues with a production system adding data? Sure, go ahead, you just saved two months.

It's the old can/should argument.

The principle of something like zero downtime or switching a vendor out to save 20% may not stack up against the reality in opportunity costs, employee burnout, and customer turnover.

My company is reorganizing and so everyone is trying to settle old philosophical debates during The Churn. But everyone is overextended and a number of us have started pushing back. Just the meetings to discuss changing vendors might push the break-even point out 6 months, before you even get into implementation costs. The more time we spend working on infrastructure instead of customer-facing features and bugs, the more likely those customers are to wander off.

Re: Zero downtime Postgres migration, done right

#59

Post by endisneigh »

yupper32 wrote: post #36
endisneigh wrote: post #32

Earlier quoted context omitted.

Why is it lazy? When things are up people have to work. Do you believe people should be working all of the time? I think regular downtime is only natural. If you had to choose between 95% availability or 100% availability other than the before mentioned downtime which would you choose?

Sorry I'm a bit confused by your argument. > When things are up people have to work. That's not true. Monitoring software and on-call rotations are well established things, and wouldn't even go away in your world. Believe it or not, people who run websites already sleep at night. > Do you believe people should be working all of the time? I'm not sure where you got that. Besides, you're already having people in the of…

I’m not making a particular argument. I’m just wondering if having regular downtime is better or worse in terms of reducing engineering complexity.
Post reply on HN