61–70 of 135 posts
Most modern rdbms/nosql database vendors allow a rolling upgrade where you roll in new servers and roll out the old ones seamlessly.
Also the fact that AWS rds doesnt do this with zero downtime by default through automating it this way is also crazy. Why pay for hosted when the upgrade story is incomplete? Take downtime to upgrade a DB in 2021? Everyone must be joking.
Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…
I really think articles of this kind are not useless, but need to explicitly narrow their audience to set expectation at the start. This particular topic is PG replication for users who aren't very sensitive to write latency.
Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…
I just don’t get all the “just shutdown the db for an hour, you’re not Netflix” comments. If you can do things properly, as an engineer, you absolutely should, even if your company serves “just” hundreds of thousands instead of hundreds of millions users. It is not like they wrote their own database for that, they just used an open source tool.
If you've got say 5 minutes of downtime budget per month, do you really think it's a good investment to spend a million dollars of engineering effort and opportunity costs to get database downtime to zero seconds? Or you could use off-the-shelf techniques for orders of magnitude less investment and suffer a few minutes of downtime at worst, well within your SLO. It's an economic decision, not a technical one.
If zero downtime at any cost is non-negotiable, well you'd better hire, plan and budget accordingly. And hope those few extra minutes per year are worth the cost.
Braintree (IIRC) had a really clever migration strategy, although I can't seem to find the blog post now. They paused all traffic at the load balancer, cut over to the new DB, and then resumed traffic. No requests failed, just a slight bump in latency while the LBs were paused. This app apparently had robust enough retry mechanisms that they were able just eat the errors and not have customer issues—Color me impresse…
I wonder how much easier software engineering would be if there were a period where things are simply not available. What problems are currently very difficult would be made trivial if 6 hours of downtime every Sunday were acceptable? 10PM-4AM EST
One service I worked on (with customers around the globe) had about 10 minutes of scheduled downtime a week. Peak user load was, interestingly enough, during our core hours, when there are a lot more hands on deck to handle any issues. So that's the time we chose.
Naturally people still complained bitterly about that ten minutes.
Getting to zero didn't seem worth the significant engineering effort required. "You're not Amazon, and you're not 911" [emergency number in the US]
I just don’t get all the “just shutdown the db for an hour, you’re not Netflix” comments. If you can do things properly, as an engineer, you absolutely should, even if your company serves “just” hundreds of thousands instead of hundreds of millions users. It is not like they wrote their own database for that, they just used an open source tool.
If you do things properly as an engineer, you've already negotiated and committed to service-level agreements with specific downtime objectives, right? Right? If you've got say 5 minutes of downtime budget per month, do you really think it's a good investment to spend a million dollars of engineering effort and opportunity costs to get database downtime to zero seconds? Or you could use off-the-shelf techniques for o…
Also engineers get better (and happier) when they do challenging tasks!
Earlier quoted context omitted.
> but if there happened to be an outage during that time now you’d have to have your team working then anyway. So you're having them work late 52 times a year AND when there's an outage vs only when there's an outage. They're working late WAY more in your plan.
The existence of planned downtime doesn’t mean they have to work during said time. I’m not following your reasoning
I guarantee that you'll have trouble hiring qualified people for a role that includes core hours of "2AM-5AM Sunday mornings" (or whatever other oddball time you've decided). Updating systems is precisely when you may need to have all-hands-on-deck responses. Been there, multiple times.
Braintree (IIRC) had a really clever migration strategy, although I can't seem to find the blog post now. They paused all traffic at the load balancer, cut over to the new DB, and then resumed traffic. No requests failed, just a slight bump in latency while the LBs were paused. This app apparently had robust enough retry mechanisms that they were able just eat the errors and not have customer issues—Color me impresse…
I think that's covered in this talk (I've not watched the video though): https://www.braintreepayments.com/blog/ruby-conf-australia-h...