Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

Post reply on HN
Print view View original

261–269 of 269 posts

Re: Postmortem of database outage of January 31

#261

Post by nhumrich »

illumin8 wrote: post #218
nhumrich wrote: post #194

Earlier quoted context omitted.

I know about the daily snapshots, but didn't know about the archive logs. Is this something I have to enable? How do I get the logs and how do I restore using them?

It's automatic. Go ahead and launch a new instance, restoring to a point in time (that's how you do restores in RDS). Notice that it gives you a calendar day/date/time fields where you can select the recovery point down to the second. This is enabled by replaying the archive logs to get you to the exact point in time.

Thank you! I never realized I could do this for some reason. I always restored from snapshots.

Re: Postmortem of database outage of January 31

#263

Post by theptip »

kuschku wrote: post #230
KayEss wrote: post #132

The engineers still seem to have a physical server mindset rather than a cloud mindset. Deleting data is always extremely dangerous and there was no need for it in this situation. They should have spun up a new server to act as secondary the moment replication failed. This new server is the one you run all of these commands on, and if you make a mistake you spin up a new one. Only when the replication is back in good…

That’s a nice idea, if you’re willing to pay the massive extra cost to actually rent all those overpriced systems. For me, personally, going from cloud servers to rented dedicated servers cut my bill by 93% – more than an order of magnitude. At same performance. In fact, it’d be cheaper to run 10x as many dedicated servers than to use cloud solutions for me.

It cut your cloud services bill by 93%, but how much did it increase your engineering bill by?

If your engineering time is free, then this calculation is complete. Otherwise it is not.

Does that 93% saving pay for a DB engineer, or enough of your developers' time to build the same quality of redundancy as you'd get with a DBaaS?

This calculus is going to be different for every DB and every company, but the OpEx impact of switching to dedicated servers is a bit more complex than you suggest above.

Re: Postmortem of database outage of January 31

#264

Post by kuschku »

theptip wrote: post #263
kuschku wrote: post #230

Earlier quoted context omitted.

That’s a nice idea, if you’re willing to pay the massive extra cost to actually rent all those overpriced systems. For me, personally, going from cloud servers to rented dedicated servers cut my bill by 93% – more than an order of magnitude. At same performance. In fact, it’d be cheaper to run 10x as many dedicated servers than to use cloud solutions for me.

It cut your cloud services bill by 93%, but how much did it increase your engineering bill by? If your engineering time is free, then this calculation is complete. Otherwise it is not. Does that 93% saving pay for a DB engineer, or enough of your developers' time to build the same quality of redundancy as you'd get with a DBaaS? This calculus is going to be different for every DB and every company, but the OpEx impac…

(a) I’m talking about projects I host in my free time (b) My server budget is fixed.

So, for me the choice was between "use cloud tools, and get performance worse than a raspberry pi", or "run dedicated, and get more performance and storage and traffic than I need, and actually the ability to run my stuff".

For less than the price of a Netflix subscription I’m able to run services that can handle tenthousands of concurrent users, and have terabytes of storage (and enough traffic that I never have to worry about that).

And the cost of setting it up was for me a few days.

For me it was a decision between being able to run services, or not being able to run them at all.

Re: Postmortem of database outage of January 31

#265

Post by theptip »

kuschku wrote: post #264
theptip wrote: post #263

Earlier quoted context omitted.

It cut your cloud services bill by 93%, but how much did it increase your engineering bill by? If your engineering time is free, then this calculation is complete. Otherwise it is not. Does that 93% saving pay for a DB engineer, or enough of your developers' time to build the same quality of redundancy as you'd get with a DBaaS? This calculus is going to be different for every DB and every company, but the OpEx impac…

(a) I’m talking about projects I host in my free time (b) My server budget is fixed. So, for me the choice was between "use cloud tools, and get performance worse than a raspberry pi", or "run dedicated, and get more performance and storage and traffic than I need, and actually the ability to run my stuff". For less than the price of a Netflix subscription I’m able to run services that can handle tenthousands of conc…

Sure, hobby/spare-time projects are one of the cases where it's perfectly reasonable to self-host; often it's fun to learn about the underlying tools by rolling your own db, and doing so can save you some cash (at the expense of your own time).

However, that paradigm is not really applicable to GitLab's OpEx calculation; they have to pay their engineers ;)

Re: Postmortem of database outage of January 31

#266

Post by kuschku »

theptip wrote: post #265
kuschku wrote: post #264

Earlier quoted context omitted.

(a) I’m talking about projects I host in my free time (b) My server budget is fixed. So, for me the choice was between "use cloud tools, and get performance worse than a raspberry pi", or "run dedicated, and get more performance and storage and traffic than I need, and actually the ability to run my stuff". For less than the price of a Netflix subscription I’m able to run services that can handle tenthousands of conc…

Sure, hobby/spare-time projects are one of the cases where it's perfectly reasonable to self-host; often it's fun to learn about the underlying tools by rolling your own db, and doing so can save you some cash (at the expense of your own time). However, that paradigm is not really applicable to GitLab's OpEx calculation; they have to pay their engineers ;)

Even for GitLab, when you have 1-2 orders of magnitude price difference, it might be more affordable to hire a DBA.

You have to remember GitLab is a 100% remote company, so they can hire DBAs from anywhere on the planet.

The cloud obviously makes a lot more sense if you have US electricity prices and Silicon Valley wages.

Re: Postmortem of database outage of January 31

#267

Post by theptip »

kuschku wrote: post #266
theptip wrote: post #265

Earlier quoted context omitted.

Sure, hobby/spare-time projects are one of the cases where it's perfectly reasonable to self-host; often it's fun to learn about the underlying tools by rolling your own db, and doing so can save you some cash (at the expense of your own time). However, that paradigm is not really applicable to GitLab's OpEx calculation; they have to pay their engineers ;)

Even for GitLab, when you have 1-2 orders of magnitude price difference, it might be more affordable to hire a DBA. You have to remember GitLab is a 100% remote company, so they can hire DBAs from anywhere on the planet. The cloud obviously makes a lot more sense if you have US electricity prices and Silicon Valley wages.

Yes, it might be more affordable. They seem to think it is, as they have chosen to go with self-hosted.

My point is simply that your posts above didn't address the complexity of their calculation, as they didn't factor the costs of switching to self-hosted.

Re: Postmortem of database outage of January 31

#268

Post by RhodesianHunter »

pgaddict wrote: post #192
RhodesianHunter wrote: post #99

Earlier quoted context omitted.

"Maybe it is OK as a simple data store for a single app, but not for a real database." Currently working at company number 2 with large (many terabytes) databases on RDS and can safely say this is horse shit. The amount of time and energy it allows our engineers to spend on our actual products instead of database management is worth all of the extra cost and lock in and then some. Edit: I just realized that you were…

We're in the business of PostgreSQL support, and some of our customers use RDS for various reasons. Not having to care about the deployment is one of the usual goals of using a managed environment, but considering they subsequently go and buy support from a third party might be a sign of something. Of course, my view is biased because we only hear about the issues - there might be a 100x more people using RDS without…

"and difficulty when migrating from RDS with minimum downtime"

They're simply uninformed then. AWS database migration service makes zero downtime migrations trivial between just about any major databases (mysql, oracle, postgres, aurora, sqlserver, etc.)

Re: Postmortem of database outage of January 31

#269

Post by RhodesianHunter »

_Codemonkeyism wrote: post #122
RhodesianHunter wrote: post #99

Earlier quoted context omitted.

"Maybe it is OK as a simple data store for a single app, but not for a real database." Currently working at company number 2 with large (many terabytes) databases on RDS and can safely say this is horse shit. The amount of time and energy it allows our engineers to spend on our actual products instead of database management is worth all of the extra cost and lock in and then some. Edit: I just realized that you were…

I'm interested in a. How many hours would you guess you are saving a month? b. What takes by a factor of 10 less time on RDS than doing it by hand? What task sees the largest time saved? Because I always wonder when reading this, what am I missing? What haven't we done? Were we lucky? We were running MySQL and Postgres for multi hundred million EUR companies with millions of users and we did not spend a lot of effort…

1. Zero effort backups.

2. Zero effort re-deployment from backups.

3. Almost zero effort encryption at rest.

4. Zero effort hot backups, automatic fail-overs, and multiple datacenter deployments

5. Low effort migrations of massive amounts of data between DBs when someone inevitably wants to refactor something

6. Zero effort logging and log aggregation

7. Almost zero effort alerting of issues via sms/email/other

I could go on but I'm on my way to work...

When you're paying engineers north of 150K all of this adds up, and I'd much rather throw the money at Amazon to handle this and pay the engineers to focus on our actual product.

Post reply on HN