Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

Post reply on HN
Print view View original

21–30 of 269 posts

Re: Postmortem of database outage of January 31

#21

Post by chadcmulligan »

jdavis703 wrote: post #19
yarper wrote: post #14

It's amazing how quickly it descends into "one of the engineers" did x or y. Who was steering this ship exactly? It's really simple to point the finger and try to find a single cause of failure - but it's a fools errand - comparable to finding the single source behind a great success.

Do you expect management to be staring over your shoulder every time you do some kind of `rm` on a production server? With great power comes great responsibility.

doesn't everyone alias rm to rm -i on prod?

likewise all tty's have red backgrounds on prod.

Re: Postmortem of database outage of January 31

#22

Post by MichaelGG »

bitshepherd wrote: post #17
illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

A great number of issues can be attributed to the selection of Azure as the platform of choice. That said, a little bird told me that the decision was largely a cost factor. "You get what you pay for" never rang more true.

Is Azure cutting deals beyond the usual free 60k over a year or two or whatever it is for cool startups? Azure seems significantly more expensive in general, problems and slowness aside.

Re: Postmortem of database outage of January 31

#23

Post by gtaylor »

sb8244 wrote: post #16
animex wrote: post #6

Earlier quoted context omitted.

What do you think would have happened if you had total data loss/failure?

I don't work for gitlab. Isn't that a put you out of business moment? I guess there is enterprise installations but the main website would be toast. I guess there is the chance there were still older backups taken intentionally or by chance that would have been worse but recoverable from

Most of their income probably comes from customers that run their own GitLab Enterprises installs. This would have really sucked for all of their non paying users, though.

Re: Postmortem of database outage of January 31

#24

Post by nilved »

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

RDS also comes with its own set of tradeoffs. There is no free lunch, and the cloud is just another word for someone else's server. There are reasons Gitlab opposes that.

Re: Postmortem of database outage of January 31

#26

Post by BinaryIdiot »

rectang wrote: post #20
meowface wrote: post #10

>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from readin…

For me, it was when I meant to execute this... rm -rf ~/foo ... but executed this instead: rm -rf ~ /foo

I did the Windows equivalent once a long time ago (I think it's deltree?) and I did it on a university computer system. It cleared out a TON of files and the computer itself pretty much stopped working. I had to hard turn it off.

Fortunately the University was using some tool that can re-image a computer each time it boots before hitting Windows so starting it back up and all the deleted system and application files were back.

Re: Postmortem of database outage of January 31

#27

Post by tedunangst »

bitshepherd wrote: post #17
illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

A great number of issues can be attributed to the selection of Azure as the platform of choice. That said, a little bird told me that the decision was largely a cost factor. "You get what you pay for" never rang more true.

But none of the issues were azure/cost related except for the slow recovery? I mean, neither AWS nor GCE can make you notice youre not getting cron mail.

Re: Postmortem of database outage of January 31

#28

Post by nathancahill »

chadcmulligan wrote: post #21
jdavis703 wrote: post #19

Earlier quoted context omitted.

Do you expect management to be staring over your shoulder every time you do some kind of `rm` on a production server? With great power comes great responsibility.

doesn't everyone alias rm to rm -i on prod? likewise all tty's have red backgrounds on prod.

I'm not sure if you're making an equally sarcastic point as your parent or not..

Re: Postmortem of database outage of January 31

#30

Post by sytse »

nowarninglabel wrote: post #15

Thanks so much for the post and transparency Gitlab! We had just finished recovering from our own outage (stemming for a power loss and subsequent cascading failures) and were scheduled to do our post-mortem on 2/1 so the original document was a refreshing and reassuring read.

Glad to hear it was of use to you.
Post reply on HN