Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

591–600 of 621 posts

Re: GitLab Database Incident – Live Report

#591

Post by lallysingh »

I'm commenting a bit late, but I hope it's still read by the gitlab team.

First, you kept heads and didn't turn on each other. That's a major success, and gives me more confidence in gitlab. The rest you can improve on only if you have this right.

Second, I'm sure you've gotten the message to test your backups and recovery plan. It's a good time to read the Google SRE book, and consider how to put together full integration tests that build up a db, back it up, nuke the original, and recover from the db. With containers this isn't actually awful to do.

But I didn't see much mentioned about load tests. A few simple scripts that hit your server (or test instance!) hard can help you find points where things fall apart under load. Even if you don't have a good way to gracefully do anything else other than alert a human, you can figure out what to monitor and how to make sure your backup/recovery plans can deal with a shit-ton of spammer data suddenly in your DB.

Re: GitLab Database Incident – Live Report

#592

Post by cabargas »

lallysingh wrote: post #591

I'm commenting a bit late, but I hope it's still read by the gitlab team. First, you kept heads and didn't turn on each other. That's a major success, and gives me more confidence in gitlab. The rest you can improve on only if you have this right. Second, I'm sure you've gotten the message to test your backups and recovery plan. It's a good time to read the Google SRE book, and consider how to put together full integ…

Thank you for your feedback. I will add your suggestions to our document!

We will be implementing new policies on backups and you can totally expect us doing load tests in the future. The whole team wants to make sure this will not happen again.

Re: GitLab Database Incident – Live Report

#593

Post by user15672 »

rhizome wrote: post #588
user15672 wrote: post #583

Earlier quoted context omitted.

Woah there! I think you may have misread the parent as it looks like some friendly advice to me (with actual reasons and stuff), rather than the "You shouldn't have done that! You've destroyed your company!!!!" you seem to have read it as. Heck, they didn't even say to retract anything from the report, just maybe to leave adding names to things until a later date in future incidents.

I'm not sure impugning their professionalism qualifies as friendly advice.

Again, you've put your own tone on things.

"It also keeps a sense of professionalism as it's clear that no blame is being assigned" is not the same as "you guys acted unprofessionally!". It's letting the GitLab guys know there's a potential problem with the communication style at that point in the story.

I find it funny that in a comments section full of comments about allowing a frank learning experience you're being so down on someone giving tips to consider learning from.

Re: GitLab Database Incident – Live Report

#594

Post by ygersie »

Funny that I haven't seen the following anecdote yet: One is none, two is one (== none). AFAICT there were only 2 Postgres instances? What gives? How would you ever feel comfortable when one goes down?

How we deal with recovery:

- run DB servers on ZFS - built a tool to orchestrate snapshotting (every 15 minutes) using an external Mutex to distribute snapshot creation for best recover accuracy. You could also have increased retention over time like:

- keep 6 snapshots of 5 minutes - 4 hourly - 1 daily - 1 weekly

Recover: choose point in time closest to fckup, the tool automatically elects the DB with closest (earlier than given time) snapshot. All other slaves are restored before that point in time and roll forward to the active state of new "master".

Instead of executing worst case recovery plans by copying data to at least 6 (minimal) db read slaves we can recover in minutes with minimal data loss (especially when you consider downtime == data loss).

There are cases where a setup like this would be a no go (think of companies where having lost transactions are absolutely devastating) but I don't think Gitlab is one of those.

Side effect of ZFS is being able to ship blocks of data as offsite backups (instead of dumping), able to `zpool import` anywhere, checksumming, compression etc etc..

Re: GitLab Database Incident – Live Report

#595

Post by brongondwana »

Washuu wrote: post #92
ckdarby wrote: post #78

"Our backups to S3 apparently don’t work either: the bucket is empty" 6/6 failed backup procedures. Looks like they are going to be hiring a new sysadmin/devops person...

The best system administrator is the one that has learned from their catastrophic fuck up. To that effect, I still have the same job as I did before I ran "yum update" without knowing it attempts to do in place kernel upgrades. Which resulted in a corrupted RedHat installation on a server we could not turn off.

"on a server we could not turn off"

I think I've found your root issue. If you can't turn any server off with a few minutes' notice, you're one hardware failure away from dead.

Re: GitLab Database Incident – Live Report

#596

Post by tssva »

Perihelion wrote: post #189
DanielDent wrote: post #153

I'm a huge Gitlab fan. But I long ago lost faith in their ability to run a production service at scale. Nothing important of mine is allowed to live exclusively on Gitlab.com. It seems like they are just growing too fast for their level of investment in their production environment. One of the only reasons I was comfortable using Gitlab.com in the first place was because I knew I could migrate off it without too much…

Hey Daniel, I want to thank you for your candid feedback. Rest assured that this sort of thing makes it back to the team and is truly appreciated no matter how harsh it is. You're absolutely right -- we need to do better. We're aware of several issues related to the .com service, mostly focused on reliability and speed, and have prioritized these issues this quarter. The site is down so I can't link directly, but her…

I have searched the gitlab website and repositories looking for processes and procedures addressing change management, release management, incident management or really anything. I have found work instructions but no processes or procedures. Until you develop and enforce some appropriate processes and the resulting procedures I'm afraid you will never be able to deliver and maintain an enterprise level service.

Hopefully this will be the learning experience which allows you to place an emphasis on these things going forward and don't fall into the trap of thinking formal processes and procedures are somehow incongruent with speedy time to market, technological innovation or in conflict with DevOps.

Re: GitLab Database Incident – Live Report

#597

Post by DanielDent »

b0ti wrote: post #543
Perihelion wrote: post #189

Earlier quoted context omitted.

Hey Daniel, I want to thank you for your candid feedback. Rest assured that this sort of thing makes it back to the team and is truly appreciated no matter how harsh it is. You're absolutely right -- we need to do better. We're aware of several issues related to the .com service, mostly focused on reliability and speed, and have prioritized these issues this quarter. The site is down so I can't link directly, but her…

I'm running a remote-only company and we moved to GitLab.com last summer from cloud hosted trac+git/svn combo (xp-dev). The reason we picked GitLab.com was because the stack is awesome and Trac is showing its age. We also wanted a solution that could be ran on premises if needed. We spent about a month migrating stuff over to GitLab from Trac. Once we were settled the reliability issues started to show. We were hopin…

Ping me and we'd be happy to discuss hosted Gitlab for you.

Re: GitLab Database Incident – Live Report

#598

Post by protomikron »

sschueller wrote: post #282
ryandrake wrote: post #185

Earlier quoted context omitted.

If you get the chance to observe pilots operating in the cockpit, I'd recommend it. Every important procedure (even though the pilot has it memorized) is done with a checklist. Important actions are verbally announced and confirmed: "You have the controls" "I have the controls". Much of flight training deals with situational awareness and eliminating distractions in the cockpit. Crew Resource Management[1]. 1: https:…

There is a neat video[1] where a Swiss flight has to make an emergency landing and just happens to have a film crew in the cockpit. [1] https://www.youtube.com/watch?v=rEf35NtlBLg

Thanks for that, this is amazing.

I speak their language (it's German, but people from Switzerland speak a pretty strong dialect) and they discuss highly technical and serious stuff, but their language is just so adorable when they mix the English and the German. I always thought this communication is English only, nowadays?

Re: GitLab Database Incident – Live Report

#599

Post by secfirstmd »

cmdkeen wrote: post #500
secfirstmd wrote: post #108

Earlier quoted context omitted.

"I intend to" is actually a phrased deliberately used a lot during military mission briefings. I always wondered why, I guess it's deliberate.

The military uses "intention" rather than "I will" because they understand that no plan survives contact with the enemy. It is also higher level so that when exhausted, stressed subordinates find themselves in life threatening circumstances the most important thing they need to remember is the intent. If they forget steps 1-5 of the plan but recall the intent they can't go that badly wrong in using their initiative.…

Much better answer! I recognise a lot of that.

Re: GitLab Database Incident – Live Report

#600

Post by ibejoeb »

jpgvm wrote: post #252
ibejoeb wrote: post #183

Earlier quoted context omitted.

I wondered why they always have so many job reqs out, and why they're out for so long. It always seemed like an interesting organization to me, and I like the product, but everything about their hiring process seems to be shoeing people off. Perhaps this will be incentive to try onboarding people or at least to bring in some consultants to shore them up.

I think part of the problem, as it has already been touched on here is true Ops people are of a different breed to standard developers. They usually have similar coding ability (but different areas of expertise, automation and systems coding over business logic) but drastically different personalities. This is an issue during hiring when companies have minimal Ops background. They (usually) attempt to interview the c…

I couldn't agree more. (My followup rant is general, and I have no knowledge of what's going on at GL.)

> Real Ops candidates are going to seem lacklustre compared to web devs when giving web dev coding tasks and inteview questions.

This drives me insane. I think 50% of my comment history this year is about how not to botch an interview.

> You probably aren't going to ask them anything actually relevant to their ability to do Ops well because you just don't know.

Yes. You won't be able to quiz that person. You can take a thoughtful look at experience and talk about that, though.

> hire based on experience

Yup. One of the spook factors at any company is when I see it only hiring down, i.e., lower than the aggregate skill level of its current team. If the VP of engineering knows more about each concentration area than the people he's hiring, there's a storm a-brewin'.

Post reply on HN