Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

181–190 of 621 posts

Re: GitLab Database Incident – Live Report

#181
post #165
post #14

Earlier quoted context omitted.

Good lesson on making command prompts on machines always tell you exactly what machine you're working on.

'db1' vs. 'db2' is still insufficiently clear, though. Even better would be e.g. to name development systems after planets and production systems after superheroes. Very few people would mistake 'superman' for either 'green-lantern' or 'pluto,' but it's really easy to mistake 'sfnypudb13' for 'sfnydudb13.'

Or skip the middleman: prod-db1, prod-db2, dev-db1, dev-db2.

Re: GitLab Database Incident – Live Report

#182
I remember when we accidentally deleted our customers' data. That was the worst feeling I ever had running our business. It was about 4% of our entire storage set and had to let our customers know and start the resyncs. Those first 12 hours of panic were physically and emotionally debilitating - more than they have the right to be. I learned an important lesson that day: Business is business and personal is personal. I remember it like it was yesterday, the momement I conciously decided I would no longer allow business operations determine my physical health (stress level, heart rate, sleep schedule).

For what it's worth, it was a lesson worth learning despite what seemed like catastrophic world-ending circumstances.

We survived, and GitLab will too. GitLab has been an extraordinary service since the beginning. Even if their repos were to get wiped (which seems not to be the case), I'd still continue supporting them (after I re-up'd from my local repos). I appreciate their transparency and hope that they can turn this situation into a positive lesson in the long run.

Best of luck to GitLab sysops and don't forget to get some sleep and relax.

Re: GitLab Database Incident – Live Report

#183
post #144

Earlier quoted context omitted.

Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…

5 interviews isn't -that- insane, it's on the high end though. But I do know for sure they have turned down extremely technically gifted people for "non-technical" reasons, the kind of people that would ensure this sort of disaster wouldn't happen. So maybe the process really isn't that great.

I wondered why they always have so many job reqs out, and why they're out for so long. It always seemed like an interesting organization to me, and I like the product, but everything about their hiring process seems to be shoeing people off. Perhaps this will be incentive to try onboarding people or at least to bring in some consultants to shore them up.

Re: GitLab Database Incident – Live Report

#184
post #14

Earlier quoted context omitted.

Good lesson on making command prompts on machines always tell you exactly what machine you're working on.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

Not as serious, but I always set root terminals on my system to be this scary red.

Re: GitLab Database Incident – Live Report

#185
post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

If you get the chance to observe pilots operating in the cockpit, I'd recommend it. Every important procedure (even though the pilot has it memorized) is done with a checklist. Important actions are verbally announced and confirmed: "You have the controls" "I have the controls". Much of flight training deals with situational awareness and eliminating distractions in the cockpit. Crew Resource Management[1].

1: https://en.wikipedia.org/wiki/Crew_resource_management

Re: GitLab Database Incident – Live Report

#186

Earlier quoted context omitted.

I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.

Nope, that would be me.

Hey man, don't beat yourself up over this. It's shitty but you found some flaws in the process, in the setup, and y'all can make things better because of that.

Re: GitLab Database Incident – Live Report

#187
I like to always try to automate stuff as much as possible to remove human error. It's easy to forget an arg or other parameter on a script or even know what the arg was for in the first place. Sounds like their last backup was 24 hrs ago. Having backups are like having good security, you don't realize how important they are until it's too late. Reminds me of this old meme:

http://www2.rafaelguimaraes.net/wp-content/uploads/2015/12/g...

Re: GitLab Database Incident – Live Report

#188
post #120

Earlier quoted context omitted.

What if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.

There's a big difference between holding together things with duct tape, and literally taking zero backups of it.

I'm not sure that there is.

Re: GitLab Database Incident – Live Report

#189

I'm a huge Gitlab fan. But I long ago lost faith in their ability to run a production service at scale. Nothing important of mine is allowed to live exclusively on Gitlab.com. It seems like they are just growing too fast for their level of investment in their production environment. One of the only reasons I was comfortable using Gitlab.com in the first place was because I knew I could migrate off it without too much…

Hey Daniel, I want to thank you for your candid feedback. Rest assured that this sort of thing makes it back to the team and is truly appreciated no matter how harsh it is.

You're absolutely right -- we need to do better. We're aware of several issues related to the .com service, mostly focused on reliability and speed, and have prioritized these issues this quarter. The site is down so I can't link directly, but here's a link to a cached version of the issue where we're discussing all of this if you'd like to chime in once things are back up: https://webcache.googleusercontent.com/search?q=cache:YgzBJm...

Post reply on HN