Earlier quoted context omitted.
Good lesson on making command prompts on machines always tell you exactly what machine you're working on.
'db1' vs. 'db2' is still insufficiently clear, though. Even better would be e.g. to name development systems after planets and production systems after superheroes. Very few people would mistake 'superman' for either 'green-lantern' or 'pluto,' but it's really easy to mistake 'sfnypudb13' for 'sfnydudb13.'
GitLab Database Incident – Live Report
181–190 of 621 posts
Re: GitLab Database Incident – Live Report
#182For what it's worth, it was a lesson worth learning despite what seemed like catastrophic world-ending circumstances.
We survived, and GitLab will too. GitLab has been an extraordinary service since the beginning. Even if their repos were to get wiped (which seems not to be the case), I'd still continue supporting them (after I re-up'd from my local repos). I appreciate their transparency and hope that they can turn this situation into a positive lesson in the long run.
Best of luck to GitLab sysops and don't forget to get some sleep and relax.
Re: GitLab Database Incident – Live Report
#183Earlier quoted context omitted.
Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…
5 interviews isn't -that- insane, it's on the high end though. But I do know for sure they have turned down extremely technically gifted people for "non-technical" reasons, the kind of people that would ensure this sort of disaster wouldn't happen. So maybe the process really isn't that great.
Re: GitLab Database Incident – Live Report
#184Earlier quoted context omitted.
Good lesson on making command prompts on machines always tell you exactly what machine you're working on.
I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
Re: GitLab Database Incident – Live Report
#185This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…
Re: GitLab Database Incident – Live Report
#186Earlier quoted context omitted.
I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.
Nope, that would be me.
Re: GitLab Database Incident – Live Report
#187http://www2.rafaelguimaraes.net/wp-content/uploads/2015/12/g...
Re: GitLab Database Incident – Live Report
#188Earlier quoted context omitted.
What if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.
There's a big difference between holding together things with duct tape, and literally taking zero backups of it.
Re: GitLab Database Incident – Live Report
#189I'm a huge Gitlab fan. But I long ago lost faith in their ability to run a production service at scale. Nothing important of mine is allowed to live exclusively on Gitlab.com. It seems like they are just growing too fast for their level of investment in their production environment. One of the only reasons I was comfortable using Gitlab.com in the first place was because I knew I could migrate off it without too much…
You're absolutely right -- we need to do better. We're aware of several issues related to the .com service, mostly focused on reliability and speed, and have prioritized these issues this quarter. The site is down so I can't link directly, but here's a link to a cached version of the issue where we're discussing all of this if you'd like to chime in once things are back up: https://webcache.googleusercontent.com/search?q=cache:YgzBJm...
Re: GitLab Database Incident – Live Report
#190"So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."