Earlier quoted context omitted.
Hey Daniel, I want to thank you for your candid feedback. Rest assured that this sort of thing makes it back to the team and is truly appreciated no matter how harsh it is. You're absolutely right -- we need to do better. We're aware of several issues related to the .com service, mostly focused on reliability and speed, and have prioritized these issues this quarter. The site is down so I can't link directly, but her…
I'm a bit curious here. Do you think that your issues with scalability and reliability have to do with your tech choice (I think it was Ruby on Rails)? Don't want to bash Rails, I'm just genuinely curious, since I come from a Rails background as well and have seen issues similar to yours in the past.
GitLab Database Incident – Live Report
331–340 of 621 posts
Re: GitLab Database Incident – Live Report
#332Earlier quoted context omitted.
Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…
>> candidates will be invited to schedule a first 45 minute behavioral interview with the Infrastructure Lead Yes, go right ahead and filter out some (disclaimer before the rant: some, not all) of the best talent. The kind of potential employee that gets rejected due to perceived personality problems is exactly the kind of person who would tell management to shove a stick up their ass for demanding a 2 week deadline…
Moreover, many with abrasive personalities are the ones who demand and take for granted excessive overtime the moment they are in senior or lead position. "I am too tired and it is late at night lets do it tomorrow" is not an option to abrasive personality.
As a rule, obsessive freak with no social skills you describe wont be able to effectively reorganize the team nor create process nor (if in extreme version) work within process someone else created. He is more likely to end up in endless quarrels about petty differences in coding style.
Ability to create, fight for and enforce sane process is not the same as being abrasive.
He is however absolutely fine when you have isolated position.
Re: GitLab Database Incident – Live Report
#333Earlier quoted context omitted.
>> candidates will be invited to schedule a first 45 minute behavioral interview with the Infrastructure Lead Yes, go right ahead and filter out some (disclaimer before the rant: some, not all) of the best talent. The kind of potential employee that gets rejected due to perceived personality problems is exactly the kind of person who would tell management to shove a stick up their ass for demanding a 2 week deadline…
Not vetting people based on behavior is perilously close to accepting the old adage "Say what you like about Mussolini, but at least he made the trains run on time".
Re: GitLab Database Incident – Live Report
#334This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…
Two points: 1) Patio11 touches on a very good lesson, in passing, in an article about Japanese business[1]: While raw programming ability might not be highly valued at many Japanese companies, and engineers are often not in positions of authority, there is nonetheless a commitment to excellence in the practice of engineering. I am an enormously better engineer for having had three years to learn under the more senior…
Why am I saying that? Because in some of the Japanese companies I've worked with were the exact opposite of that. To be sure, lip service was duly paid to the aforementioned "commitment to excellence", and every release procedure had its own operational manual, sometimes 300 steps long. Repeated manually for every server. Out of 100-200.
Configuration updates? Sure, let's log in to every server and vi the config file. How do we keep excellence? Just diff with prev and verify (with your eyes that is) that the result is the same as in your manual. After every "cd" you had to do a pwd and make sure that you moved to the directory you meant to. After every cp you diffed to the original file to make.
Releases obviously took all day or often all night, and engineers were stressed and fatigued by Sisyphean manual with its 300 steps of red tape. They invariably made silly mistake, because this is what you get when you use human beings as glorified tty+diff. We had release issues and service outages all the time.
We've fortunately managed to move away to modern DevOps practices with a lot of top down effort. But please don't tell me every Japanese company magically delivers top quality. Some of them do, some of them don't, even in the same industry. Insane levels of bureaucracy could be found all across the board, but whether that bureaucracy actually encourages or deters quality is an entirely different story.
Re: GitLab Database Incident – Live Report
#335Earlier quoted context omitted.
What if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.
Startups only, you say? Everything, everywhere, is held together by ducttape!
Re: GitLab Database Incident – Live Report
#336I've been using the PS1 trick they mention for the last couple of years and I've found it to be a really good visual check (red=prod, yellow=staging, green=dev). We then also apply the colorscheme to the header in our admin pages too. Those of us that are jumping between environments are a big risk to data :-)
Re: GitLab Database Incident – Live Report
#337Earlier quoted context omitted.
Hey Daniel, I want to thank you for your candid feedback. Rest assured that this sort of thing makes it back to the team and is truly appreciated no matter how harsh it is. You're absolutely right -- we need to do better. We're aware of several issues related to the .com service, mostly focused on reliability and speed, and have prioritized these issues this quarter. The site is down so I can't link directly, but her…
I'm a bit curious here. Do you think that your issues with scalability and reliability have to do with your tech choice (I think it was Ruby on Rails)? Don't want to bash Rails, I'm just genuinely curious, since I come from a Rails background as well and have seen issues similar to yours in the past.
Re: GitLab Database Incident – Live Report
#338This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…
Two points: 1) Patio11 touches on a very good lesson, in passing, in an article about Japanese business[1]: While raw programming ability might not be highly valued at many Japanese companies, and engineers are often not in positions of authority, there is nonetheless a commitment to excellence in the practice of engineering. I am an enormously better engineer for having had three years to learn under the more senior…
Re: GitLab Database Incident – Live Report
#339I haven't seen or done anything of this scale before, but I did have a very sobering moment while working on a large online retailers stack as a systems engineer. We were rolling out a new stack in another data center across the country and before replication went live, I decided to connect and check things out. Our chef work hadn't completed for the database hosts, so I decided to install some OS updates by hand usi…
Re: GitLab Database Incident – Live Report
#340Earlier quoted context omitted.
I'm not sure there is a "late" at night or tired in the incident report. All times are UTC and it's unclear where all the team is located but if in SF then this is at 4pm which is when the incident just occurred. It doesn't necessarily change that you shouldn't be firefighting for extremely long cycles in hero-mode for that long, but not exactly the same as exhausted and powering away for hours.
Gitlab's team is worldwide. YP seems to be someone in EU time, which right now is UTC+1. That would mean that the incident happened around 11pm or midnight in YP's local time.