Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

421–430 of 621 posts

Re: GitLab Database Incident – Live Report

#421

Post by byroot »

throw2016 wrote: post #381
byroot wrote: post #355

Earlier quoted context omitted.

Why do you question Rails while the entire report is about Postgres only ? And as someone working on one of the biggest and oldest Rails codebase out there, I can tell you that in term of scaling, Rails is the least of our concerns. Sure it's not as efficient, so it's gonna cost you more in CPU and RAM, but it's trivial to scale horizontally. The real worry are the databases, they are fundamentally harder to scale wi…

Depending on hundreds of gems means you are depending on the decisions of hundreds of developers with packages which are in constant churn. Apps like Gitlab and Discourse that depend on hundreds of gems and require end users to have complex build environment and compile software are I think operating a broken user hostile model. The potential for compilation failures, version mismatches and Ruby oddities like RVM is…

Regardless of wether I agree or disagree with your critique, it has absolutely no relevance in the context of the current outage.

You don't like Ruby / Rails we get it. But that's totally out of topic.

Re: GitLab Database Incident – Live Report

#422

Post by [deleted] »

CrLf wrote: post #415
nanch wrote: post #413

Earlier quoted context omitted.

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…

In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)

[deleted]

Re: GitLab Database Incident – Live Report

#423

Post by NamTaf »

pmlnr wrote: post #373
unscaled wrote: post #334

Earlier quoted context omitted.

One of thing that I learned the hard way about "Japanese companies" - despite western conceptions, every company in Japan has its own unique culture (and takes pride in having its own culture!). What's more, often departments and division inside the same companies work in very different way. Why am I saying that? Because in some of the Japanese companies I've worked with were the exact opposite of that. To be sure, l…

Unfortunately I had similar experiences as well; incredibly manual processes, frighteningly long manual procedure descriptions instead of scripted solutions. My opinion: script it. Always. It doesn't matter if it's ansible, bash, puppet, python, whatever, just make sure it's not an ad-hoc command. Test the script on a server which can be sacrificed. Test as long as there is a single glitch. Run it in production. It's…

Oh, absolutely. Where something can be scripted, script it. Why? Because scripting is a process development. You write something, validate it and then remove the human error element.

For things that you can't script, you write abstracted processes that force the executor to write down the things that could cause Bad Things to happen, and use that writing down stage to verify that it's not going to cause a Bad Thing. That forces people to pause and consider what they're doing, which is 80% of the effort towards preventing these issues.

eg: Forcing YP to write down which database they were scorching would've triggered an 'oh fuck' moment. Having a process that dodged naming databases as 'db1' and 'db2' would've prevented it. etc. etc. etc.

Re: GitLab Database Incident – Live Report

#424

Post by kozak »

imglorp wrote: post #414
kozak wrote: post #379

Earlier quoted context omitted.

Does the phrase "I'll show you" also have a predefined special meaning? He seems to repeat it quite often.

I heard that as "I show you xx". One guy is flying with his head up, the other one talking most is monitoring with his head in the instruments, and helping the pilot flying getting confident data. Remember they're driving 1970's technology, redundant everything, and they all grew up flying "steam gauges", where the culture includes tapping on the glass to make sure the needle didn't stick. They want to compare every…

So "I show you X" stands for "I see X on the instruments" (as opposed to just stating X as a fact)?

Re: GitLab Database Incident – Live Report

#426

Post by NamTaf »

adrianN wrote: post #313
NamTaf wrote: post #269

Earlier quoted context omitted.

Two points: 1) Patio11 touches on a very good lesson, in passing, in an article about Japanese business[1]: While raw programming ability might not be highly valued at many Japanese companies, and engineers are often not in positions of authority, there is nonetheless a commitment to excellence in the practice of engineering. I am an enormously better engineer for having had three years to learn under the more senior…

That is certainly true of some of the red tape, but in no way it's true for the majority. A lot of "process" is created because the people in charge of the process need to validate their existence.

Undoubtably that's a problem, I agree. Competent management helps minimise that, though. And I would not agree that the majority is existance justification. Then again, I'll disclaim that by saying I work in a Mech Eng. role, where process for safety's sake has been established and engrainedi nto the culture for literal centuries.

A good employment environment is one where you may ask why a process exists and receive valid justifications therein, but where the idea of not following it, no matter how bad, never crosses your mind. I acknowledge I'm really lucky to work in an industry that doesn't fall too far from that target.

Re: GitLab Database Incident – Live Report

#427

Post by NamTaf »

patio11 wrote: post #404
unscaled wrote: post #334

Earlier quoted context omitted.

One of thing that I learned the hard way about "Japanese companies" - despite western conceptions, every company in Japan has its own unique culture (and takes pride in having its own culture!). What's more, often departments and division inside the same companies work in very different way. Why am I saying that? Because in some of the Japanese companies I've worked with were the exact opposite of that. To be sure, l…

(Only mentioning since I wrote the above quote: I agree with the general thrust of this comment.)

For the public record, by quoting you I wasn't implying that you agreed with my #2 either. I just felt I gained a lot out of both points, that they both resonated with my experiences, and that they both articulated the lessons I'd learnt in my career.

edit: I split this with my parent reply to try to make the two separate points clearer

Re: GitLab Database Incident – Live Report

#428

Post by koolba »

jfindley wrote: post #350

"The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented" This is true of many databases, but in my experience is particularly true of postgres. It's a marvelous single-instance product, but I've never really found any replication/ha tech for it that I've been that happy with. I've always been a bit nervous about postgres-backed products for thi…

Any replication / backup / restore process that is not regularly tested to verify that it works is fragile. Even more so when it's a series of manual steps to stonith, promote, etc. There's nothing particularly fragile about Postgres and there's a wealth of backup options ranging from logical (pg_dump), to replication (built-in WAL based), to physical archiving (pg_basebackup + WAL archiving). There are a number of cloud DBaaS vendors that run huge fleets of Postgres databases (Heroku and Amazon RDS come to mind). This is a solved problem.

The root of this clusterfuck for GitLab is that they didn't test anything. Looks like there was some cargo culting of "Yah we need backups so let's run something ... yada yada S3 ..." but no verification of anything actually being backed up, or error alerts for missed backups, let alone testing a full restoration from said backups.

Re: GitLab Database Incident – Live Report

#430

Post by mateus1 »

CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

Yep, I'd feel awful if I were the employee in this headline "GitLab Goes Down After Employee Deletes the Wrong Folder" [0].

It's the process and the team who are at fault.

[0] https://www.bleepingcomputer.com/news/hardware/gitlab-goes-d...

Post reply on HN