Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

301–310 of 621 posts

Re: GitLab Database Incident – Live Report

#301

Post by onion2k »

elementalest wrote: post #88

>1. LVM snapshots are by default only taken once every 24 hours. YP happened to run one manually about 6 hours prior to the outage >2. Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size. >3. SH: It looks like pg_dump may be failing because PostgreSQ…

failing silently

I really wish all the applications I use had an option to never do that.

Re: GitLab Database Incident – Live Report

#302

Post by bdcravens »

ben_jones wrote: post #293
sschueller wrote: post #282

Earlier quoted context omitted.

There is a neat video[1] where a Swiss flight has to make an emergency landing and just happens to have a film crew in the cockpit. [1] https://www.youtube.com/watch?v=rEf35NtlBLg

It's amazing how decisive both pilots are despite the large amount of process going on.

It's like frameworks in programming: it frees brain cycles to focus on what's important

Re: GitLab Database Incident – Live Report

#303

Post by rrdelaney »

hueving wrote: post #230
throwaway3347 wrote: post #209

Earlier quoted context omitted.

Like you, I would like to add my 2 cent, which I hope will be taken positively, as I would like to see them provide healthy competition for GitHub for years to come. Since GitLab is so transparent about everything, from their marketing/sales/feature proposals/technical issues/etc., they make it glaringly obvious, from time to time, that they lack very fundamental core skills, to do things right/well. In my opinion, t…

Why would they try to recruit from Microsoft? Most of the software engineers at Microsoft are not focused on developing scalable web services architectures. And the ones that do have built up all of their expertise with Microsoft technologies (.net running on Windows server talking to mssql). >Microsoft and others that have top talent/domain expertise, will have the advantage. Again, Microsoft isn't even in this same…

One of the main drivers of revenue for Microsoft is Office 365, with 23.1 million subscribers[0]. Along with Azure, MS runs some of the largest web services around. Most developers at MS don't necessarily work on these products, but to say that all the devs working on them use a simple .NET stack + SQL Server is discrediting a lot of work that they do.

Disclaimer: I work for Microsoft in the Office division and opinions are my own

[0] https://www.microsoft.com/en-us/Investor/earnings/FY-2016-Q4...

Re: GitLab Database Incident – Live Report

#304

Post by fnord123 »

gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

You missed the most important part: writ down the backout plan.

Re: GitLab Database Incident – Live Report

#305

Post by elygre »

developer2 wrote: post #246
overcast wrote: post #74

Earlier quoted context omitted.

Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…

>> candidates will be invited to schedule a first 45 minute behavioral interview with the Infrastructure Lead Yes, go right ahead and filter out some (disclaimer before the rant: some, not all) of the best talent. The kind of potential employee that gets rejected due to perceived personality problems is exactly the kind of person who would tell management to shove a stick up their ass for demanding a 2 week deadline…

Not vetting people based on behavior is perilously close to accepting the old adage "Say what you like about Mussolini, but at least he made the trains run on time".

Re: GitLab Database Incident – Live Report

#306

Post by rrdelaney »

motles wrote: post #296
lbotos wrote: post #270

Earlier quoted context omitted.

That choice to go to bare metal was reversed: https://gitlab.com/gitlab-com/infrastructure/issues/727#note... https://webcache.googleusercontent.com/search?q=cache:M2CRY7... (cache link until GL.com is back up.)

Does Azure not have managed postgres?

Azure does not provide a managed postgres service. However it does have Azure SQL[0], which is based on SQL Server.

Disclaimer: I work for Microsoft

[0] https://azure.microsoft.com/en-us/services/sql-database/

Re: GitLab Database Incident – Live Report

#307

Post by PKop »

sschueller wrote: post #282
ryandrake wrote: post #185

Earlier quoted context omitted.

If you get the chance to observe pilots operating in the cockpit, I'd recommend it. Every important procedure (even though the pilot has it memorized) is done with a checklist. Important actions are verbally announced and confirmed: "You have the controls" "I have the controls". Much of flight training deals with situational awareness and eliminating distractions in the cockpit. Crew Resource Management[1]. 1: https:…

There is a neat video[1] where a Swiss flight has to make an emergency landing and just happens to have a film crew in the cockpit. [1] https://www.youtube.com/watch?v=rEf35NtlBLg

Here's a great documentary [0] by Errol Morris about the United Flight 232 crash in 1989 [1].

"..the accident is considered a prime example of successful crew resource management due to the large number of survivors and the manner in which the flight crew handled the emergency and landed the airplane without conventional control."

I highly recommend it

[0] https://www.youtube.com/watch?v=2M9TQs-fQR0

[1] https://en.wikipedia.org/wiki/United_Airlines_Flight_232

Re: GitLab Database Incident – Live Report

#309

Post by echelon »

gnufied wrote: post #56
echelon wrote: post #44

Earlier quoted context omitted.

This doesn't really help if there are multiple production databases. It could be sharded, replicated, multi-tenant, etc.

Why would it matter? In my last job we had user home directories synced via puppet (I am overly simplifying this) which enabled any ops guy to have same set of shell and vim configuration settings on production machines too. I daresay - having hostname as part of prompt saves lot of trouble.

Having the hostname on the prompt is a good idea, but I don't think it would have helped with this process failure.

I work at a company where we have hundreds of database machines. Running this kind of command _anywhere_ without some kind of plan would be foolish. (It's one of the reasons why we have a datastore team that handles database administration.)

But the same lesson applies to application servers as well. Don't run deleterious commands out of curiosity. Have a peer-reviewed roll plan to act on when doing things like this. A role plan would have called for verifying the host before running the command.

But even before that, the issue should have been investigated more!

All of these things contributed to the failure. There should ideally be better ownership through dedicated roles, peer-reviewed processes for dangerous activities, and a better process for investigation that does not involve deleting things haphazardly.

Post reply on HN