Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

321–330 of 621 posts

Re: GitLab Database Incident – Live Report

#321

Post by atmosx »

foota wrote: post #288
atmosx wrote: post #274

Earlier quoted context omitted.

Maybe add a bash function to check the path and ask the magic question: "Do you really want to delete the dir XYZ in root@domain.com?" .... but then again when you're in panic mode you might either misread the host or hit 'y' without really reading what's in front of you. The best thing to do is to never operate with 2 terminals simultaneously, when one of them is a production env, better login/logout or at least min…

Perhaps, Do you really want to delete the dir XYZ with 300000000000 bytes in it?

The problem in this case was that YP confused the host. YP thought he was operating in db2 (the host that went out of sync) and not db1 (the host that held the data), a message that doesn't display the current host wouldn't help in this case.

Re: GitLab Database Incident – Live Report

#322

Post by catwell »

ryandrake wrote: post #185
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

If you get the chance to observe pilots operating in the cockpit, I'd recommend it. Every important procedure (even though the pilot has it memorized) is done with a checklist. Important actions are verbally announced and confirmed: "You have the controls" "I have the controls". Much of flight training deals with situational awareness and eliminating distractions in the cockpit. Crew Resource Management[1]. 1: https:…

Great operations teams have incident response procedures based on checklists and clear communication channels like that as well. If this interests you I recommend David Mytton's talk at dotScale 2015: https://www.youtube.com/watch?v=4qGcTOQRvEU

Re: GitLab Database Incident – Live Report

#323

Post by rgawdzik »

virtuabhi wrote: post #85
click170 wrote: post #75

Earlier quoted context omitted.

BBC's Horizon has a really good episode about checklists and how they're used to prevent mistakes in hospitals, and how they're being adopted in other environments in light of that success. It's called How To Avoid Mistakes In Surgery for the interested.

Here's the book on this topic- The Checklist Manifesto https://smile.amazon.com/Checklist-Manifesto-How-Things-Righ...

A must read if you don't want to make mistakes, it's given to all new Square employees.

Re: GitLab Database Incident – Live Report

#324

Post by imsodrunklol »

late2part wrote: post #217
schleppy_oc wrote: post #208

Earlier quoted context omitted.

The best Ops people I have worked with (looking at you Dennis and Alan) repeat everything back that I say. More than once I have caught mistakes in my approach simply by hearing someone else repeat back exactly what I just said.

The best ops people have made all the mistakes they're helping you avoid.

Amen brother.

My worst nightmares are nothing but a blimp on a developers mind. I've have lost it all. I have lost it all on multiple servers. On multiple volumes.

No one should ever experience that. Ugh, developing, sometimes, is just so frustrating.

Re: GitLab Database Incident – Live Report

#325

Post by rmc »

NamTaf wrote: post #269
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

Two points: 1) Patio11 touches on a very good lesson, in passing, in an article about Japanese business[1]: While raw programming ability might not be highly valued at many Japanese companies, and engineers are often not in positions of authority, there is nonetheless a commitment to excellence in the practice of engineering. I am an enormously better engineer for having had three years to learn under the more senior…

> When you wonder why big orgs are glacially slow compared to more nimble startup competitors, understand that those startups have yet to experience the Bad Things that the big org has probably already endured.

Another explaination for big orgs vs teeny start ups: What level of failure is acceptable? For a teeny start up, a few hours being down is not so important. For (say) a bank, being down for a few hours might be mentioned in the national newspapers.

Re: GitLab Database Incident – Live Report

#326

Post by rleigh »

TimWolla wrote: post #50
echelon wrote: post #40

Earlier quoted context omitted.

Does this mean whatever was in that database is gone, with no available backups? Is this an SOA where important data might lie in another service or data store, or is this a monolithic app and DB that is responsible for many (or all) things? What was stored in that database? Does this affect user data? Code?

The doc says that there is a LVM snapshot being 6 hours old. And there should be a regular logical backup with at most 24 hours age as well (they just can't find it for whatever reason). (Scratch that, my doc did not update, despite Google saying it should automatically update). Regarding what's gone: The production PostgreSQL database. This suggests that the code itself is fine, but the mappings to the users are gon…

And LVM snapshots aren't exactly the most reliable way to do a filesystem backup either, in terms of filesystem consistency, data consistency or reliability of the snapshot itself. And even kernel locking bugs when creating and destroying them, plus udev races when it fires off events when changes occur. I stopped using them years ago due to the sheer numver of problems.

Have gitlab considered something like ZFS snapshots? They don't have the consistency problems or the space wasting and reliability problems that LVM brings. You could have it take snapshots every few minutes.

Re: GitLab Database Incident – Live Report

#327

Post by bpchaps »

danielsamuels wrote: post #283
kelnos wrote: post #227

Earlier quoted context omitted.

You shouldn't let yourself get to that point. Your alerting system should alert you when disk is at 70% or something with a ton of margin. If it's not set up that way, go stop what you're doing and fix that. (Seriously.) If you're running your systems so they usually run at 90% disk usage, go give them more disk (or rotate logs sooner). And even assuming all that fails and I'm in that situation where I have seconds u…

> You shouldn't let yourself get to that point. Your alerting system should alert you when disk is at 70% or something with a ton of margin. What would your recommendation be?

I'm curious, too. Proper disk space management and monitoring is probably the most difficult problem I know of in the ops field. I haven't seen anybody do it in a way that prevents 3am wakeup calls or a 24/7 ops team.

For example, a 3am network blip that causes the application server (still logging in DEBUG from the last outage) to fill up its log partition while it can't communicate to some service nobody monitors anymore. Not sure how you'd solve that one.

Re: GitLab Database Incident – Live Report

#328

Post by arximboldi »

hobarrera wrote: post #134
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.

This goes on my machines:

    sudo apt install trash-cli
    alias rm="echo This is not the command you want to use"
This way I managed to get unlearn my rm trigger-happyness and use trash instead. I had too many incidents already... ':|

Re: GitLab Database Incident – Live Report

#329

Post by nstj »

YorickPeterse wrote: post #10
CameronBanga wrote: post #5

Earlier quoted context omitted.

I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.

Nope, that would be me.

We've all been there! Feel better man, it could be a lot worse :)

Re: GitLab Database Incident – Live Report

#330

Post by reis4g »

YorickPeterse wrote: post #10
CameronBanga wrote: post #5

Earlier quoted context omitted.

I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.

Nope, that would be me.

Was just trying to push earlier today and found out about the issue. Sorry man! Drinks on me in Montevideo, Uruguay. This stuff happens, more than most of us are willing to accept so, here is for your transparency and you know, fix it, learn it and on you go!
Post reply on HN