Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

411–420 of 621 posts

Re: GitLab Database Incident – Live Report

#411

Post by mverwijs »

CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

At one company I worked for we had a saying: "You're not one of the team until you've brought down the network. "

We all mess up. Much respect to gitlab for being open about.

Re: GitLab Database Incident – Live Report

#412

Post by [deleted] »

jjawssd wrote: post #410
CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

Maybe these individuals don't mind, it could just be a cultural thing.

[deleted]

Re: GitLab Database Incident – Live Report

#413

Post by nanch »

CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem.

GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names).

To be honest, through all of the comments, yours seems the most high-strung, and you're the one complaining about high-pressure situations like having your boss looking over your shoulder. Relax buddy. :)

Re: GitLab Database Incident – Live Report

#414

Post by imglorp »

kozak wrote: post #379
Fuzzwah wrote: post #234

Earlier quoted context omitted.

I didn't hear any "I agree" in that one. I found this one which does: https://www.youtube.com/watch?v=PAaMuTRGP6k&t=1h7m30s

Does the phrase "I'll show you" also have a predefined special meaning? He seems to repeat it quite often.

I heard that as "I show you xx". One guy is flying with his head up, the other one talking most is monitoring with his head in the instruments, and helping the pilot flying getting confident data.

Remember they're driving 1970's technology, redundant everything, and they all grew up flying "steam gauges", where the culture includes tapping on the glass to make sure the needle didn't stick. They want to compare every sensor output for sanity so they can disregard one if needed.

This vid is also the source of cockpit audio for the FSim shuttle simulator game if you like this stuff.

Re: GitLab Database Incident – Live Report

#415

Post by CrLf »

nanch wrote: post #413
CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…

In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault.

Transparency doesn't mean publicly throwing people under the bus.

I'm not a GitLab customer, I'm relaxed. :)

Re: GitLab Database Incident – Live Report

#416

Post by dmichulke »

jfindley wrote: post #362
dijit wrote: post #353

Earlier quoted context omitted.

I don't share your observation at all. If we're talking normal replication then I can tell you for certain that the built in replication system is completely rock solid (especially compared to MySQL). If you're talking multi-master replication, then Citus is pretty solid, but not nearly as solid as replication. If you're talking statement based replication, well, it's the same as all databases. Here be dragons.

The built-in replication may be good - but it's pretty new. Replication slots, which are IMO vital for making the built-in replication non-fragile, only arrived in 9.4, which is pretty recent of a release. I wonder how widely tested it is, given that? How many people are actually using it for large workloads? Citrus I hadn't heard of though - that's interesting, thanks.

> only arrived in 9.4, which is pretty recent of a release.

Release Date: 2014-12-18 (from https://www.postgresql.org/docs/9.4/static/release-9-4.html)

Two years do surely not confer the status of "battle-tested" but I wouldn't call it "recent" either. Then again, DB service level and a standard application service level might differ by a few nines/sigma here.

Is it that what you're referring to?

Re: GitLab Database Incident – Live Report

#417

Post by mahnunchik »

antocv wrote: post #354
sytse wrote: post #48

Earlier quoted context omitted.

Thanks, not feeling great about the job we're doing today, but we'll learn from this.

The way you are open about this and do not blame the engineer who did "rm -Rvf" (he knows he fucked up, suffers enough already), and seeing that improvements can be made and you're willing to do them. Applying for work with you now, and moving all my stuff to GitLab.

Move files to GitLab: git status & rm -Rvf .

Re: GitLab Database Incident – Live Report

#418

Post by avar »

CrLf wrote: post #415
nanch wrote: post #413

Earlier quoted context omitted.

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…

In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)

That guy is going to be interviewing at some company with someone who's obsessive enough about outage reports to remember a then-obscure one years later, but enough of an idiot to not understand that people aren't personally to blame for this sort of stuff?

Sounds like even in that very contrived scenario the guy involved would dodge a bullet in not being hired by a bunch of idiots.

Re: GitLab Database Incident – Live Report

#420

Post by OskarS »

CrLf wrote: post #415
nanch wrote: post #413

Earlier quoted context omitted.

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…

In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)

Honestly, if I were interviewing the guy, that would almost be a bonus! Like, everyone makes mistakes, we're all human, but I can guarantee you that THAT person will never make that particular mistake ever again. And he's going to be 10 times more diligent than the average engineer in making sure there are good backup/restore procedures.
Post reply on HN