Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

571–580 of 621 posts

Re: GitLab Database Incident – Live Report

#571

Post by neurotech1 »

sgift wrote: post #501
songzme wrote: post #493

Earlier quoted context omitted.

I've heard this anecdote before and it never sat well with me. Forgetting to fuel a plane as a plane mechanic exposes a serious character flaw that could lead to something devastating if allowed to continue (perhaps next time he forgets to oil the engine? Grease the brakes?). Sensationalizing this story could actually do alot of harm. The plane mechanic should have been fired for failing such an important task. If he…

When people are afraid to loose their jobs if they make an error you can be pretty sure they will do everything in their power to hide the fact that they made an error, which is the exact opposite of the behavior you want. To allow process improvements it must be absolutely clear that errors will not be punished, but used to help everyone to learn.

The JAL 2 mishap is legendary in the aviation world. Learning from mistakes is a big part of aviation safety

https://en.wikipedia.org/wiki/Japan_Airlines_Flight_2#The_.2...

The Captain basically got up before the NTSB and when asked what happened, he responded "I F__ked Up!" instead of trying to deflect blame onto an unforeseen system glitch or other excuse. Its since been known as the "Asoh Defense"

They also have the NASA ASRS for reporting near misses, and incidents without fear of FAA enforcement.

https://en.wikipedia.org/wiki/Aviation_Safety_Reporting_Syst...

Re: GitLab Database Incident – Live Report

#572

Post by chousuke »

weavie wrote: post #386
theptip wrote: post #18

Earlier quoted context omitted.

Also, as a safety net, sometimes you don't need to run `rm -rf` (a command which should always be prefaced with 5 minutes of contemplation on a production system). In this case, `rmdir` would have been much safer, as it errors on non-empty directories.

Alternatively, if you have the luxury - `zfs snapshot`.

Having a snapshotting storage system (NetApp) once saved a lot of pain when I accidentally deleted the wrong virtual machine disk from an internal server (hit the system disk instead of a removed data disk) I was able to recover the root disk from a snapshot and bring up the machine in less than an hour.

Snapshots are not a backup strategy, but they make me sleep better at night regardless.

Re: GitLab Database Incident – Live Report

#573

Post by scaryclam »

YorickPeterse wrote: post #476
CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.

I think perhaps, you want to not do this in the future.

Incident reports are about focusing on the "what" and "when" not the "who". This is not about taking responsibility (you don't need to be published on the internet to do that) and you can always have a follow up post after the incident report has been published as a "what I learned during incident X".

While it's great you're OK with publishing your name, you've now set a precedent that says it's OK to do this to other developers. A blanket policy on keeping names out of the incident report protects others who may not be as willing to get their name of HN (as well as not having to make amendments or retractions if the initial assumptions are incorrect). It also keeps a sense of professionalism as it's clear that no blame is being assigned. I know that you guys are not assigning blame, but if I was to show this to someone outside of this discussion, they'd assume that it was a fingerpointing exercise, which does not reflect well on Gitlab.

Re: GitLab Database Incident – Live Report

#574

Post by tveita »

wolfgang42 wrote: post #553
luckystartup wrote: post #504

I have at least 10 private repos on GitLab, and many public ones. Even so, this is no big deal to me. That's the beauty of git. Even if all of their backups fail, I can just do a push and everything is back up there. I just hope my laptop doesn't die before they get it back online. EDIT: Was fun to put this little command together. Run this from your code directory, and it will push all of your gitlab repos. I'm goin…

From the incident report: > Git repositories are NOT lost, we can recreate all of the projects whose user/group existed before the data loss, but we cannot restore any of these projects issues, etc. Your fancy snippet will report that it has pushed no changes. The data that was lost was new issues, PRs, issue comments, and so on; I've never heard of anyone keeping backups of these on their local laptops.

Time for a hosted Fossil service? :)

http://fossil-scm.org

(Actually, googling that it turns out one already exists...)

Re: GitLab Database Incident – Live Report

#575

Post by scaryclam »

mastre_ wrote: post #549
jawilson2 wrote: post #483

Earlier quoted context omitted.

That's awesome, but why publicize it? This isn't an act of contrition for you, no one outside your team really needs to see your dirty laundry, and actually comes off as unprofessional to me. The gitlab team is a team, and you take responsibility as a team. Placing names and initials in the liveblog makes it look like SOMEONE is trying to assign and pass off blame, even if that is not what is happening. Presumably in…

IMO the idea of secrecy equating to professionalism is _the_ problem with many things. "Information wants to be free." It's also more personable, especially to those who use their product - to me, it shows they're on top of it, they care and are taking responsibility. Gives you a sense like you're part of the team (or they part of yours).

Keeping names out of incident reports isn't about secrecy. There's nothing stopping the folks at Gitlab posting up a retrospective blog post. Incident reports are formal documents published to let users and customers know what's going on. The names can come later, if all parties are OK with it.

Re: GitLab Database Incident – Live Report

#577

Post by musicmatze »

procrastitron wrote: post #548
musicmatze wrote: post #318

They say that git data (repos and wikis) are not affected... well ... if they would have had their PRs and Issues in git repositories, too... Disclaimer: Worked on a POC for exactly this last semester and going to publish my results in the next few weeks.

GitLab has an open issue ( https://gitlab.com/gitlab-org/gitlab-ce/issues/4084 ) to use git-appraise ( https://github.com/google/git-appraise ) for storing pull requests in the repository.

I just checked git-appraise and while it looks rather mature, it uses (as far as I can see by now) an approach which is not that nice if you have multiple public remotes. Also, as far as I can see, each submitter must be able to push to that remote - please correct me if I'm wrong.

We have a different approach for this, which is more powerful (talking about how the data is stored).

Of course, our tool is not mature yet. Maybe gitlab can learn from what we've researched...

Re: GitLab Database Incident – Live Report

#578

Post by musicmatze »

jschulenklopper wrote: post #480
musicmatze wrote: post #318

They say that git data (repos and wikis) are not affected... well ... if they would have had their PRs and Issues in git repositories, too... Disclaimer: Worked on a POC for exactly this last semester and going to publish my results in the next few weeks.

> if they would have had their PRs and Issues in git repositories, too Well, perhaps a git repository isn't the right persistency structure for non-code items like issues and pull requests. That a git repository is a good structure for document-like entities (like code files) and tracking changes/versions, doesn't mean that it's a good choice for highly-relational or highly-dynamic objects.

Well, from what we've experienced during the implementation and while using our tool, it works _really_ well!

Re: GitLab Database Incident – Live Report

#579

Post by dotancohen »

hobarrera wrote: post #134
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.

On some machines I've actually masked `rm` with an echo to remind me not to use it, and I would delete with `rrmm`. That would give me pause to ensure that I really mean to remove what I'm removing, and more importantly, that I didn't type `rm` when I meant `mv` (which I actually have done by accident).

Re: GitLab Database Incident – Live Report

#580

Post by rhizome »

scaryclam wrote: post #573
YorickPeterse wrote: post #476

Earlier quoted context omitted.

I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.

I think perhaps, you want to not do this in the future. Incident reports are about focusing on the "what" and "when" not the "who". This is not about taking responsibility (you don't need to be published on the internet to do that) and you can always have a follow up post after the incident report has been published as a "what I learned during incident X". While it's great you're OK with publishing your name, you've…

I think you're blowing it out of proportion. If you showed it to someone and they told you they assumed it was about fingerpo...Look, it's not that big a deal. They decided to do it, not everything is a blame game.
Post reply on HN