Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

blog.ycombinator.com

Post reply on HN
Print view View original

31–40 of 320 posts

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#31

Post by cookiecaper »

geofft wrote: post #15
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

It's unfair to characterize the entire organization by one failure, but their data loss issue is not attributable to some obscure, convoluted technical issue or freak accident. The simple fact is that they didn't verify that their backup and recovery infrastructure was working correctly. Moreover, they didn't have any automated monitoring confirming that the backups even appeared to be working. Those are things that every moderately-experienced IT person knows needs to be in place and occurring routinely.

If you look up the doc, they have like 6 different places where they looked for backups and each point is like "We should have these backups, checking for them now... woops, it hasn't been working."

The only reason GitLab's data loss wasn't even more disastrous is that an employee just so happened to take a manual snapshot 6 hours before the incident occurred.

Now, I'm sure we've all been in situations like that. I don't think it necessarily reflects on the individual technical employees of GitLab. I think it says more about the organizational and management resources at the top. It was their job to ensure that the appropriate resources were committed to data integrity, and they failed to do so.

90% of us probably work at places with similar problems, but publishing "the secret to [managing technical employees]" fresh on the heels of such a spectacular organizational failure is probably unwise.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#32

Post by Perihelion »

cjbprime wrote: post #5
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.

Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things.

But you're right, the timing of this piece was probably not ideal.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#33

Post by bpicolo »

eknkc wrote: post #27
bpicolo wrote: post #21

Earlier quoted context omitted.

Personally, not a fan of how a readme-less repository just has an entirely useless main page. I much prefer having the file view a'la github on that page. That's the main UX complaint for me. Still use gitlab bunches. Main complaint overall is the incredibly sluggish git push speed.

You can change that on your personal prefs to show the file list by default, with Readme on bottom: http://c.ekin.io/3z1V2v3P041b git push speed is fine on our own ce installation but gitlab.com is beyond shitty.

That is kind of a bizarre thing to be tucked away in settings =| Noted, though

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#34

Post by chousuke »

mbiondi wrote: post #14
Spooky23 wrote: post #7

Earlier quoted context omitted.

Not really germane to the topic. This type of op fuck-up happens everywhere. It's hard to build solid process, particularly in growth phases. Unless there are 2x a year restore tests, I personally assume a 60% backup fail rate.

The only reason they are still in existence is due to a chance backup they took for a tangential reason. From the sounds of it, their solution is held together with bubble gum, some tape and lots of hand waving. Being in 160 different locations probably doesn't help much either.

I'm pretty sure most solutions on the internet consist of bubble gum, some tape and lots of handwaving. Gitlab's screwup is hardly unique, even if it was very public.

It's difficult and expensive to build and maintain a solid system, and even if you want to, time and financial pressures often just don't let you, on top of the issue of just communicating the need for solid engineering, as it usually only becomes apparent when the problems start occurring.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#36

Post by scrollaway »

cjbprime wrote: post #5
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.

Why? It's not a puff piece boasting about their backup plans, they're talking about their remote work strategy.

What's tone-deaf is the vitriol some people are spilling in this thread.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#37

Post by onion2k »

cjbprime wrote: post #5
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.

Companies are run by people and consequently they're fallible. Every company makes mistakes occasionally. Gitlab handled theirs incredibly well. A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. Gitlab's openness means we can all learn a little from their event. I think that's great. I wish more companies shared more when these things happen.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#38

Post by geofft »

detaro wrote: post #30
geofft wrote: post #15

Earlier quoted context omitted.

"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

The "trigger" of the failure (accidentally deleting the wrong database) is bad luck that happens, but the lack of preparedness for it isn't. If I understood the document right even if the regular backups would have worked, they only happened every 24 hours (same as the staging replicas they recovered from, except one was manually created out of schedule that day). Maybe being fully remote helps to let stuff like this…

What's the best-practice approach for taking backups that are significantly more frequent than every 24 hours, but also robust to things like an rm -rf or a DELETE FROM table;? Something like continuous data protection seems like it would be far too much data for an active database server, no?

(Or are we just saying that they should have been taking backups every 15 minutes or hour or so?)

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#40

Post by overcast »

geofft wrote: post #15
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

Are you kidding? How is not ensuring your backups have EVER worked, or have even been setup, considered "bad luck", or "anything particularly more wrong". We're not even talking testing backups, literally if they have even been setup to run at all. The only reason this wasn't a complete meltdown, was that the tech decided he'd snapshot before he did work six hour earlier.
Post reply on HN