Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

blog.ycombinator.com

Post reply on HN
Print view View original

21–30 of 320 posts

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#21

Post by bpicolo »

zenlikethat wrote: post #11
deepaksurti wrote: post #4

>> 2:41 – GitLab values boring solutions: our product should be exceptional Exceptional products have exceptional UX. Gitlab IMHO has the worst UX of all git based products out there, I much rather take BitBucket over Gitlab. I tried using Gitlab, but no, I would much rather pay the 7$ to GH for my private repos. I sincerely hope they make an exceptional product. And 'should' better be 'must'!

I'm curious what you dislike about the UX. Personally I like the "feel" of the website generally (though it's not as polished as GitHub) and just find it to be slow, which I think they're working on improving.

Personally, not a fan of how a readme-less repository just has an entirely useless main page. I much prefer having the file view a'la github on that page. That's the main UX complaint for me. Still use gitlab bunches.

Main complaint overall is the incredibly sluggish git push speed.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#22

Post by problems »

geofft wrote: post #15
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

> didn't do anything particularly more wrong than anyone else?

This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101.

Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple would have solved their entire issue. I do this and we run a much smaller shop than GitLab. Heck, if we were larger I'd have hot spare database servers in another datacenter in case the primary got nuked by disk failure, network outages or mistakes.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#23

Post by Perihelion »

deepaksurti wrote: post #4

>> 2:41 – GitLab values boring solutions: our product should be exceptional Exceptional products have exceptional UX. Gitlab IMHO has the worst UX of all git based products out there, I much rather take BitBucket over Gitlab. I tried using Gitlab, but no, I would much rather pay the 7$ to GH for my private repos. I sincerely hope they make an exceptional product. And 'should' better be 'must'!

Hey Deepak, thanks for the feedback. We've built out a UX team and improving that aspect of the product is high on our list of priorities, but please feel free to reach out if you have any specific issues. Happy to document those for you.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#24

Post by wheelerwj »

geofft wrote: post #15
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

Look, everyone is supportive of gitlab and how they recovered. but to come out boasting after being down for that long is a hard sell.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#25

Post by sirclueless »

mbiondi wrote: post #14
Spooky23 wrote: post #7

Earlier quoted context omitted.

Not really germane to the topic. This type of op fuck-up happens everywhere. It's hard to build solid process, particularly in growth phases. Unless there are 2x a year restore tests, I personally assume a 60% backup fail rate.

The only reason they are still in existence is due to a chance backup they took for a tangential reason. From the sounds of it, their solution is held together with bubble gum, some tape and lots of hand waving. Being in 160 different locations probably doesn't help much either.

Unless I'm misreading things, the reason they only lost 6 hours of data instead of 24 hours was a chance backup, but there was never an existential crisis here.

Downtime happens to pretty much every service out there. In this case the company was incredibly forthright and so we can make fun of their stupid mistakes, but really most mistakes are stupid when you look at them -- when you make thousands of decisions a day, some of them will seem silly in hindsight. We just never learn about most of them.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#26

Post by Spooky23 »

snarf21 wrote: post #19
Spooky23 wrote: post #7

Earlier quoted context omitted.

Not really germane to the topic. This type of op fuck-up happens everywhere. It's hard to build solid process, particularly in growth phases. Unless there are 2x a year restore tests, I personally assume a 60% backup fail rate.

"It's hard to build solid process, particularly in growth phases." Hard is an understatement, it is insanely difficult. However, the interview seems to suggest that "write everything down" is the way to solve this problem and was what allowed for all their growth and success. So the solid process they have built is to write everything down, which they obviously didn't do or they wouldn't have had a 7 layer disaster r…

"Write everything down" is the ante to function at all!

I agree... but I guess I read that coming from the perspective of someone who isn't a super fan of distributed teams. Many teams think that futzing around in slack or hangouts is enough. "Writing things down" is practically discovering the wheel to some places!

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#27

Post by eknkc »

bpicolo wrote: post #21
zenlikethat wrote: post #11

Earlier quoted context omitted.

I'm curious what you dislike about the UX. Personally I like the "feel" of the website generally (though it's not as polished as GitHub) and just find it to be slow, which I think they're working on improving.

Personally, not a fan of how a readme-less repository just has an entirely useless main page. I much prefer having the file view a'la github on that page. That's the main UX complaint for me. Still use gitlab bunches. Main complaint overall is the incredibly sluggish git push speed.

You can change that on your personal prefs to show the file list by default, with Readme on bottom: http://c.ekin.io/3z1V2v3P041b

git push speed is fine on our own ce installation but gitlab.com is beyond shitty.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#28

Post by cjbprime »

geofft wrote: post #15
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

I think you're confusing sympathy and solidarity towards their engineers (which I was very glad to see) with "wow, this could happen to anyone". Quoting from their post-mortem:

> So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

It's not true that having zero tested working backup/restore strategies happens to everyone. It's a catastrophic failure in process. Let's not blame any specific engineers, but it seems totally appropriate to blame the CEO (who was the subject of the puff piece!) and decrease your trust in the company appropriately.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#29

Post by ridruejo »

manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

Every company goes through rough patches, specially one that is growing as fast as they are doing. Very few companies are as open as transparent as they are, even in the fuckups. That is really hard to do and kudos to them

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#30

Post by detaro »

geofft wrote: post #15
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

The "trigger" of the failure (accidentally deleting the wrong database) is bad luck that happens, but the lack of preparedness for it isn't. If I understood the document right even if the regular backups would have worked, they only happened every 24 hours (same as the staging replicas they recovered from, except one was manually created out of schedule that day).

Maybe being fully remote helps to let stuff like this "slip through the cracks". (Not great phrasing, but I don't think anyone made a conscious decision "loosing X hours of data is ok", and nobody questioned what goal the current practices could (not) achieve.) I don't know, but it seems at least a question one might ask.

Post reply on HN