Wow. You all in the comments must be a bunch of perfect people, huh? Good grief. Also, nice cameo by the office cat.
Imperfect doesn't mean you can be completely negligent of basics.
GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
51–60 of 320 posts
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#52Earlier quoted context omitted.
Not really germane to the topic. This type of op fuck-up happens everywhere. It's hard to build solid process, particularly in growth phases. Unless there are 2x a year restore tests, I personally assume a 60% backup fail rate.
The only reason they are still in existence is due to a chance backup they took for a tangential reason. From the sounds of it, their solution is held together with bubble gum, some tape and lots of hand waving. Being in 160 different locations probably doesn't help much either.
I'm not defending them -- but that is the norm.
I had a customer once in the 90s take a 30 hour outage that cost them nearly $6M in fines because some asshole put a budget freeze on anything related to cleaning, including tape drive cleaner carts. The dopey ops guy kept using one tape on multiple drives, making them do nothing.
I could personally rattle off a dozen stories like this at late stage startups, Fortune 10 and .gov.
The only reason many businesses are alive is luck and reliable SAN.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#53Everybody is prone to making a mistake! As long as you learn something from it, so that it never happens again, it's ok. And I really don't get why people give them bad rep for this post. I mean it's obvious why it's posted and we all know what happened week ago, but consider their size and scale, it's not easy to manage so many people remotely.
"...but consider their size and scale, it's not easy to manage so many people remotely." This is very true but the title of the post is "GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down".
That is the backlash, don't say you've figured out the secret then make a near catastrophic mistake.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#54Earlier quoted context omitted.
"It's hard to build solid process, particularly in growth phases." Hard is an understatement, it is insanely difficult. However, the interview seems to suggest that "write everything down" is the way to solve this problem and was what allowed for all their growth and success. So the solid process they have built is to write everything down, which they obviously didn't do or they wouldn't have had a 7 layer disaster r…
"Write everything down" is a terrible process, for two main reasons. First, no one has the discipline to write much of anything down. It's a very boring process and it's always going to be low-resolution. Your most meticulous documentation writers will get fired for failing to get their "real work" done. If you personally recognize the value of the documentation they furnish and thus refuse to fire them, all of their…
Right, and the biggest problem is that they never get updated so they don't match the reality of today's system.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#55Earlier quoted context omitted.
You can change that on your personal prefs to show the file list by default, with Readme on bottom: http://c.ekin.io/3z1V2v3P041b git push speed is fine on our own ce installation but gitlab.com is beyond shitty.
That is kind of a bizarre thing to be tucked away in settings =| Noted, though
Where else would you put it?
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#56Earlier quoted context omitted.
Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.
Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things. But you're right, the timing of this piece was probably not ideal.
Not ideal perhaps but on the other hand one bad mistake doesn't negate everything you do or say forever that would be extremely unforgiving.
...and it's not like any large web company hasn't had some howlers over the years.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#57Earlier quoted context omitted.
"This" being remote work? Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)
> didn't do anything particularly more wrong than anyone else? This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101. Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple…
As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually.
GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime.
They have probably documented their own setup somewhere.
"Automated failover can be achieved with pacemaker alongside STONITH network management. Keep in mind that application servers need to be prepared for transitioning to the new network addresses.
In this situation you can also opt to synchronize the database via a database specific protocol instead of DRBD. In the documentation for each database you can find out more about the options for MySQL and the options for PostgreSQL." https://about.gitlab.com/high-availability/#filesystem-stora...
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#58160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate.
We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural - organizations become complex as they grow, and individuals are busy doing their thing. You often have to bring the important data to them.
On another note, working remote is awesome. I recommend everyone give it a spin once in their careers - but try to find a team that embraces it. I've heard mixed experiences from those who were the single remote person on a team.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#59After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.
Every company goes through rough patches, specially one that is growing as fast as they are doing. Very few companies are as open as transparent as they are, even in the fuckups. That is really hard to do and kudos to them
Thanks. I do agree that the timing of this post is not good. The video was recorded a few weeks ago. We are eating humble pie right now and work on fixing things instead of promotion.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#60After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.
I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my C…
They have picked a strategy consisting in hiring large numbers of less experienced developers, rather than a few great ones.
This is very much impossible to reverse, so now, they have to make it work with the people they have.