Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

blog.ycombinator.com

Post reply on HN
Print view View original

61–70 of 320 posts

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#61

Post by Spooky23 »

cookiecaper wrote: post #41
snarf21 wrote: post #19

Earlier quoted context omitted.

"It's hard to build solid process, particularly in growth phases." Hard is an understatement, it is insanely difficult. However, the interview seems to suggest that "write everything down" is the way to solve this problem and was what allowed for all their growth and success. So the solid process they have built is to write everything down, which they obviously didn't do or they wouldn't have had a 7 layer disaster r…

"Write everything down" is a terrible process, for two main reasons. First, no one has the discipline to write much of anything down. It's a very boring process and it's always going to be low-resolution. Your most meticulous documentation writers will get fired for failing to get their "real work" done. If you personally recognize the value of the documentation they furnish and thus refuse to fire them, all of their…

These are human problems. Doctors used to bitch about shit like this, because they felt it was beneath them to follow checklists. As a result people have had limbs removed and life threatening, unnecessary surgery. Guess what? The insurance companies demand checklists and controls to prevent fuckups.

There is an easy solution to dealing with people who refuse to read things. (Hint: It doesn't involve recording hours of meetings.) You need standard operating procedures to document what people do at an appropriate level of detail. Period.

Operational procedure is different than code documentation.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#62

Post by danpalmer »

bgxor wrote: post #43
cookiecaper wrote: post #8

Earlier quoted context omitted.

I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my C…

Those salary modifiers are atrocious. Their range in my city for a senior, very experienced engineer is comparable to what most entry level jobs actually offer around here. I 100% think you are on the right track. The only people from my area that would take a GitLab salary just can't get a job elsewhere.

Yeah, I put in some pretty conservative estimates for my skills/experience/etc, and got out a salary estimate that is not good for my area (London, UK). I tried the city I went to university in and there is no way they would be competitive there with other companies.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#63

Post by scaryclam »

pc86 wrote: post #47
cookiecaper wrote: post #8

Earlier quoted context omitted.

I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my C…

Just ran my numbers and got about 55% of my current salary assuming a lead position (what I have now, which I would probably not get with them) and "a lot of experience" which I'm on the fence about and they would probably disagree with. Getting rid of the COL deduction by selecting NYC as my location actually makes it reasonable so something tells me not pinning the US salary floor to 1.0 is saving them money and si…

Having that sort of calculator also encourages developers to embellish their competancy level as well, which is not going to help in finding quality developers. I'd much rather a company figure out where I am on their scale (each company I've worked at seems to have different ideas about what makes a junior/mid-level/senior/lead) than taking my word for it.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#64

Post by detaro »

geofft wrote: post #38
detaro wrote: post #30

Earlier quoted context omitted.

The "trigger" of the failure (accidentally deleting the wrong database) is bad luck that happens, but the lack of preparedness for it isn't. If I understood the document right even if the regular backups would have worked, they only happened every 24 hours (same as the staging replicas they recovered from, except one was manually created out of schedule that day). Maybe being fully remote helps to let stuff like this…

What's the best-practice approach for taking backups that are significantly more frequent than every 24 hours, but also robust to things like an rm -rf or a DELETE FROM table;? Something like continuous data protection seems like it would be far too much data for an active database server, no? (Or are we just saying that they should have been taking backups every 15 minutes or hour or so?)

I'm not really qualified to confidently talk about best practices, so these points have question marks attached, since I can't judge what's possible in their setup and what's not. I guess I'm not sure if they did judge it, see last paragraph.

More frequent backups would of course help, if their impact is tolerable. They also need to be tested to actually exist and work.

At least LVM snapshots apparently are cheap enough that they can be done out-of-schedule just to get slightly newer data to staging, so they likely could have been done more often (but they probably weren't thought of as backups, which is why 24 hours seemed enough and nobody had a prepared plan to get back to production from them).

Similarly, Azure-side snapshots are mentioned as not enabled for the database hosts. Maybe that's not viable to do (they had performance issues with Azure, and I don't know how much overhead they case), maybe just something they forgot to set up.

Others have asked "why is there only one replica", which also seems like a good question (but my experience with database replication is close to non-existent, and maybe the higher load of more replicas would have caused other issues. Don't know.).

My point is more that I don't have the impression that the 24 hours are a figure that they arrived on by evaluating what "service level" they could achieve, but more a result of someone at some point setting up a backup with some interval. I don't think 24 hours is a figure where they can say "that's the best we currently can achieve", or even "that's what we planned for", but "that's what we have because that's what we have and nobody has paid closer attention". There is at the very least a cultural angle to it, which is why how they work could be relevant.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#65

Post by cookiecaper »

pc86 wrote: post #47
cookiecaper wrote: post #8

Earlier quoted context omitted.

I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my C…

Just ran my numbers and got about 55% of my current salary assuming a lead position (what I have now, which I would probably not get with them) and "a lot of experience" which I'm on the fence about and they would probably disagree with. Getting rid of the COL deduction by selecting NYC as my location actually makes it reasonable so something tells me not pinning the US salary floor to 1.0 is saving them money and si…

Yeah, same here. If I max out the possible compensation by setting lead w/ a lot of experience, it's roughly 60% of what I make now (as a non-lead with a lot of experience).

The kicker is that I'm employed remotely by by a company based in a city whose CoL hit is even larger than the one I currently live in. The NYC range they give is plausible for me as someone who lives in a place that is much cheaper than NYC, but it's not exciting and wouldn't really motivate me to apply.

GitLab's numbers may look right on paper (and they do explain their methodology in the second link, discussing rent indexes, etc), but they're absolutely not comparable to what good people earn in the real world. Good talent commands good wages everywhere. A skilled remote worker is not going to come to GitLab for an average-for-their-area salary.

Even when working locally, talented people make much more than the average for their area. I know this because I've hired many good people and unless they're entry-level (meaning they're good but haven't had a chance to prove it yet), they don't start coming in the door until you're offering at least a 50% price premium over the reported median. In fact, we had a lot of bad candidates come in asking for around that much. The senior people I hired would make more than double the median for their nominal role. The stats on these government reports don't account for seniority, skill, niche demand, etc., and likely include some people that self-report as holding a title when they're really just aspiring to that position.

Once you're a mid-level dev, you should be pulling in a bare minimum of 80-85k, no matter where you live. I say this not as a bubble-dwelling San Franciscan (who surely finds such numbers laughable even for entry-level), but someone who has lived in various parts of "flyover country" his entire life. When local opportunities for at least this compensation are not forthcoming, go contract or remote.

There's no reason for good people to leave money on the table, and GitLab doesn't appear to get that.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#66

Post by notcolin »

Perihelion wrote: post #32
cjbprime wrote: post #5

Earlier quoted context omitted.

Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.

Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things. But you're right, the timing of this piece was probably not ideal.

> This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process.

Could you elaborate? I recently heard about GitLab probably less than a week ago and was under the impression they're a great company to work for.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#67

Post by Perihelion »

cookiecaper wrote: post #41
snarf21 wrote: post #19

Earlier quoted context omitted.

"It's hard to build solid process, particularly in growth phases." Hard is an understatement, it is insanely difficult. However, the interview seems to suggest that "write everything down" is the way to solve this problem and was what allowed for all their growth and success. So the solid process they have built is to write everything down, which they obviously didn't do or they wouldn't have had a 7 layer disaster r…

"Write everything down" is a terrible process, for two main reasons. First, no one has the discipline to write much of anything down. It's a very boring process and it's always going to be low-resolution. Your most meticulous documentation writers will get fired for failing to get their "real work" done. If you personally recognize the value of the documentation they furnish and thus refuse to fire them, all of their…

> First, no one has the discipline to write much of anything down.

This is why it's baked into the company culture. It's very common for someone to ask where something is in the handbook or if an issue has been created for something.

> It's a very boring process and it's always going to be low-resolution. Your most meticulous documentation writers will get fired for failing to get their "real work" done. If you personally recognize the value of the documentation they furnish and thus refuse to fire them, all of their peers will feel that they are dead weight, which is still a ticket out of the company.

Fwiw, everyone is responsible for maintaining the handbook/our process and procedure documentation. The docs team isn't on the hook for it, nor is it the sole responsibility of engineering.

> Second, after all that work, no has the discipline to read much of anything that you've written!

After enough reminders, you'd be amazed at how quickly people learn to RTFM at work.

> They skim. They glance. They Ctrl+F. They don't read. When you're in a pickle and you can pick out a life-saving bit of documentation, it's amazing, but that happens quite rarely and requires a lot of energy.

It's true that people skim the handbook (the guide with all of the "Here's how you get access to Twitter accounts" stuff), but runbooks are actually looked at when stuff hits the fan. I think it's important to differentiate these two things since they have different purposes. Imo runbooks should be as lean as possible for that very reason.

> Good programmers are both lazy and dumb!

This level of documentation is helpful for non-developers who make up a significant part of many organizations. We're not just talking about documenting code here.

> If you want documentation that means something, it needs to be part of the process of actually working.

100%, this is the only way it works.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#68

Post by the_commentary »

cookiecaper wrote: post #8
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my C…

I would agree with this -- Gitlab seems like a company with a great philosophy and I'm sure a lot of people would love to help them achieve their goals... but I just used the calculator for my own area and my pay would be around half what I'm getting currently. I have to assume that makes it really hard to get serious talent, and makes it more of a pass-through for people to get their foot in the door.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#69

Post by connorshea »

bryanh wrote: post #58

160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate. We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural -…

I didn't know Zapier was a fully remote company, that's awesome :)

I pretty much agree with your thoughts about over-communicating, it's super important and we've run into exactly the same things, where people aren't always caught up on everything going on at the company. In my experience this happens in physical workplaces as well, just to a lesser extent.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#70

Post by geofft »

fsiefken wrote: post #57
problems wrote: post #22

Earlier quoted context omitted.

> didn't do anything particularly more wrong than anyone else? This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101. Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple…

As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually. GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime…

Synchronizing the database isn't quite what you want. It's true that in the case of an errant rm -rf it would almost certainly have helped, but it's approximately as easy to run a "DELETE FROM importantdata" and leave off the "WHERE" clause, which would get replicated. And certainly if you're using DRBD (replicate the volume, not the database), an rm -rf will get replicated.

I'm just genuinely unsure what a better outcome would have been here. (It's certainly a process failure that no backups existed other than the manual 6-hour-old snapshot, but I'm not sure you can do much better than automating that.)

Post reply on HN