Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

blog.ycombinator.com

Post reply on HN
Print view View original

71–80 of 320 posts

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#71

Post by cookiecaper »

Spooky23 wrote: post #61
cookiecaper wrote: post #41

Earlier quoted context omitted.

"Write everything down" is a terrible process, for two main reasons. First, no one has the discipline to write much of anything down. It's a very boring process and it's always going to be low-resolution. Your most meticulous documentation writers will get fired for failing to get their "real work" done. If you personally recognize the value of the documentation they furnish and thus refuse to fire them, all of their…

These are human problems. Doctors used to bitch about shit like this, because they felt it was beneath them to follow checklists. As a result people have had limbs removed and life threatening, unnecessary surgery. Guess what? The insurance companies demand checklists and controls to prevent fuckups. There is an easy solution to dealing with people who refuse to read things. (Hint: It doesn't involve recording hours…

Yeah, I'm not disagreeing with checklists and SOPs. I'll edit my post to make sure that's clear as it's absolutely not what I'm trying to say at all. Quite the contrary: I'm trying to say that to the fullest extent possible, those SOPs should be integrated into the process so that skipping them isn't an option. Well-run open-source projects have given us good examples of how this works.

The goal is to marry the SOP and the actual completion of the task, to block off the shortcuts and require a minimum expenditure from external discipline/motivation reserves to get people to follow those SOPs.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#72

Post by problems »

geofft wrote: post #70
fsiefken wrote: post #57

Earlier quoted context omitted.

As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually. GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime…

Synchronizing the database isn't quite what you want. It's true that in the case of an errant rm -rf it would almost certainly have helped, but it's approximately as easy to run a "DELETE FROM importantdata" and leave off the "WHERE" clause, which would get replicated. And certainly if you're using DRBD (replicate the volume, not the database), an rm -rf will get replicated. I'm just genuinely unsure what a better ou…

You can easily add snapshotting using on-disk technologies like LVM or ZFS to that and achieve reliability against such an issue though, as well as being able to do a full text backup (ie: to SQL) from your replicated server at higher performance than you would on production.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#73

Post by Perihelion »

bryanh wrote: post #58

160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate. We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural -…

Honestly, I haven't seen a company that does the hybrid remote/on-site thing well. I'm sure they exist, but every time I've worked at one the people who were remote were out of the loop on almost everything.

Hybrid remote/on-site requires some great discipline. If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was said.

+1 for working remotely on a team that makes an effort to make it work.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#74

Post by ekidd »

cookiecaper wrote: post #8
manojlds wrote: post #2

After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.

I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my C…

Hmm, in Boston they only pay 2/3rds of what Google does for somebody with my job title (not counting stock or bonuses). And as soon as you get outside of the "hot" markets, those multipliers are pretty bad. If I compare their Rochester, NY salaries with what I've actually earned working in rural college towns, well, I've gotten better paychecks in academia.

They seem like nice folks, but as an experienced remote worker (with a history of both remote employment and remote contracting), I wouldn't bother to apply based on what I currently know.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#75

Post by connorshea »

notcolin wrote: post #66
Perihelion wrote: post #32

Earlier quoted context omitted.

Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things. But you're right, the timing of this piece was probably not ideal.

> This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Could you elaborate? I recently heard about GitLab probably less than a week ago and was under the impression they're a great company to work for.

She's referring to issues in the backup process specifically, not in the company culture (which IMO is pretty good).

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#76

Post by ashark »

pc86 wrote: post #47
cookiecaper wrote: post #8

Earlier quoted context omitted.

I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication). GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my C…

Just ran my numbers and got about 55% of my current salary assuming a lead position (what I have now, which I would probably not get with them) and "a lot of experience" which I'm on the fence about and they would probably disagree with. Getting rid of the COL deduction by selecting NYC as my location actually makes it reasonable so something tells me not pinning the US salary floor to 1.0 is saving them money and si…

Set it to "lead" and "a lot of experience" (I'd grade myself intermediate-to-senior and above-average experience IRL) and the top of that range is less than I make now, and I'm not big on self-promotion/networking and could probably make quite a bit more if I were. Plus their benefits are worse than what I get at my tiny company. Their comp calculation sucks for my area.

They're paying talentless Wordpress dev prices for their mid-tier devs, by local standards. This is in a mid-sized midwestern US city. LOLWUT? No one decent in my city would accept a job there.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#78

Post by Perihelion »

notcolin wrote: post #66
Perihelion wrote: post #32

Earlier quoted context omitted.

Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things. But you're right, the timing of this piece was probably not ideal.

> This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Could you elaborate? I recently heard about GitLab probably less than a week ago and was under the impression they're a great company to work for.

When I say process, I mean some of our operational processes like making sure backups work. There was an incident with data loss last week: https://news.ycombinator.com/item?id=13537052

Like any company, we have our own business operations issues too but it's still one of the best jobs I've ever had. We're pretty open about our flaws, to the point where the CEO even has a page that lists his out: https://about.gitlab.com/handbook/people-operations/ceo-pref...

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#79

Post by sotojuan »

Perihelion wrote: post #73
bryanh wrote: post #58

160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate. We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural -…

Honestly, I haven't seen a company that does the hybrid remote/on-site thing well. I'm sure they exist, but every time I've worked at one the people who were remote were out of the loop on almost everything. Hybrid remote/on-site requires some great discipline. If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was…

It works at our company because the only remote guy is the only one in his "domain" (iOS). He probably doesn't care about the frontend watercooler chat and anything important about the backend will be mentioned in standup.

Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

#80

Post by snarf21 »

Perihelion wrote: post #73
bryanh wrote: post #58

160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate. We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural -…

Honestly, I haven't seen a company that does the hybrid remote/on-site thing well. I'm sure they exist, but every time I've worked at one the people who were remote were out of the loop on almost everything. Hybrid remote/on-site requires some great discipline. If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was…

Exactly, it requires an active investment in over communication. Hopefully someday the tools will evolve to help reduce the need for this to be so manual.

Bouncing an idea off of someone remote gets difficult. I feel bad for interrupting them and they feel annoyed at being interrupted. Culture also plays a big role in that as well.

Post reply on HN