Earlier quoted context omitted.
In case where I work for a given company and it's internal tool, always.
Imagine this scenario. You work on product A. Your VP of very important things declares that every product should use library L. Library L has been designed to incestously interact with product D and makes a bunch of assumptions about what the "working environment" should be. Teams B and C have already made the transition, so L made some kludgy APIs to make B and C interaction possible, though it is maintained with l…
GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
171–180 of 320 posts
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#172160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate. We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural -…
We have found that many wikis don't have good "related pages" functionality, i.e. they are hard to browse.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#173Earlier quoted context omitted.
Synchronizing the database isn't quite what you want. It's true that in the case of an errant rm -rf it would almost certainly have helped, but it's approximately as easy to run a "DELETE FROM importantdata" and leave off the "WHERE" clause, which would get replicated. And certainly if you're using DRBD (replicate the volume, not the database), an rm -rf will get replicated. I'm just genuinely unsure what a better ou…
You can easily add snapshotting using on-disk technologies like LVM or ZFS to that and achieve reliability against such an issue though, as well as being able to do a full text backup (ie: to SQL) from your replicated server at higher performance than you would on production.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#174160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate. We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural -…
Honestly, I haven't seen a company that does the hybrid remote/on-site thing well. I'm sure they exist, but every time I've worked at one the people who were remote were out of the loop on almost everything. Hybrid remote/on-site requires some great discipline. If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was…
Management is onboard with it and has taken the openly stated position of "we don't care as long as you're reachable during the hours it's reasonable for someone to be online" (and the obvious: "deliver your deliverables when they're due"). In my case as the ops lead, I have additional hours but we are so well baked into slack that if someone needs me, I get pinged via mobile, reply from where I am or delegate it. Heck just last week I did a couple of deployments from 4 timezones away, tested it, and left a PR note in the channel with a link to the JIRA ticket. It got worked, updated, fixed and closed while I slept.
I wouldn't say our team made a concerted effort, we just set the expectation that if you're going to be away, let us know, and during production hours the absolute bare minimum is that you'll be reachable if absolutely needed. Heck, even if you need to leave early to take care of home stuff or what have you: go to team chat "Hey I'm taking off early to take care of thing" and go. You'll get replies like "Take it easy" and "Cya tomorrow".
Every team meeting has a join.me session for remote people to join, they speak up and contribute if asked or needed, even our stand-up comes with an auto generated Google Hangouts link that gets published to slack every morning at 10am. If you miss the stand-up or can't connect, people have been really great about just typing up what they're working on when they're available again.
Otherwise, no one really cares. I've absolutely loved it. This is one of the better teams I've worked on just by how well everyone communicates without bombarding each other. People stay out of each other's way, but will ask when they need help and there's never any fear that you wont get someone willing to at least look at a problem with you; even if they don't know the answer-I've had devs pipe up and offer ideas and a second pair of eyes. Problem gets fixed, I shoot over a thumbs up emoji and a "thank you" and we go back to our own little worlds. Read-only Friday comes along, and we go drink.
Contrasted with my last job where I negotiated the right to work remotely because the distance constituted a total four hour commute. I was consistently turning in deliverables on-time or early, conducting client calls via an expensed voip phone and getting several orders more work done from home because I wasn't waking up hours earlier than normal to make the 2 hour drive in.
That privilege was arbitrarily lost because of a few critical management slip ups (owner took a vacation with more than a couple projects open in the lurch, and left no instruction behind for me in his wake, so things failed in a big way when). Shortly after becoming an 'office worker' again and putting in 12 hour days between commuting and sitting at my desk, I was out the door.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#175Earlier quoted context omitted.
Would love to get more input from you to make sure the calculator reaches its goal of providing fair market compensation in your location. Can you please send me an email to let me know your city and whatever further data you are willing to share? ernst@gitlab.com
My local city, but the data isn't so much mine as data that the US government shares. Sacramento, CA. Mean Salary (per latest BLS data) [0] for Software Developer, Applications: $107,540; for Web Developer: $78,050 Sacramento, CA Gitlab locality pay index: 0.39+0.25=0.51 Vs. New York City, same BLS figures [1]: $108,770/$81,430 NYC Gitlab locality pay index: 1.00+0.25=1.25 Sac/NYC Salary Ratio (BLS): 0.989/0.958 Sac/…
Didn't notice it until after the edit window was closed.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#176Earlier quoted context omitted.
Imagine this scenario. You work on product A. Your VP of very important things declares that every product should use library L. Library L has been designed to incestously interact with product D and makes a bunch of assumptions about what the "working environment" should be. Teams B and C have already made the transition, so L made some kludgy APIs to make B and C interaction possible, though it is maintained with l…
Note that I said it's my preferred way but by any means that doesn't mean that if there is some documentation I won't look at it.
Having struggled with very large tools that were anything-but, I beg to differ... but I guess it is a matter of personal experiences.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#177160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate. We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural -…
Honestly, I haven't seen a company that does the hybrid remote/on-site thing well. I'm sure they exist, but every time I've worked at one the people who were remote were out of the loop on almost everything. Hybrid remote/on-site requires some great discipline. If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was…
Long story short: I actually started on-site but negotiated part-time remote. Then after about a year of that my spouse accepted a dream job offer on the other side of the state and so it was no question that we were moving. They wanted to keep me on and so I transitioned to full-time remote.
It has it's ups and downs. From my experience, it depends on the team and ultimately the lead and/or the manager. If they get it, it's good. If they don't, don't bother.
My last team, it was beautiful. Everything worked great. It became obvious to me after transitioning away from that team just how much extra work my manager was doing to keep his remote employees in the loop. With the team I'm on now, it's not working at all. Why? Because, as you so eloquently put it:
> the people who were remote were out of the loop on almost everything... If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was said.
This is exactly the problem I am facing down. About a month ago, I even tried talking to my manager about it. Even going so far as using this exact phrasing. His response was that I should come to office for 1 or 2 day trips! Completely missed the point. I'm not surprised, I guess I just expected some effort.
I'm trying to remain positive but it's tiring. I can do good work and make progress but I'm with a lead who is insulating me from meetings and it feels deliberate. The few times I am invited in to a meeting, it's easy to see that he accepts and assumes credit for _the teams_ efforts. He was hired recently as a Senior, our manager wanted him to transition to a Lead, and he's actually landed somewhere around trying to be an architect. Ranting aside, when I bring my concerns up to him it's obvious that there's no attempt to better understand the issues/challenges of the hybrid approach. It doesn't help that he's never worked with remote employees before, either.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#178Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#179Earlier quoted context omitted.
The "trigger" of the failure (accidentally deleting the wrong database) is bad luck that happens, but the lack of preparedness for it isn't. If I understood the document right even if the regular backups would have worked, they only happened every 24 hours (same as the staging replicas they recovered from, except one was manually created out of schedule that day). Maybe being fully remote helps to let stuff like this…
What's the best-practice approach for taking backups that are significantly more frequent than every 24 hours, but also robust to things like an rm -rf or a DELETE FROM table;? Something like continuous data protection seems like it would be far too much data for an active database server, no? (Or are we just saying that they should have been taking backups every 15 minutes or hour or so?)
In the event of accidental deletion, the most recent snapshot will be restored and the transactions up to a certain timestamp will be replayed on top of it. If done correctly, this can prevent all but a few minutes of data loss.
AWS Aurora (and possibly other database-as-a-service providers) has this functionality built-in.
Re: GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
#180Earlier quoted context omitted.
> didn't do anything particularly more wrong than anyone else? This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101. Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple…
As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually. GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime…
That's what lead to the problem. We were trying to fix it, but ran the wrong procedure on the primary instead of the secondary
Thus wiping the primary, instead of what might have been left on the secondary.
The secondary was already removed at that point.
Basically the procedure was the following:
1. Secondary falls behind too much, stops replicating. At this point you need to manually re-sync the whole thing
2. A re-sync requires an empty PostgreSQL data directory, so this data was removed
3. Re-sync doesn't work, leading to the other problems
4. At some point team-member-1 thought previous re-sync attempts left data behind, so team-member-1 wiped the data directory again to be sure; except team-member-1 ran this on db1.cluster.gitlab.com (the primary) instead of db2.cluster.gitlab.com (the secondary)