The ‘flawed five’ engineering productivity metrics

Personally, if I get a small PR, I review it in about 30 seconds. If I get a giant PR, I review it in a matter of hours. You are usually aware that such things are coming; that's why people provide daily-ish status updates and that sort of thing.

I have also found that high review latency begets large code reviews. It's kind of a vicious feedback loop. If you always review PRs immediately, people are less afraid of splitting up their work, and PRs will become smaller. (It's hard to "stack" reviews, so people avoid it. That means that 3 PRs become 1 PR, so they can be mentally done with the task and start something brand new while waiting a week for your code review. The alternative is hoping that the bottom of your stack is approved without any requests to change anything, but what happens is that large changes are requested and the two floating PRs are now impossible to merge back in, or are irrelevant. If the bottom of the stack could be reviewed in an hour, then all that work wouldn't have been wasted, and it reinforces the good habit of doing 1 small thing at a time.)

I look at all of this as a systemic issue, and not a personal issue. The team needs to set an aggressive SLA, guided by what the engineer is expected to do while waiting for a review. Then engineers need to treat the objective seriously; code review is as important as any code you're writing. That's why you're part of a team, and aren't a 1 person army (which is a totally viable approach; I write a ton of code in my free time by myself, but I don't necessarily think that that's how I should behave at work.)

Personally, I have Github's Slack notifications on, and almost always start the review as soon as I get the notification. I know that the person waiting for the review has no work to do other than to build on top of this foundation; the less they do without feedback, the better. (Plus, they want to get their code Out There, that's why they wrote it.) I don't think your organization has to be that aggressive, but I'd aim closer to "hard interrupt" than "within 1-2 business days". (No, I don't review PRs that come in at 7PM on Friday night. Those wait until Monday morning.)

The other metrics listed a few comments above are really a function of organizational policies. Make your releases easy, so you do them often. Make sure that your PMs know what users want, so you don't implement code and throw it away. Make sure that CI is fast, so there isn't a delay in pushing to production. And make sure your process is realistic. My team used to spend a lot of time manually testing stuff in staging before releasing to production. It never found any issues that the test suite didn't find, and we prefer the automated tests anyway, so eventually we just made every commit to master a deployment to production. Click the "merge" button and it's out to the users. No unusual number of surprise production outages, just a vastly reduced time from idea to being in the hands of the customer. (Code reviews ended up being the limiting factor. People really don't like doing them!)

I'm not making a legal argument here. I would never claim that I could verifiably prove these statements. However, I believe them to be true, and have personally heard from people who would know.

What I would tell you that is verifiably true, is that myself and other engineers all worked under the impression that these metrics were being used. Whether or not they actually were I cannot say for sure, but I can definitely tell you that it affected the culture.

The OLR process at Amazon is notoriously opaque. The HR and management at Amazon are notoriously smart. Amazon has a bad working culture (obviously, this is in my opinion). I would say that it is my duty to share elements of it that were bad for my wellbeing, to help others make informed decisions about their own futures.

Actually, they seem to be presupposing that you're using these metrics as a target to judge employee performance by (in the style of key productivity indicators), rather than as a diagnosic tool to debug your/management's own mistakes/potential improvements in how the team is organized, and then pointing out in what way the former approach - which always fails regardless of which metrics you use due to Goodhart's law - fails with this particular choice of metrics.

Edit: granted, sometimes the result of said debugging is that it really is a hardware problem, and with how unstandardized people are, that's actually common enough that you'll likely run into it. But you don't fix (for example) a disk drive that reports writes as complete before they're peristed by stack-ranking drives and trying to pick the ones that lose data less; you fix it either by replacing the faulty drive immediately (assuming you can get working drives), or (more relevantly to dealing with people) by working around it in software.

0: http://en.wikipedia.org/wiki/Goodhart's_law

> the business people who have zero clue how hard our work really is

Note that this goes both ways; the'll both wildly overestimate and wildly underestimate how hard various tasks are, often in the same conversation.

See eg https://www.explainxkcd.com/wiki/index.php/1425.

She already got promoted to big sister and we are not having another one any time soon.