Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Please don't waste maintainers' time on your KPI grabbing patches”

lkml.org

Post reply on HN
Print view View original

161–170 of 277 posts

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#161

Post by sdesol »

jart wrote: post #124
acover wrote: post #112

Earlier quoted context omitted.

Edit: Nvm, I misunderstood the initial problem and my solution doesn't address it. Why not just keep commits as a metric, then estimate the quality of those commits by sampling. If the organization appears to be gaming the stats then flag it as such.

I tend to look on commit count as a negative when reviewing open source projects. TeX is probably the highest quality open source project and it's has had maybe 15 commits since I was born. Bash has had about 15 commits in the last five years. Then there's everyone else who's playing to win the Github game with 500 stars and 5,000 commits. That level of activity might impress normal people but it doesn't impress me.

> I tend to look on commit count as a negative when reviewing open source projects.

I do agree that tracking commit counts isn't very useful but tracking commit frequency, in my opinion, is quite useful. I'm obviously biased but I use commit frequency to tell me how fast a project is moving and what is the investment.

Take the following for example:

https://public-001.gitsense.com/insights/github/repos?p=impa...

by breaking down how frequently contributors commit on a daily basis, you can get a very good sense of investment and speed. In the case of the vscode project, Microsoft is investing a lot of resources into vscode and it is evolving at an extremely fast pace.

Software metrics in general isn't the issue, it's the lack of context around them that is the issue.

Disclaimer: I'm the creator of the tool that I'm linking to

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#162

Post by croes »

lmilcin wrote: post #38

I have worked in team with some kernel developers at Samsung, years ago, so let me put some perspective for those who do not understand the dynamics. In some companies, the amount of patents or Linux kernel patches you get accepted is direct measure of your success. As you know, whatever you measure becomes a target -- these guys feel very pressed to get ANY kernel commits accepted, no matter how small, irrelevant or…

"When a measure becomes a target, it ceases to be a good measure."

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#163

Post by Tainnor »

lmilcin wrote: post #140
Clewza313 wrote: post #137

Earlier quoted context omitted.

Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Humans are extremely good at gaming KPIs, and will do so as long as they're rewarded for it.

Overuse law. If you overuse a law too much it stops being meaningful. Organizations have hard time improving without measuring their performance and communicating incentives. There isn't magical solution you can scribe on a paper and tell everybody -- this is exactly what you need to do to achieve success. The best what you can do is compromise, it is unavoidable. So we know setting targets can make wrong incentives.…

The point they're trying to make is, I believe, that metrics shouldn't be targets. In a sense I think that this is obvious to anyone with a rudimentary understanding of applied statistics.

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#164

Post by spfzero »

qsmlfkqjsd wrote: post #119
lmilcin wrote: post #73

Earlier quoted context omitted.

The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Not to mention amount of work required which is exactly the issue. But mostly, maintainers want to just focus on their work and not be bothered by corporations and their developers trying to game the system just to prop up their position.

> The issue is that creating objective measure of contribution value is just unfeasible if at all possible. Could you give an example of a non trivial patch which value would be unfeasible to assess objectively? I am not familiar enough with the maintenance process to understand how hard it is to measure objectively the quality of a patch.

It might be feasible to assess a single contribution, but not feasible to assess all of them. To make this KPI work, maintainers would have to publish an objective quality rating for each accepted commit. That's an enormous amount of work, and impossible to get right anyway since the value of a change might not be immediately obvious, or might be negated by a future change, etc.

The advice to companies using this KPI would be: let your own dev managers evaluate the quality of the commits of their engineers, and rate them on quality as well as quantity, then submit. Engineers will quickly stop creating busywork when their managers get wise to it.

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#165

Post by belter »

Aperocky wrote: post #151
belter wrote: post #132

Earlier quoted context omitted.

As I am not a Chinese speaker..do you confirm there is in the thread a comment that the maintainer is also Chinese ? I am concerned about possible pressure from Huawei and its "shareholders": https://www.bbc.com/news/business-53172057 to a Kernel maintainer. Looking at the heavy down votes my post got its not a concern here...

Your post are downvoted because it is borderline racist. The maintainer happened to be Chinese and that gave you 'chills'. There are 1.4 billion Chinese and most of them live in China, there are also open source projects hosted in China, and a lot of open source contribution coming from people living in China, if only because there are many people there. The media links you posted are also made of conjecture ('Trump…

The comment is not racist because if you did read it the concern was NOT about the maintainer being Chinese and I made that VERY CLEAR.

The concern was that in the Chinese group discussion commentators were mentioning he was Chinese. Now why would that be relevant indeed ?

It can be relevant because it open the possibility to him being pressured by a company that is partly owned by the Chinese Army. That is where the chill was coming from... Hopefully he is based in the US.

Now tell me, where is the racism ?

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#166

Post by Tainnor »

TimWolla wrote: post #158
nneonneo wrote: post #93

The fellow who is being named in this particular PR has been _quite_ busy lately, submitting mostly typo fixes and whitespace fixes: https://lore.kernel.org/lkml/?q=f%3Athunder.leizhen Sometimes their patches aren't even valid - they tried to fix "borken" to "broken" and the maintainer was not happy: https://lore.kernel.org/lkml/YK3wOkX6I78j73zD@gmail.com/ (this does come down to not being familiar with this particul…

> they tried to fix "borken" to "broken" and the maintainer was not happy […] this does come down to not being familiar with this particular bit of slang - but they push back and argue a bit which doesn't help I did not know "borken" either, but I am aware of "borked" and "broken". Based on that email thread someone else already attempted to fix this in the past. Maybe it's an indication that the so-called "joke" is…

Agreed. While the kernel developers can obviously do whatever they want in terms of in-house jokes, the claim that "this is just normal usage, Google it" is a bit ridiculous. They could have just said, "we know it's wrong, we just thought it's funny".

("Borken" is a city in Germany, that's my only association with it )

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#167

Post by belter »

belter wrote: post #165
Aperocky wrote: post #151

Earlier quoted context omitted.

Your post are downvoted because it is borderline racist. The maintainer happened to be Chinese and that gave you 'chills'. There are 1.4 billion Chinese and most of them live in China, there are also open source projects hosted in China, and a lot of open source contribution coming from people living in China, if only because there are many people there. The media links you posted are also made of conjecture ('Trump…

The comment is not racist because if you did read it the concern was NOT about the maintainer being Chinese and I made that VERY CLEAR. The concern was that in the Chinese group discussion commentators were mentioning he was Chinese. Now why would that be relevant indeed ? It can be relevant because it open the possibility to him being pressured by a company that is partly owned by the Chinese Army. That is where the…

Who Owns Huawei ?

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3372669

Who Owns Huawei? The Company Tried to Explain. It Got Complicated.

https://www.nytimes.com/2019/04/25/technology/who-owns-huawe...

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#168

Post by TimWolla »

Tainnor wrote: post #166
TimWolla wrote: post #158

Earlier quoted context omitted.

> they tried to fix "borken" to "broken" and the maintainer was not happy […] this does come down to not being familiar with this particular bit of slang - but they push back and argue a bit which doesn't help I did not know "borken" either, but I am aware of "borked" and "broken". Based on that email thread someone else already attempted to fix this in the past. Maybe it's an indication that the so-called "joke" is…

Agreed. While the kernel developers can obviously do whatever they want in terms of in-house jokes, the claim that "this is just normal usage, Google it" is a bit ridiculous. They could have just said, "we know it's wrong, we just thought it's funny". ("Borken" is a city in Germany, that's my only association with it )

> ("Borken" is a city in Germany, that's my only association with it )

Being German myself it's also the plural of "Borke" (i.e. the "bark" of a tree): https://de.wikipedia.org/wiki/Borke

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#169

Post by Tainnor »

Similar thing happened with the Hacktoberfest incident last year: https://blog.domenic.me/hacktoberfest/

In short, DigitalOcean unwisely incentivised a bunch of people to make PRs on open source repositories unaffiliated with DO so they could get a T-Shirt. It went so far that some guy actually made a very popular YT video about how you can make a PR for a small typo fix (I guess I still fail to understand why you'd need a YT video to explain that). Maintainers were none too pleased.

Re: “Please don't waste maintainers' time on your KPI grabbing patches”

#170

Post by sosborn »

jart wrote: post #139
alxlaz wrote: post #129

Earlier quoted context omitted.

KPI = Key Performance Indicator. It's basically a list of things that are considered during your evaluation. The more items off that check list you cross (if they're one-time items, like "ensure an uptime of X"), or the more of each you do (if they're things like "upstream patches" or "patents granted"), the better your score is. It's a shit system that does exactly the opposite of what it's supposed to do IMHO, but…

I didn't know open source programmers were subjected to those kinds of performance reviews. It sounds to me like the pandas have come and visited the penguins and there's a whole lot of shock and horror as the penguins get eaten because you'd think the pandas only want bamboo.

A great many open source programmers are being paid salary and sitting in a corporate cubicle.
Post reply on HN