Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On the Impact of Programming Languages on Code Quality

arxiv.org

Post reply on HN
Print view View original

11–20 of 138 posts

Re: On the Impact of Programming Languages on Code Quality

#11

Post by faitswulff »

jph wrote: post #6

(Edited-- I was reading the original paper, which showed good results for TypeScript; the new paper linked above refutes that result. My mistake. Thanks to the posters who corrected me!) The winners in this paper are Clojure, Haskell, Scala. I look forward to seeing Rust added. The authors go into substantial detail about what they consider errors, and in my personal experience Rust solves many of them. See e.g. "Som…

That's not at all what I got from the abstract:

> ...only four languages are found to have a statistically significant association with defects, and even for those the effect size is exceedingly small.

Emphasis mine.

Re: On the Impact of Programming Languages on Code Quality

#12

Post by idoubtit »

My own take of this:

- This study tries to enhance the original work by better metrics (e.g. confusion with C/C++), using better statistics (e.g. uncertainty).

- Automatically evaluating the quality of a code is hard and time-consuming. They had to use peer review of labellisation.

- Most of the old claims are not confirmed.

- Only 2 languages in this study had both a significant impact (impact coeff) and enough fiability (p-value): Clojure and Haskell. But the impact is still small, and there may be other influencing factors.

- They propose several ways to further enhance their work (e.g. taking regression tests into account).

Re: On the Impact of Programming Languages on Code Quality

#13

Post by [deleted] »

salmonellaeater wrote: post #10

Of note is that these are all associations. It's possible the associations are all due to selection effects (or even that the "better" language are actually worse but their effects are reversed by selecting for programmers who are better). In the early days of Python, some employers used it as a way to filter for candidates who were ahead of the curve. I believe Paul Graham mentioned Lisp being good for attracting be…

[deleted]

Re: On the Impact of Programming Languages on Code Quality

#14

Post by idoubtit »

panpanna wrote: post #4

Maybe off topic, but isn't the language in the abstract a bit aggressive? I mean, this type of study is somewhat open to interpretation. It's not like everyone should arrive to the exact same numbers..

The conclusion makes it clear that this work has no aggressive intent:

Unfortunately, our work has identified numerous problems in the FSE study that invalidated its key result. Our intent is not to blame, performing statistical analysis of programming languages based on large-scale code repositories is hard.

[...]

Acknowledgments. We thank Baishakhi Ray and Vladimir Filkov for sharing the data and code of their FSE paper. Had they not preserved the original files and part of their code, reproduction would have been more challenging

Re: On the Impact of Programming Languages on Code Quality

#15

Post by dahart »

One of the simplest but more important conclusions that affects me all the time... it’s easy to search for things, but hard to know whether you’ve got all the answers you want and hard to know if all the answers you got are answers you want.

6.3 Grep considered harmful

Simple analysis techniques may be too blunt to provide useful answers. This problem was compounded by the fact that the search for keywords did not look for words and instead captured substrings wholly unrelated to software defects. When the accuracy of classification is as low as 36%, it becomes difficult to argue that results with small effect sizes are meaningful as they may be indistinguishable from noise. If such classification techniques are to be employed, then a careful post hoc validation by hand should be conducted by domain experts.

Re: On the Impact of Programming Languages on Code Quality

#16

Post by hannofcart »

salmonellaeater wrote: post #10

Of note is that these are all associations. It's possible the associations are all due to selection effects (or even that the "better" language are actually worse but their effects are reversed by selecting for programmers who are better). In the early days of Python, some employers used it as a way to filter for candidates who were ahead of the curve. I believe Paul Graham mentioned Lisp being good for attracting be…

Indeed, and in addition to selection bias towards better programmers, there is another selection bias implicit in this.

A shop that makes the decision to write their stack in Haskell do it with an intention to focus on correctness. I reckon that such companies would consequently invest in other practices like more tests, higher code coverage and more intensive code reviews resulting in code with fewer errors.

This is in stark contrast to a startup that may have hacked together a POC in Python, and once it worked, with a little bit more spit and polish put it into production.

Re: On the Impact of Programming Languages on Code Quality

#17

Post by notacoward »

salmonellaeater wrote: post #10

Of note is that these are all associations. It's possible the associations are all due to selection effects (or even that the "better" language are actually worse but their effects are reversed by selecting for programmers who are better). In the early days of Python, some employers used it as a way to filter for candidates who were ahead of the curve. I believe Paul Graham mentioned Lisp being good for attracting be…

> It's possible the associations are all due to selection effects

FWIW, the original authors (Ray et al) said as much.

Re: On the Impact of Programming Languages on Code Quality

#18

Post by goto11 »

salmonellaeater wrote: post #10

Of note is that these are all associations. It's possible the associations are all due to selection effects (or even that the "better" language are actually worse but their effects are reversed by selecting for programmers who are better). In the early days of Python, some employers used it as a way to filter for candidates who were ahead of the curve. I believe Paul Graham mentioned Lisp being good for attracting be…

The paper goes further though. It states: Not only is it not possible to establish a causal link between programming language and code quality based on data at hand, but even their correlation proves questionable.

I find that a surprising result! This might also call into question the theory that some languages attract better developers.

Re: On the Impact of Programming Languages on Code Quality

#20

Post by scott_s »

jph wrote: post #6

(Edited-- I was reading the original paper, which showed good results for TypeScript; the new paper linked above refutes that result. My mistake. Thanks to the posters who corrected me!) The winners in this paper are Clojure, Haskell, Scala. I look forward to seeing Rust added. The authors go into substantial detail about what they consider errors, and in my personal experience Rust solves many of them. See e.g. "Som…

I believe you are confusing the conclusions from the original paper with the claims from this paper. I think this is the case because this paper removes TypeScript from their dataset entirely:

"4.1.2 Removal of TypeScript. In the original dataset, the first commit for TypeScript was recorded on 2003-03-21, several years before the language was created. Upon inspection, we found that the file extension .ts is used for XML files containing human language translations. Out of 41 projects labeled as TypeScript, only 16 contained TypeScript. This reduced the number of commits from 10,063 to an even smaller 3,782. Unfortunately, the three largest remaining projects (typescript-node-definitions, DefinitelyTyped, and the deprecated tsd) contained only declarations and no code. They accounted for 34.6% of the remaining TypeScript commits. Given the small size of the remaining corpus, we removed it from consideration as it is not clear that we have sufficient data to draw useful conclusions."

Post reply on HN