Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On the Impact of Programming Languages on Code Quality

arxiv.org

Post reply on HN
Print view View original

91–100 of 138 posts

Re: On the Impact of Programming Languages on Code Quality

#91

Post by dragonwriter »

stcredzero wrote: post #52
robbrit wrote: post #46

Seems like they're just doing associations, and not controlling for a lot of factors. For example, you can come up with some narratives to explain a lot of these findings: * Python, JS, PHP, Java, C++ are all very common first languages; I'd imagine the average experience level for these is dragged down a bit compared to others like Golang, Haskell, or Scala. One of the recent Stack Overflow Developer surveys found G…

Ruby's community has an obsessive culture towards unit testing, that could impact bug levels. Like Smalltalk, they're forced to do this to catch the errors that would've been caught by type annotation. So then, could the argument be made that less stringent type annotation hurts code quality by removing effort from unit testing?

> Like Smalltalk, they're forced to do this to catch the errors that would've been caught by type annotation.

Very few of the unit tests I've written for Ruby/Python catch errors that would be caught by typical type system use in C/Java/etc. (Haskell, maybe, but...) The main thing I miss in dynamic languages vs. static is the more powerful automatic testing possible when test specs can leverage type information (e.g., QuickCheck/ScalaCheck), not an absence of need for testing.

Re: On the Impact of Programming Languages on Code Quality

#92

Post by joshlemer »

dragonwriter wrote: post #91
stcredzero wrote: post #52

Earlier quoted context omitted.

Ruby's community has an obsessive culture towards unit testing, that could impact bug levels. Like Smalltalk, they're forced to do this to catch the errors that would've been caught by type annotation. So then, could the argument be made that less stringent type annotation hurts code quality by removing effort from unit testing?

> Like Smalltalk, they're forced to do this to catch the errors that would've been caught by type annotation. Very few of the unit tests I've written for Ruby/Python catch errors that would be caught by typical type system use in C/Java/etc. (Haskell, maybe, but...) The main thing I miss in dynamic languages vs. static is the more powerful automatic testing possible when test specs can leverage type information (e.g.…

I've only ever done property based testing in Scala, but I thought it was pretty popular in dynamic languages like clojure and erlang. What would prevent it from working in Ruby/Python?

Re: On the Impact of Programming Languages on Code Quality

#93

Post by dragonwriter »

joshlemer wrote: post #92
dragonwriter wrote: post #91

Earlier quoted context omitted.

> Like Smalltalk, they're forced to do this to catch the errors that would've been caught by type annotation. Very few of the unit tests I've written for Ruby/Python catch errors that would be caught by typical type system use in C/Java/etc. (Haskell, maybe, but...) The main thing I miss in dynamic languages vs. static is the more powerful automatic testing possible when test specs can leverage type information (e.g.…

I've only ever done property based testing in Scala, but I thought it was pretty popular in dynamic languages like clojure and erlang. What would prevent it from working in Ruby/Python?

Without type information already in the code (for parameters, etc.) there would be a lot more overhead in setting it up; it's possible, but higher marginal friction in the ideal case.

Plus, AFAIK, no one has written the libraries, making it even higher marginal friction in practice even more than than is true in the ideal case.

Re: On the Impact of Programming Languages on Code Quality

#94

Post by treebog »

codazoda wrote: post #67
JamesBarney wrote: post #49

Earlier quoted context omitted.

Emphasis on "Formal code review" which is practiced in far fewer shops than "code review". For those who don't happen to know the difference, formal code review basically uses a check list of common best practices and possible defects to look for versus doing each code review ad-hoc.

I thrive on checklists and hate the ad-hoc way I do code reviews on my team. This sounds great, I'm going to go look for some basic formal checklists. I need language agnostic versions and/or versions for javascript, php, and python, if anyone has suggestions.

The Art of Software Testing by Glenford Myers has an excellent language-agnostic checklist. See the chapter on “walkthroughs”

Re: On the Impact of Programming Languages on Code Quality

#95

Post by brlewis »

scott_s wrote: post #23

A common thread I'm seeing in the discussion here is confusing the claims from the original paper with conclusions made by this paper . This is a replication study, and it starts by stating what the original paper found. What the original paper found is not what this paper finds. For the most part, this is a replication failure, for a myriad of reasons, including several kinds of misclassification. From this paper: T…

tl;dr

Reanalysis showed positive association between defects and C++, negative association for Clojure, Haskell and Ruby.

The original analysis found positive association for C++, Objective-C, C, PHP, Python, JavaScript. Negative association for TypeScript, Clojure, Scala, Haskell, Ruby.

Re: On the Impact of Programming Languages on Code Quality

#96

Post by pron »

The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The original paper explains: "With effects coding, each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects." So the entire discussion is on the relative effect of particular languages relative to the total effect of language choice, which is less than 1%. The original paper reported stronger relative effects for particular languages, while this one found the effects to be smaller, and in many cases they disappeared altogether, thus summarizing the effects of the languages as "exceedingly small."

So even the comparison of C++ vs. Haskell, that has ~25% difference, is within that 1%. This is similar to a study comparing the effect of running shoes on running speed that shows that Nike positively affects your running speed 25% more than Adidas, to the extent running shoes affect your speed at all, which is less than 1%.

-----

[1] One should take care not to overestimate the impact of language on defects. While these relationships are statistically significant, the effects are quite small. In the analysis of deviance... we see that activity in a project accounts for the majority of explained deviance... The next closest predictor, which accounts for less than one percent of the total deviance, is language. (https://web.cs.ucdavis.edu/~filkov/papers/lang_github.pdf)

Re: On the Impact of Programming Languages on Code Quality

#97

Post by scott_s »

pron wrote: post #96

The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The original paper explains: "With effects coding, each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects." So the entire discussion is on the re…

I disagree that the combination of the two studies supports an effect of language choice on bug rate. Given that the original study showed a negative association for about half the languages; the replication has significant disagreement on about half of the original positive associations; and the extremely small associations across all languages, I think we're looking at noise.

Re: On the Impact of Programming Languages on Code Quality

#98

Post by pron »

scott_s wrote: post #97
pron wrote: post #96

The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The original paper explains: "With effects coding, each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects." So the entire discussion is on the re…

I disagree that the combination of the two studies supports an effect of language choice on bug rate. Given that the original study showed a negative association for about half the languages; the replication has significant disagreement on about half of the original positive associations; and the extremely small associations across all languages, I think we're looking at noise.

But if we consider as the main result that the total effect of language choice is less than 1%, even "looking at noise" means looking at an effect of 0, and either way, the effect is tiny to nonexistent. So if someone had the hypothesis that the effect of language choice is large, because lack of evidence of a big effect is evidence of lack of one, this study is some evidence against such a hypothesis. That this finding agrees with the observations in the industry lends it even more credence.

Re: On the Impact of Programming Languages on Code Quality

#99

Post by pron »

robbrit wrote: post #46

Seems like they're just doing associations, and not controlling for a lot of factors. For example, you can come up with some narratives to explain a lot of these findings: * Python, JS, PHP, Java, C++ are all very common first languages; I'd imagine the average experience level for these is dragged down a bit compared to others like Golang, Haskell, or Scala. One of the recent Stack Overflow Developer surveys found G…

It doesn't matter. They are looking at effects relative to the total effect of language choice, which they report as less than 1%. Instead of looking at these secondary effects, the main practical result is that language choice hardly matters.

Re: On the Impact of Programming Languages on Code Quality

#100

Post by maxkwallace »

sidlls wrote: post #77
msluyter wrote: post #31

Earlier quoted context omitted.

"... and there may be other influencing factors." Selection bias? Consider the type of person likely to be attracted to languages like Haskell or Clojure. (Note, didn't read the original study so I don't know the selection criteria. Were code samples in different languages from the same programmers compared, or were there different sets of programmers per language? My comment really only applies to the latter case.)

What is the type of person likely to be attracted to languages like Haskell or Clojure?

I can think of selection bias effects both in-favor of more defects and against it.

On one hand, Haskell or Clojure probably attract more academic types who have a reputation for (consistent with my experience) on-average worse-quality code.

On other hand, the barrier-to-entry for these languages is higher so they tend to be programmed by more experienced folks. I assume that people pick functional languages in part because they believe the functional paradigm leads to higher-quality code, so the sample probably cares more about code quality to begin with.

Post reply on HN