On the Impact of Programming Languages on Code Quality • Hacker News Board

Re: On the Impact of Programming Languages on Code Quality

#111

Post by [deleted] » Thu, Jun 06, 2019, 12:48 AM UTC

jrumbut wrote: post #107

pron wrote: post #98

Earlier quoted context omitted.

But if we consider as the main result that the total effect of language choice is less than 1%, even "looking at noise" means looking at an effect of 0, and either way, the effect is tiny to nonexistent. So if someone had the hypothesis that the effect of language choice is large , because lack of evidence of a big effect is evidence of lack of one, this study is some evidence against such a hypothesis. That this fin…

If I were sitting on a thing that could decrease bugs 1% I'd be really excited. The only reason this looks like a paltry reduction is that it's hugely expensive and difficult to change languages midstream. This is evidence against a shop retraining C++ devs in Haskell (which I'm sure someone somewhere wants to do). But if it was something like switching editor fonts? I'd do it for 1%.

[deleted]

Re: On the Impact of Programming Languages on Code Quality

#112

Post by yogthos » Thu, Jun 06, 2019, 1:09 AM UTC

pron wrote: post #96

The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The original paper explains: "With effects coding, each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects." So the entire discussion is on the re…

Two things this studies haven't measured are time to deliver features, and the number of developers necessary. For example, if a project in Haskell can be delivered faster by a smaller team than an equivalent project in C++, that would be an important factor.

Re: On the Impact of Programming Languages on Code Quality

#113

Post by shereadsthenews » Thu, Jun 06, 2019, 1:16 AM UTC

mcguire wrote: post #78

shereadsthenews wrote: post #40

Earlier quoted context omitted.

Considering the astonishing density of defects in Linux we can treat its author’s advice on software development as anti-advice.

What is the density of defects in Linux? Any published comparisons?

I don’t know about relative quality but Syzkaller finds a new kernel bug every minute or two and Coverity says there are thousands of unfixed defects that their static analysis can detect.

Re: On the Impact of Programming Languages on Code Quality

#114

Post by larzang » Thu, Jun 06, 2019, 1:20 AM UTC

dragonwriter wrote: post #91

stcredzero wrote: post #52

Earlier quoted context omitted.

Ruby's community has an obsessive culture towards unit testing, that could impact bug levels. Like Smalltalk, they're forced to do this to catch the errors that would've been caught by type annotation. So then, could the argument be made that less stringent type annotation hurts code quality by removing effort from unit testing?

> Like Smalltalk, they're forced to do this to catch the errors that would've been caught by type annotation. Very few of the unit tests I've written for Ruby/Python catch errors that would be caught by typical type system use in C/Java/etc. (Haskell, maybe, but...) The main thing I miss in dynamic languages vs. static is the more powerful automatic testing possible when test specs can leverage type information (e.g.…

Yeah it's easy to see writing in dynamic languages with optional typing and static analysis tools (e.g. JS with TS/Flow, PHP with phpstan or psalm), the type analysis finds all kinds of different and subtle problems that unit tests probably wouldn't catch. Large integration tests often would, but are even harder to write exhaustive cases for.

Types just make everything easier to do right the first time.

Re: On the Impact of Programming Languages on Code Quality

#115

Post by bunderbunder » Thu, Jun 06, 2019, 1:27 AM UTC

yogthos wrote: post #112

pron wrote: post #96

The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The original paper explains: "With effects coding, each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects." So the entire discussion is on the re…

Two things this studies haven't measured are time to deliver features, and the number of developers necessary. For example, if a project in Haskell can be delivered faster by a smaller team than an equivalent project in C++, that would be an important factor.

There's a concept I discovered in the realm of political science, the "fundamentally unanswerable question" - ones where it's not actually possible to generate the kinds of counterfactuals you'd need to try and run an experiment.

I think this is probably one of those. Haskell and C++ are very different languages, that are designed for very different purposes, and are popular in very different problem domains. To start with, that would make it difficult to come up with a task that wouldn't handicap one language or the other. (For example, a video game would favor C++.) Even if you found that, you'd have a hard time coming up with a good test. Under random assignment, Haskell is presumably going to be handicapped relative to C++ by the fact that most languages are imperative and use Algol-style syntax. You could try to avoid that by using participants with zero programming experience, but then you're studying learning curve for absolute beginners rather than productivity among experts. And so on and so on.

Long story short, probably best to leave this one to the spittle-flecked Internet arguments.

Re: On the Impact of Programming Languages on Code Quality

#116

Post by [deleted] » Thu, Jun 06, 2019, 1:38 AM UTC

yogthos wrote: post #112

pron wrote: post #96

The most practically relevant finding of the original paper remains in effect: the choice of language has an effect of less than 1% on the bug rate[1]. The original paper explains: "With effects coding, each coefficient indicates the relative effect of the use of a particular language on the response as compared to the weighted mean of the dependent variable across all projects." So the entire discussion is on the re…

Two things this studies haven't measured are time to deliver features, and the number of developers necessary. For example, if a project in Haskell can be delivered faster by a smaller team than an equivalent project in C++, that would be an important factor.

[deleted]

Re: On the Impact of Programming Languages on Code Quality

#117

Post by DannyBee » Thu, Jun 06, 2019, 2:48 AM UTC

brlewis wrote: post #95

scott_s wrote: post #23

A common thread I'm seeing in the discussion here is confusing the claims from the original paper with conclusions made by this paper . This is a replication study, and it starts by stating what the original paper found. What the original paper found is not what this paper finds. For the most part, this is a replication failure, for a myriad of reasons, including several kinds of misclassification. From this paper: T…

tl;dr Reanalysis showed positive association between defects and C++, negative association for Clojure, Haskell and Ruby. The original analysis found positive association for C++, Objective-C, C, PHP, Python, JavaScript. Negative association for TypeScript, Clojure, Scala, Haskell, Ruby.

It also found the effect is very small in all cases.

Re: On the Impact of Programming Languages on Code Quality

#118

Post by pfdietz » Thu, Jun 06, 2019, 3:10 AM UTC

treebog wrote: post #104

pfdietz wrote: post #102

Earlier quoted context omitted.

That is a book that, even in the third edition, has a paragraph that disparages random testing. It's difficult for me to see how, in the era of jsfunfuzz, Csmith, AFL, and other such tools, such a source should be taken seriously.

I’ve never found a book on software I agreed 100% with. Even my favorites have some bits I think are dead wrong. It doesn’t mean the other parts aren’t valuable.

It means the book has not been updated, at least in that part. And the first edition of the book was written back when computing power was seven orders of magnitude more expensive (or more). Is advice about testing from then really going to be valid now?

Re: On the Impact of Programming Languages on Code Quality

#119

Post by yogthos » Thu, Jun 06, 2019, 1:23 PM UTC

bunderbunder wrote: post #115

yogthos wrote: post #112

Earlier quoted context omitted.

Two things this studies haven't measured are time to deliver features, and the number of developers necessary. For example, if a project in Haskell can be delivered faster by a smaller team than an equivalent project in C++, that would be an important factor.

There's a concept I discovered in the realm of political science, the "fundamentally unanswerable question" - ones where it's not actually possible to generate the kinds of counterfactuals you'd need to try and run an experiment. I think this is probably one of those. Haskell and C++ are very different languages, that are designed for very different purposes, and are popular in very different problem domains. To star…

I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping domains. The test would to treat projects as black boxes, and look for trends across many projects. If you saw a trend that Haskell projects are consistently delivered faster by smaller teams, then you could make a hypothesis that Haskell is a factor. At the end of the day, we've come up with ways to test hypothesis like quantum mechanics. I'm pretty sure measuring impact of a language on software development process is a tractable problem.

Re: On the Impact of Programming Languages on Code Quality

#120

Post by bunderbunder » Thu, Jun 06, 2019, 1:32 PM UTC

yogthos wrote: post #119

bunderbunder wrote: post #115

Earlier quoted context omitted.

There's a concept I discovered in the realm of political science, the "fundamentally unanswerable question" - ones where it's not actually possible to generate the kinds of counterfactuals you'd need to try and run an experiment. I think this is probably one of those. Haskell and C++ are very different languages, that are designed for very different purposes, and are popular in very different problem domains. To star…

I really don't see how that would be the case. Both Haskell and C++ are general purpose languages, there are plenty of projects written in many overlapping domains. The test would to treat projects as black boxes, and look for trends across many projects. If you saw a trend that Haskell projects are consistently delivered faster by smaller teams, then you could make a hypothesis that Haskell is a factor. At the end o…

Quantum mechanics is really a very different beast from the social sciences. Experimental physicists don't have to tease out any confounding effects due to the life history and social baggage that electrons bring with them to the lab.

A huge, huge, huge part of the replication crisis in social sciences is due to researchers getting overconfident about the capabilities of their experimental methods.