> You'll have to justify this: "Prior studies were riddled with inadequacies in controlling for confounders and outright methodological errors."
That topic has filled whole blog posts. But in a nutshell, no previous study was able to compare directly the effect of typing disciplines due to confounders that they couldn’t control for in their experimental setup. For instance, virtually all previous studies compared different programming languages, which obviously differ by more than just their typing discipline. Many studies had very small sample sizes due to using human test subjects to write sample programs. These test subjects were almost exclusively students without real-world experience — or, in some case, any prior experience in the respective programming languages. Thus, many studies — from the outset — tested beginner-friendliness of languages rather than everyday use. Fair enough, but not the same as type checking benefits. Furthermore, most of these studies, for the same reason, were restricted to testing on artificial, academic, small toy programs rather than real-world applications.
The few studies that looked at big real-world data sets (the biggest and most rigorous being linked above) didn’t even look at typing discipline as an individual factor — again because they couldn’t regress it out as a single factor.
> Your recent study is pay walled
The link I posted has the full text PDF (that’s the whole reason for me to share Paperpile links rather than the original or DOI). See also an academic discussion [1] of the manuscript.
> Its also specifically targeting JavaScript
Which is explained in the paper: no other constellation allows to compare the effect of added static type checks as directly as JavaScript/TypeScript/Flow. That’s because (a) these languages only differ in their type checker, and are gradually typed, i.e. types can be added to just a subset of the program, which enabled the study methodology. And (b) there’s an extensive database of real-world code to analyse.
> For example, it would be interesting to know how many bugs were due to weak typing.
Fair enough. I’d expect the effect to be (much) less pronounced in a language with stronger type guarantees. Then again, the 15% number is absolutely an underestimate to begin with (see the paper and the blog post [1]).
> adding the typed annotations should be a blind activity to be rigorous
That’s not at all obvious, and from experience I disagree. Thinking about types is automatic neither in static nor in dynamic languages. Once you’ve figured out the correct type, yes, it’s a blind activity … but that’s a pretty empty statement.
> having to add types can introduce other forms of bugs
Again, this claim is far from obvious, beyond the trivial “if I make this type an `int` even though it should be a `string` then that’s a bug”. Fair enough, but adding the type annotation and performing a type check merely reveals this bug. The bug itself was present in the programmer’s flawed assumptions about the invariants in the code. Your assertion is equivalent to saying “compiling or interpreting the code [rather than writing it on a piece of paper and never touching it] can introduce bugs”. — The bugs are already in the code, we just didn’t detect them.
> Would have been interesting too to analyse how long it took for the bug fix commit
They only considered bugs they could fix within a highly constrained time frame, and only looking at local code around the bug. Hence, again, why the 15% is an extreme underestimate. Furthermore,
> So you would know the real cost of them.
The study calculated the token cost of adding types (and provide a justification for this metric). In a nutshell, the cost is negligible, especially in languages with strong type inference capabilities (Flow outperforms TypeScript here).
> Clojure and Ruby were both outliers that outdid almost all static languages.
Right, and I’d expect the same to still hold. But it shows one of the crucial shortcomings of previous studies: they primarily did not examine the effect of static vs dynamic typing. Rather, they examined the effect of different programming languages, to which typing is just one contributor amongst many. It should come as no surprise to experienced programmers that some languages (regardless of static vs dynamic discipline) vastly outperform others. For me, this is all the more reason to be excited about the headline topic: If done well, statically type checked Ruby could be an amazing language, with the added benefit of static typing at (as shown) virtually no cost.
[1] https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-q...