Experiment: Unit testing isn't enough; You need static types, too • Hacker News Board

Re: Experiment: Unit testing isn't enough; You need static types, too

#221

Post by ufo » Wed, Jun 20, 2012, 9:01 PM UTC

jerf wrote: post #98

cyrus_ wrote: post #56

Earlier quoted context omitted.

Many of the unit tests would have to be written in a simply-typed language like Haskell too -- the author notes that only a handful of tests could be entirely eliminated due by the rewrite. Probably worth further study.

In my admittedly-limited personal experience doing something similar, it's really hard to characterize this in a sane way. What you end up with is a pile of unit tests which are "really testing something", yet, some non-trivial percentage of the tests are still redundant to the type system. You feel like you can't throw it away because of the percentage that is a real test of functionality, but if you'd been starting…

I think the cleanest example is the Quickcheck library. In Haskell an important focus is proving static invariants while in traditional unit tests an important focus is proving code coverage (since stupid type bugs like to hide in uncovered code).

Re: Experiment: Unit testing isn't enough; You need static types, too

#222

Post by johnkchow » Wed, Jun 20, 2012, 9:04 PM UTC

papsosouid wrote: post #57

johnkchow wrote: post #23

Earlier quoted context omitted.

From my personal experience, the saving in development time is very much real. I used to work on .NET, and you'd have to program against crazy, non-intuitive patterns in order to have your code "clean" (I hated IoC containers as well as writing all that boilerplate code for Attributes. And for Java, remember that hilarious post about Factory-Factory-Factory patterns http://discuss.joelonsoftware.com/default.asp?joel.…

>From my personal experience, the saving in development time is very much real. But your personal experience is comparing expressive languages to unexpressive languages, and then making the mistake of thinking you are comparing dynamically typed languages to statically typed languages. What does .net's ugly (in your opinion) API have to do with static typing? Factory-Factory-Factory patterns is a joke even in java, b…

Got to say this is hard to admit, but after reading everyone else's comments, I am indeed ignorant. Thanks for the reality check.

Re: Experiment: Unit testing isn't enough; You need static types, too

#223

Post by hackinthebochs » Wed, Jun 20, 2012, 9:08 PM UTC

davesims wrote: post #188

hackinthebochs wrote: post #165

Earlier quoted context omitted.

I'm gonna have to disagree with you about the conclusion you're drawing. Yes, they are using the generic phrasing of "static typing" vs "dynamic typing", but this is because the study was intended to test the concept of static vs dynamic typing, not particular instances of it. However, seeing as we only have specific instances from which to test, it used the best one currently in widespread use. I don't see this as a…

> but this is because the study was intended to test the concept of static vs dynamic typing, not particular instances of it Help me out here -- since the study confines itself to a handful of small Python programs translated to an idiosyncratic language like Haskell, how can the scope of the study possibly in any way qualify as a study on something so broad as "the concept of static vs. dynamic typing"? Are you not…

>Help me out here -- since the study confines itself to a handful of small Python programs translated to an idiosyncratic language like Haskell, how can the scope of the study possibly in any way qualify as a study on something so broad as "the concept of static vs. dynamic typing"?

You raise a good objection here. Is it possible to draw conclusions about the class of type systems labelled "static typing" vs dynamic typing by using a small sample of programs? I think this is where the impedence mismatch is occurring. The author seems to take static typing to mean "what can be currently accomplished through static typing", and thus he was justified in using the strongest static type system in use to do the study. Taking it this way, then the study seems meaningful.

Taking the other meaning, the class of type systems labelled static typing, then you end up with a very large set of languages each with (perhaps) varying amounts of power. Doing a study with just one static language does seem inadequate. Although, depending on the class of errors caught, it may still be valid. As far as I've seen, Haskell doesn't catch new classes of errors that are impossible in other systems, it just makes it a lot easier to do so. So essentially Haskell has the same power as other common type systems. If this holds, then the study would still be valid. (Admittedly I know very little about Haskell so I could be completely wrong).

TLDR: I see what you're saying, and I do agree that there needs to be more said before his conclusion can be supported by the study.

Re: Experiment: Unit testing isn't enough; You need static types, too

#224

Post by tikhonj » Wed, Jun 20, 2012, 9:09 PM UTC

davesims wrote: post #143

papsosouid wrote: post #114

Earlier quoted context omitted.

>which makes an unwarranted equivalence of all languages that have 'static type checking': No it doesn't. Read what you quoted, it says nothing even remotely resembling "this benefit applies to all languages with static typing". It is testing static typing, not a specific language. It uses the best static typing system to do so. You are entirely inventing the notion that this must then apply to java. > If the study w…

>it says nothing even remotely resembling "this benefit applies to all languages with static typing". That is precisely what it says, and that is reiterated later: "...the conclusion can be reached that...in practice [unit testing] is an inadequate replacement for static type checking." I'm not sure what you're reading, but there's no qualifications in the language used here regarding the idea of 'static type checkin…

When it says "static type checking" it does not mean "all static type checking" but rather "good static type checking". And this is what the study showed (ignoring issues of methodology and sample size for the sake of argument): a (good) static type system would have caught more errors than unit testing, therefore static typing is good.

Generalizing any comment to all static type systems is silly: there are language like C that have a static system but provide basically no additional safety at all. You can easily provide examples of really bad statically typed or dynamically typed languages, but these examples say nothing of static or dynamic typing in general: they're just bad. Questions about static vs dynamic typing can only be answered by the best (or at least good) examples of each.

Showing that a good statically typed system is more robust than a good dynamically typed system is a useful proxy for comparing static typing to dynamic typing. This is similar to a study on seat belts ignoring poor seat belts that strangle the passengers in the event of a crash.

In short: just because static typing is better does not mean all static type systems are better, because you can always come up with a sufficiently bad example of static typing.

Re: Experiment: Unit testing isn't enough; You need static types, too

#225

Post by cageface » Wed, Jun 20, 2012, 9:10 PM UTC

gnuvince wrote: post #159

cageface wrote: post #91

Earlier quoted context omitted.

I started a parallel implementation of a bunch of machine learning algorithms in Clojure and Scala. I figured the Clojure version would pull out ahead thanks to my previous lisp experience and lisp's history in that domain. I was very surprised to discover that my Scala code was a lot easier to understand and maintain and had fewer bugs.

What do you attribute this to? The static typing of Scala or the libraries that you use?

Definitely typing. I wasn't using any libraries.

Re: Experiment: Unit testing isn't enough; You need static types, too

#226

Post by scott_s » Wed, Jun 20, 2012, 9:16 PM UTC

papsosouid wrote: post #154

davesims wrote: post #146

Earlier quoted context omitted.

Had the study qualified itself to merely "Haskell vs. Python" with deference given to the statistical significance of the sample size, you'd have a point. It wasn't me that brought all static typing , which of course includes Java, into the question at hand -- it was the study itself.

Yes, it was you. Why do you think the comparison should be "really bad static type system" vs "really good dynamic type system"? In what way does that make the test more useful? Allow me to say this again, as I do not know how to be any clearer: You do not test the potential of something by using the worst possible example of it. The only point of your desire is to reinforce the strawman that java = static typing. A…

I don't think davesims is saying that should be the comparison. This particular complaint is about the conclusions, not the methodology. (I recognize he also criticized the methodology.) Conclusions should be useful. People shouldn't have to squint at the wording of your conclusion to determine what that means for them. So, you should bend over backwards in your conclusion, and err on the side of being clear.

With that in mind, I agree with davesims that the conclusion in the blog post is too strong. It is: "The application of static type checking to many programs written in dynamically typed programming languages would catch many defects that were not detected with unit testing" I say it is too strong because the author has not bent over backwards to make clear that this conclusion only applies to the "best" type systems, like Haskell.

For the record, I like the study, and once I run the author's conclusions through my bend-over-backwards-filter, I find them interesting. I upvoted this article. I also upvoted davesims' post because it is academic-reviewer level feedback.

Re: Experiment: Unit testing isn't enough; You need static types, too

#227

Post by ufo » Wed, Jun 20, 2012, 9:17 PM UTC

pimentel wrote: post #38

sbmassey wrote: post #14

Earlier quoted context omitted.

Haskell type classes are sufficiently flexible to represent things like 'does this type have a next method' without having to instrument the actual type.

But could that kind of type system be considered similar to the usual Java-like little-flexible static typing? I think I would prefer a static vs dynamic implemented as Java vs Python (since those are usually the subjects on every one of these discussions)

ALthough the name is confusing type classes are very much like interfaces in Java: you define a set of operations that you want and then instantiate concrete types to that interface.

The major differences between type classes and interfaces are: * In Java interfaces the interfaced type is restricted to the first argument (the this). In Haskell interfaces the interfaced type can also appear on return values and arguments. * In Java you need to decide what interfaces to implement when you create your class. Haskell allows interfaces to be implemented for previously existing types. * Java allows for subtyping. You can turn a monomorphic program into a polymorphic one just by creating subclasses while in Haskell you would need to rewrite your code to be explicitly polymorphic.

Re: Experiment: Unit testing isn't enough; You need static types, too

#228

Post by scott_s » Wed, Jun 20, 2012, 9:18 PM UTC

papsosouid wrote: post #199

anamax wrote: post #195

Earlier quoted context omitted.

> You do not test the potential of something by using the worst possible example of it. So? Folks don't use the "potential", they use the real. They're asking questions like "should I use Java or Python". > do airbags help prevent deaths" would be a very poor test if it used anything other than the best possible airbag technology. That's not how things actually work. You decide between what's available. The performan…

>So? Folks don't use the "potential", they use the real. Haskell is real. >They're asking questions like "should I use Java or Python". That's wonderful, but it has nothing to do with the subject at hand, which was the question "can static typing reduce the number of bugs?". If you want an answer to a different question, don't complain about the answer given for this question, go find someone answering the question y…

I believe you are confusing criticisms of the methodology with criticisms of the strength of conclusions.

Re: Experiment: Unit testing isn't enough; You need static types, too

#229

Post by scott_s » Wed, Jun 20, 2012, 9:21 PM UTC

CodeMage wrote: post #209

davesims wrote: post #143

Earlier quoted context omitted.

>it says nothing even remotely resembling "this benefit applies to all languages with static typing". That is precisely what it says, and that is reiterated later: "...the conclusion can be reached that...in practice [unit testing] is an inadequate replacement for static type checking." I'm not sure what you're reading, but there's no qualifications in the language used here regarding the idea of 'static type checkin…

> That is precisely what it says This is a very strong claim and it's false. The article doesn't say that anywhere. You interpret it that way. I would hazard a guess that presenting your own interpretation as fact is what brought on those downvotes you complain about below.

When you present conclusions in an academic paper, the onus is on the author to bend over backwards to prevent the reader from interpreting a stronger conclusion than intended. I think davesims' interpretation is fair given the language, and I were I reviewing the paper, I would have asked the author to temper his conclusions in a similar manner.

Re: Experiment: Unit testing isn't enough; You need static types, too

#230

Post by cageface » Wed, Jun 20, 2012, 9:24 PM UTC

dochtman wrote: post #109

cageface wrote: post #10

I'm convinced that dynamically typed languages are a transitional technology that will be superseded once we develop type systems that are both usefully strict but also flexible. After over ten years working in dynamic languages I'm very happy to have a compiler on my side again.

Seems to me that that's what static typing proponents have been saying for years (if not decades), and it still isn't true. That said, I'm currently hacking on a compiler to see if I can come up with a design for a static language that's almost as pleasant to use as a dynamic one, so I'm not completely hopeless. But it certainly seems like the data point that until now no such superior type system (strict but flexibl…

That's because it takes years or even decades to really get this right, like a lot of other sophisticated technology. Look how long it took to get the JVM to where it us today. This is fundamental research and hard stuff.