Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experiment: Unit testing isn't enough; You need static types, too

evanfarrer.blogspot.ca

Post reply on HN
Print view View original

231–240 of 276 posts

Re: Experiment: Unit testing isn't enough; You need static types, too

#231

Post by scott_s »

wissler wrote: post #160
davesims wrote: post #48

The sample size of this study is statistically too insignificant to draw the conclusions given. The counterfactuals to the claims of dynamic type advocates were already true and provable, so even if the sample size of the codebases studied were statistically significant (not to mention vetted for quality of unit test as well as coverage ) the conclusions are nevertheless trivial. In addition, the hidden assumption is…

You make a good point -- I don't think any statistical study will ever be able to show that static typing is better. I do think that a rational argument can show it, but my argument is too long to fit into the margin.

I think that a study over a broad set of applications of considerable complexity could provide enough statistical evidence that most people would be comfortable coming to a conclusion. That study, though, would take a very large effort. Large enough that it may never be done.

Re: Experiment: Unit testing isn't enough; You need static types, too

#232

Post by tikhonj »

davesims wrote: post #185
davesims wrote: post #143

Earlier quoted context omitted.

>it says nothing even remotely resembling "this benefit applies to all languages with static typing". That is precisely what it says, and that is reiterated later: "...the conclusion can be reached that...in practice [unit testing] is an inadequate replacement for static type checking." I'm not sure what you're reading, but there's no qualifications in the language used here regarding the idea of 'static type checkin…

From the downvotes I can only conclude that many of you wish the study didn't claim what it claims and are merely shooting the messenger. If anyone can point out rhetoric within the study that qualifies it in such a way as to make comparisons of other statically typed languages with other dynamically typed languages out-of-bounds or expressing a false equivalence within the scope of the conclusions of the study itsel…

To simplify: there is a difference between "static typing is better than dynamic typing" and "all static typing is always better than all dynamic typing". It's basically the difference between ∃ and ∀.

Saying that "static typing is better than dynamic typing" is like the former: there exists some static typing system that is better than dynamic typing. Saying that "all static type systems are better than any dynamic system" is like the second. All the paper ever says is the first: "Based on these results, the conclusion can be reached that while unit testing can detect some type errors, in practice it is an inadequate replacement for static type checking." Note how it never claims to apply for all possible static type systems; rather, it just says that tests are an inadequate replacement for type systems in general (i.e. there exists some type system that catches more errors than tests). This is exactly like my first example.

In summary: a being better than b does not mean that all a is always better than all b. Just because static typing is better than dynamic typing does not imply that Java is always better than Python; it merely implies that some statically typed language is better than Python.

Re: Experiment: Unit testing isn't enough; You need static types, too

#233

Post by [deleted] »

papsosouid wrote: post #213
x1 wrote: post #207

Earlier quoted context omitted.

> In the rest of my program, those functions would be the only way to talk to the database. There's your guarantee. Honest question. Take these pseudo sql calls: //Bad Person username = "lastname'; drop table user--" //Good Programmer query = "select * from users where name like %[username]%"; input = {"username":"frank"}; result = execute(query,input); //Bad Programmer query = "select * from users where name like '%…

In your static example, "Bad Programmer" would be fine, because the Query constructor does escaping. You could do this in a dynamically typed language too, but notice that you don't, you just use strings. The difference between static and dynamic is that with static typing, you can't compile your incorrect program. With dynamic typing, you find out at run time that you forgot to escape the string (turning it into a Q…

[deleted]

Re: Experiment: Unit testing isn't enough; You need static types, too

#234

Post by scott_s »

tikhonj wrote: post #232
davesims wrote: post #185

Earlier quoted context omitted.

From the downvotes I can only conclude that many of you wish the study didn't claim what it claims and are merely shooting the messenger. If anyone can point out rhetoric within the study that qualifies it in such a way as to make comparisons of other statically typed languages with other dynamically typed languages out-of-bounds or expressing a false equivalence within the scope of the conclusions of the study itsel…

To simplify: there is a difference between "static typing is better than dynamic typing" and "all static typing is always better than all dynamic typing". It's basically the difference between ∃ and ∀. Saying that "static typing is better than dynamic typing" is like the former: there exists some static typing system that is better than dynamic typing. Saying that "all static type systems are better than any dynamic…

I agree with your characterization in your first paragraph, but I agree with davesims that the conclusions are too strong. If one has to do the level of analysis of the conclusions that you present in your second paragraph, then they are poorly worded. I find davesims' interpretation a reasonable one, which leads me to agree that the conclusions need to be tempered and clarified.

Re: Experiment: Unit testing isn't enough; You need static types, too

#235

Post by tikhonj »

glenjamin wrote: post #8

I applaud the effort to try and dissect the problem scientifically. If a whole program has 1 bug due to being implemented with dynamic types over static types, and that bug has gone unnoticed, then it can't be particularly important. The other common proposition is that dynamically typed languages are faster to write in than statically typed languages. If this is true then we need to compare the saving in development…

An interesting thought: a good type system can actually make a language more expressive.

A perfect example is QuickCheck. QuickCheck allows you to write complicated tests very simply by relying on the type system. You just write out the invariant and the type system automatically figures out which random generators to use to run the tests.

QuickCheck has been ported to a bunch of other languages, but it's more complex and seems harder to use in dynamically typed languages as compared to Haskell.

A simpler example in the same vein is Haskell's read function. Essentially, read is the opposite of toString--it goes from a string to some value. The beauty is that you never need to specify what type you're parsing; it can figure out what type it needs to be thanks to the type system. So instead of having a bunch of functions like parseDouble and parseInt, you have a single read function. This also makes the library prettier by maintaining the symmetry between show and read (toString and fromString).

Re: Experiment: Unit testing isn't enough; You need static types, too

#236

Post by [deleted] »

papsosouid wrote: post #213
x1 wrote: post #207

Earlier quoted context omitted.

> In the rest of my program, those functions would be the only way to talk to the database. There's your guarantee. Honest question. Take these pseudo sql calls: //Bad Person username = "lastname'; drop table user--" //Good Programmer query = "select * from users where name like %[username]%"; input = {"username":"frank"}; result = execute(query,input); //Bad Programmer query = "select * from users where name like '%…

In your static example, "Bad Programmer" would be fine, because the Query constructor does escaping. You could do this in a dynamically typed language too, but notice that you don't, you just use strings. The difference between static and dynamic is that with static typing, you can't compile your incorrect program. With dynamic typing, you find out at run time that you forgot to escape the string (turning it into a Q…

[deleted]

Re: Experiment: Unit testing isn't enough; You need static types, too

#237

Post by davesims »

CodeMage wrote: post #217
davesims wrote: post #211

Earlier quoted context omitted.

Can you show how I've misinterpreted the plain language of the conclusion section? I'm under the (perhaps mistaken) assumption that in academic papers people tend to mean what they say and choose their language carefully, particularly in the conclusion section. If the following are not in fact broad, strong claims about the nature of static and dynamic languages in general , then won't you please explain to me how I…

Honestly, at this point I can no longer tell whether you're misinterpreting or misrepresenting the conclusions. I'll make an honest attempt to argue, nevertheless. "Static type checking" and "unit testing" are two concepts. There are numerous concrete implementations of these two concepts. The former is implemented in several languages, including C++ and Java and Haskell. The latter is implemented in several framewor…

"All in all, the sweeping generalization about the concrete languages was introduced by you."

I think the plain, direct language of the paper's conclusion is clear enough without me having to embellish it, and without its defenders extrapolating all of the qualifications and subtexts that they think I missed. You really don't have much to work with, because the paper's clumsy conclusion is small, blunt and unqualified in its scope. It takes a handful of small Python programs translated to an idiosyncratic language like Haskell and concluded:

"in practice [dynamic typing with unit testing] is an inadequate replacement for static type checking."

This is unequivocal language. There's no qualifications about language, context, or any kind of variables that might possibly dilute the strength of the conclusion.

On the other hand, Peter Cooper gives a great example elsewhere on this thread of a much better paper with much broader scope, more stats, and much more modest, qualified conclusions. This is the kind of language that is useful and gives me confidence that the authors didn't start out with an axe to grind and merely followed what metrics they had to the warranted conclusion, no more, no less:

"Even though the experiment seems to suggest that static typing has no positive impact on development time, it must not be forgotten that the experiment has some special conditions: the experiment was a one-developer experiment. Possibly, static typing has a positive impact in larger projects where interfaces need to be shared between developers. Furthermore, it must not be forgotten that previous experiments showed a positive impact of static type systems on development time."

http://www.cs.washington.edu/education/courses/cse590n/10au/...

Re: Experiment: Unit testing isn't enough; You need static types, too

#238

Post by tikhonj »

pyre wrote: post #168
Peaker wrote: post #139

Earlier quoted context omitted.

In Python, there are a lot of errors like "NoneType has no attribute '...'", and those disappear too. Missing imports and redundant imports also go away. Lots of lots of invariants in the program can be encoded as types, too, so any bugs relating to them go away too. When you want parallelism, you get useful guarantees about not changing the deterministic result you had before you added parallelism. I used to use Pyt…

> "NoneType has no attribute '...'" C is statically typed, but I can still attempt to dereference a null pointer. Static typing doesn't save me here, nor does the compiler, as it's possible for these issues to happen at runtime. This may be something that Haskell doesn't allow, but it's not something inherent to static-typing.

Intercal is dynamically typed, but it doesn't save me any development time or make my code any shorter! (Well, maybe compared to Java :)).

On top of this, the C type system is not really about correctness at all. My understanding is that it primarily helps with performance, memory management (e.g. you know the size of stuff) and not accidentally using a non-pointer as a pointer. I'm not a C person, but C does not give off a vibe of caring about correctness.

In fact, C is particularly unsafe: you can get all sorts of fun things like bus errors and segfaults that are basically impossible in other languages. C definitely has a place, but only if correctness is much less important than performance.

You can ultimately come up with a sufficiently bad language for anything.

Also, the way Haskell avoids null errors like this is with the static type system. So while it's certainly not inherent to all static type systems (then again, nothing has to be inherent to static type systems except being verified at compile time), it is a property of the type system.

Re: Experiment: Unit testing isn't enough; You need static types, too

#240

Post by jaylevitt »

hackinthebochs wrote: post #165
davesims wrote: post #162

Earlier quoted context omitted.

> Yeah, no shit. Stop drinking warm coke. Fine, then don't claim something like "All cokes in all contexts at all temperatures are better than all pepsis in all contexts at all temperatures." This is equivalent to what the study does with static vs. dynamic. Your argument, if you actually had a point, would be something along the lines of, "wait I'm talking about this boutique hand-crafted cola (Haskell) I get at Who…

I'm gonna have to disagree with you about the conclusion you're drawing. Yes, they are using the generic phrasing of "static typing" vs "dynamic typing", but this is because the study was intended to test the concept of static vs dynamic typing, not particular instances of it. However, seeing as we only have specific instances from which to test, it used the best one currently in widespread use. I don't see this as a…

But isn't it problematic that it compared real-world average unit tests with the best-available type system?
Post reply on HN