Live data from Hacker News

Experiment: Unit testing isn't enough; You need static types, too

evanfarrer.blogspot.ca

261–270 of 276 posts

Re: Experiment: Unit testing isn't enough; You need static types, too

#261
post #208
post #135

Earlier quoted context omitted.

I think Go's type system is decades behind the state of the art. I can't believe they repeated the mistake of Boolean Blindness [1]. Sum types and pattern-matching are crucial for useful strictness with flexibility. This results in funny things like encoding the optional error result in Go as a type product rather than a sum, allowing reading of a result even if it does not exist due to an error. [1]: http://existent…

...your problem with Go is that it has a boolean type?

No, did you read that article?

My problem with Go is that it does not have sum-types and pattern-matching. This means that branching (conditionals) in Go do not gain any type-information. And that means that programmers have to manually keep track of the invariants that hold true in each of their conditionals, and if they get them wrong, they get no help from the compiler.

Re: Experiment: Unit testing isn't enough; You need static types, too

#262
post #251

Earlier quoted context omitted.

>So? Folks don't use the "potential", they use the real. Haskell is real. >They're asking questions like "should I use Java or Python". That's wonderful, but it has nothing to do with the subject at hand, which was the question "can static typing reduce the number of bugs?". If you want an answer to a different question, don't complain about the answer given for this question, go find someone answering the question y…

> can static typing reduce the number of bugs No one claims otherwise. However, that's true of Java's type system too. > Why can't anyone follow a simple line of reasoning without resorting to fallacies? I followed your simplistic line of reasoning just fine. It was wrong. Admit that and move on. Of course you can't, which is how you got there. The biggest obstacle to Haskell becoming more popular is its advocates. A…

> And, it will never replace Java, C, Python, or even PHP.

What do you mean by that?

Many people, including myself, have had Haskell replace Python.

Re: Experiment: Unit testing isn't enough; You need static types, too

#263

Earlier quoted context omitted.

I'm gonna have to disagree with you about the conclusion you're drawing. Yes, they are using the generic phrasing of "static typing" vs "dynamic typing", but this is because the study was intended to test the concept of static vs dynamic typing, not particular instances of it. However, seeing as we only have specific instances from which to test, it used the best one currently in widespread use. I don't see this as a…

But isn't it problematic that it compared real-world average unit tests with the best-available type system?

I don't think so -- anyone is free to choose to use the best-available type system. You can't just choose to write the best possible unit tests.

He could only compare one of the best possible environments for writing dynamically typed code and unit tests to one of the best possible environments for writing statically typed code.

Re: Experiment: Unit testing isn't enough; You need static types, too

#264
post #207

Earlier quoted context omitted.

> In the rest of my program, those functions would be the only way to talk to the database. There's your guarantee. Honest question. Take these pseudo sql calls: //Bad Person username = "lastname'; drop table user--" //Good Programmer query = "select * from users where name like %[username]%"; input = {"username":"frank"}; result = execute(query,input); //Bad Programmer query = "select * from users where name like '%…

In your static example, "Bad Programmer" would be fine, because the Query constructor does escaping. You could do this in a dynamically typed language too, but notice that you don't, you just use strings. The difference between static and dynamic is that with static typing, you can't compile your incorrect program. With dynamic typing, you find out at run time that you forgot to escape the string (turning it into a Q…

If he builds the final query string before giving it to Query, his valid query parts that rely on not being escaped would also be escaped.

To make a safe query type you'd have to provide non-string primitives to build one, if I understand correctly. You can't allow just a full query string (with all of the injections already in place) to be converted to a Query type (as in his Bad Programmer example).

Re: Experiment: Unit testing isn't enough; You need static types, too

#265
post #175

Earlier quoted context omitted.

>I never suggested that Yes, you did suggest that. I am not sure how this level of cognitive dissonance is possible. What possible purpose does your example serve then if it doesn't impart any sort of meaning at all? >I'm not even sure what we are arguing about Clearly. Please, take the time to think through the subject and present a clear point that you will not later pretend you didn't make.

> What possible purpose does your example serve then if it doesn't impart any sort of meaning at all? The example shows that static typing doesn't do anything more than what it says. It doesn't solve problems/fix bugs or provide some magical insight to the system as you seem to believe. I'm genuinely curious as to your position and why you are so... clearly opinionated. I'll take the "idiot banner" for today. Please…

I think there are two points that are being mostly missed in this discussion:

A) How to define the Query type such that it would convert injection bugs to type-checking errors (No, it cannot simply be a function from a full String containing a query to a Query type, as you demonstrated).

B) Sure, you could define the same Query primitives to do the same in a dynamically typed language. The main difference is that the type-checking errors due to incorrect use of the query primitives would be caught at runtime.

As for A, you would want to define primitives that build query strings safely. That is: Query(unsafe_string_here) wouldn't work. Either because it allows too much (still can inject the original string) or disallows too much (escapes everything, makes the query invalid).

Instead, you would define "select", "update" and other querying primitives as non-string primitives you can use to build queries. You would basically mirror SQL or the query language you use into non-string primitives that allow constructing safe queries.

B) Yes, you could do this with dynamically typed languages. Right before executing your query you would need to do an isinstance() check or some other way to validate that the query was generated using the safe machinery. This means no duck typing. If you allow other, unsafe implementations of the query type here, you get the unsafety back.

Re: Experiment: Unit testing isn't enough; You need static types, too

#266

Earlier quoted context omitted.

I don't think page counts of specs or feature lists really tell you that much. I didn't find it that hard too get up to speed in Scala but it seems to scare away too many Java people. My main criticism is that it allows too much syntactic flexibility.

I'm a programmer that in general prefers dynamic languages. Recently I've started writing production code in Scala for a couple of web services for which I really needed the performance and flexibility of the JVM. Scala does have problems. But NOT the language. I find the language to be extremely elegant and well designed. The problem lies with the community. I find Scala libraries to be an abomination of taste and c…

David Pollak has long been one of Scala's greatest champions but his grudging acceptance that Scala is hard, from a lot of experience in the field, is worth a read:

http://blog.goodstuff.im/yes-virginia-scala-is-hard/

Re: Experiment: Unit testing isn't enough; You need static types, too

#267
post #251

Earlier quoted context omitted.

>So? Folks don't use the "potential", they use the real. Haskell is real. >They're asking questions like "should I use Java or Python". That's wonderful, but it has nothing to do with the subject at hand, which was the question "can static typing reduce the number of bugs?". If you want an answer to a different question, don't complain about the answer given for this question, go find someone answering the question y…

> can static typing reduce the number of bugs No one claims otherwise. However, that's true of Java's type system too. > Why can't anyone follow a simple line of reasoning without resorting to fallacies? I followed your simplistic line of reasoning just fine. It was wrong. Admit that and move on. Of course you can't, which is how you got there. The biggest obstacle to Haskell becoming more popular is its advocates. A…

>I followed your simplistic line of reasoning just fine. It was wrong. Admit that and move on.

You are wrong, admit it and move on. Oh gee, does that not actually make a constructive argument?

>The biggest obstacle to Haskell becoming more popular is its advocates.

What does this have to do with anything?

>And, it will never replace Java, C, Python, or even PHP

It already has. You might be too foolish to take advantage of that fact, but how does your foolishness matter to me?

Re: Experiment: Unit testing isn't enough; You need static types, too

#268
post #253

Earlier quoted context omitted.

Then you should be proposing he use whatever language you feel is better than python at being the best dynamic type system. The best unit tests is entirely irrelevant.

> Then you should be proposing he use whatever language you feel is better than python at being the best dynamic type system. Nope. > The best unit tests is entirely irrelevant I can find errors in programs with a spell checker. Suppose that those programs have unit tests. Do you really think that spell checker is better than unit tests?

Are you trolling or incapable of reading? Nobody, at any point in time suggested that static typing was an alternative to unit testing. You haven't posted a single constructive thing in this entire thread, and you waited till it was over to do your trolling so you could avoid downvotes. Grow up, or go back to reddit.

Re: Experiment: Unit testing isn't enough; You need static types, too

#269

Earlier quoted context omitted.

>From my personal experience, the saving in development time is very much real. But your personal experience is comparing expressive languages to unexpressive languages, and then making the mistake of thinking you are comparing dynamically typed languages to statically typed languages. What does .net's ugly (in your opinion) API have to do with static typing? Factory-Factory-Factory patterns is a joke even in java, b…

Got to say this is hard to admit, but after reading everyone else's comments, I am indeed ignorant. Thanks for the reality check.

There shouldn't be a stigma about ignorance like there is. Everyone is ignorant of lots of stuff, nobody knows everything. Knowing that you don't know something is good, not something to be ashamed of. If you are aware of what you don't know, then you can learn it.

Re: Experiment: Unit testing isn't enough; You need static types, too

#270

The sample size of this study is statistically too insignificant to draw the conclusions given. The counterfactuals to the claims of dynamic type advocates were already true and provable, so even if the sample size of the codebases studied were statistically significant (not to mention vetted for quality of unit test as well as coverage ) the conclusions are nevertheless trivial. In addition, the hidden assumption is…

I (the author) appreciate the feedback. I believe that many of your criticisms are addressed in the actual paper. First of all I completely agree that my sample size is too small for a conclusive proof. I mention in the paper that I hope that others will try and replicate this experiment on other pieces of software. I do think it's appropriate when conducting an experiment to publish a conclusion, not that the experi…

Thanks for the response! I appreciate the effort you went to here, this was no small task you set yourself to.

I appreciate the clarification. I think now I see better where your emphasis was: the purpose of the paper was to refute an argument, and of course the level of burden of proof is different and far less in that case. I think this misunderstanding on my part is what caused me to call the conclusions 'trivial' -- too strong and dismissive language on my part anyway.

The irony is, you were attempting to do to the unit-testing-is-sufficient argument what I was attempting to do to what I assumed yours was: provide one counter-example to falsify a broad and generalized thesis.

That said, I think I would have liked to have seen your original unit-testing-is-sufficient argument punched up and qualified into something a little more reasonable and real-world. As you stated the argument, it seems like a straw man to me. It seems one could reduce your version of the argument to something like: "Dynamic languages with unit test coverage will always catch errors that statically-typed environments will." And of course this is far too broad and unqualified a statement, and that is precisely why all you needed was one counter-factual to refute it. You didn't even need a handful of Python programs, or 9 or 20 or 100 errors to prove your point. You only needed one, as you stated above. This is why the burden of proof for your thesis was so small, but also why, in my opinion, even with that reduced scope and more modest conclusion, we haven't really learned much.

As someone who has spent most of my career in statically-typed environments and the last 6 years or so mostly in dynamic environments, and also as someone who has made something like the argument you were attempting to refute, I have to say I would definitely never have made such a brittle and unqualified statement as the one you refuted in your paper. To put it more directly, I think I'm probably a poster-child for the kind of developer you were aiming your thesis at, and I don't feel that my perspective was adequately or reasonably represented. More importantly, having looked at the examples given in your paper, I may have learned a bit about the kinds of errors that Haskell can catch automatically that some coders might miss in a dynamic environment, but not much useful to me in my everyday work context.

I think a more reasonable version of the argument, but more qualified and therefore requiring a far larger sampling of code to prove or refute, would be something like: "Programs written in a dynamic language with adequate or near-100 percent unit test coverage, are no more prone to defects than programs written in a statically-typed language with a comparable level of unit test coverage."

I agree this is a very important conversation to have, and again kudos to the work you put in here. Obviously people have strong opinions both directions, and the discussion, however heated at various moments, is an important one, so thanks for this!

Post reply on HN