Experiment: Unit testing isn't enough; You need static types, too • Hacker News Board

Re: Experiment: Unit testing isn't enough; You need static types, too

#251

Post by anamax » Thu, Jun 21, 2012, 2:31 AM UTC

papsosouid wrote: post #199

anamax wrote: post #195

Earlier quoted context omitted.

> You do not test the potential of something by using the worst possible example of it. So? Folks don't use the "potential", they use the real. They're asking questions like "should I use Java or Python". > do airbags help prevent deaths" would be a very poor test if it used anything other than the best possible airbag technology. That's not how things actually work. You decide between what's available. The performan…

>So? Folks don't use the "potential", they use the real. Haskell is real. >They're asking questions like "should I use Java or Python". That's wonderful, but it has nothing to do with the subject at hand, which was the question "can static typing reduce the number of bugs?". If you want an answer to a different question, don't complain about the answer given for this question, go find someone answering the question y…

> can static typing reduce the number of bugs

No one claims otherwise. However, that's true of Java's type system too.

> Why can't anyone follow a simple line of reasoning without resorting to fallacies?

I followed your simplistic line of reasoning just fine. It was wrong. Admit that and move on.

Of course you can't, which is how you got there.

The biggest obstacle to Haskell becoming more popular is its advocates.

And, it will never replace Java, C, Python, or even PHP. (One of my professional goals is to never use Java.)

Re: Experiment: Unit testing isn't enough; You need static types, too

#252

Post by anamax » Thu, Jun 21, 2012, 2:31 AM UTC

papsosouid wrote: post #199

anamax wrote: post #195

Earlier quoted context omitted.

> You do not test the potential of something by using the worst possible example of it. So? Folks don't use the "potential", they use the real. They're asking questions like "should I use Java or Python". > do airbags help prevent deaths" would be a very poor test if it used anything other than the best possible airbag technology. That's not how things actually work. You decide between what's available. The performan…

>So? Folks don't use the "potential", they use the real. Haskell is real. >They're asking questions like "should I use Java or Python". That's wonderful, but it has nothing to do with the subject at hand, which was the question "can static typing reduce the number of bugs?". If you want an answer to a different question, don't complain about the answer given for this question, go find someone answering the question y…

> can static typing reduce the number of bugs

No one claims otherwise. However, that's true of Java's type system too.

> Why can't anyone follow a simple line of reasoning without resorting to fallacies?

I followed your simplistic line of reasoning just fine. It was wrong. Admit that and move on.

Of course you can't, which is how you got there.

The biggest obstacle to Haskell becoming more popular is its advocates.

And, it will never replace Java, C, Python, or even PHP. (One of my professional goals is to never use Java.)

Re: Experiment: Unit testing isn't enough; You need static types, too

#253

Post by anamax » Thu, Jun 21, 2012, 2:40 AM UTC

papsosouid wrote: post #200

anamax wrote: post #193

Earlier quoted context omitted.

> It uses the best static typing system to do so. It doesn't use the best dynamic language or best unit tests.

Then you should be proposing he use whatever language you feel is better than python at being the best dynamic type system. The best unit tests is entirely irrelevant.

> Then you should be proposing he use whatever language you feel is better than python at being the best dynamic type system.

Nope.

> The best unit tests is entirely irrelevant

I can find errors in programs with a spell checker. Suppose that those programs have unit tests. Do you really think that spell checker is better than unit tests?

Re: Experiment: Unit testing isn't enough; You need static types, too

#254

Post by carterschonwald » Thu, Jun 21, 2012, 3:45 AM UTC

pja wrote: post #42

carterschonwald wrote: post #32

Earlier quoted context omitted.

False, ocaml has type classes now! (or maybe I'm thinking of coq)

I don't think so: If you want the equivalent of Haskell type classes in OCaml then you have to either use a functors or some sort of class IIRC. I'm sure Oleg will have done something though. I believe they're experimental in Coq.

yup, you're right, you got to use functors or go home.

Yeup, some folks did show the functors + modules \equiv type classes in some sense, for system F or a variant thereof. Theres also an oleg approach too I think.

Re: Experiment: Unit testing isn't enough; You need static types, too

#255

Post by dllthomas » Thu, Jun 21, 2012, 3:51 AM UTC

saraid216 wrote: post #239

dllthomas wrote: post #189

Earlier quoted context omitted.

What's wrong with "dynamism"?

Consider: staticity staticness statism

That doesn't work because statism comes from "state", not "static" - I don't think there's a comparable derivation for "dynamism".

Re: Experiment: Unit testing isn't enough; You need static types, too

#256

Post by efarrer » Thu, Jun 21, 2012, 5:00 AM UTC

davesims wrote: post #48

The sample size of this study is statistically too insignificant to draw the conclusions given. The counterfactuals to the claims of dynamic type advocates were already true and provable, so even if the sample size of the codebases studied were statistically significant (not to mention vetted for quality of unit test as well as coverage ) the conclusions are nevertheless trivial. In addition, the hidden assumption is…

I (the author) appreciate the feedback. I believe that many of your criticisms are addressed in the actual paper. First of all I completely agree that my sample size is too small for a conclusive proof. I mention in the paper that I hope that others will try and replicate this experiment on other pieces of software. I do think it's appropriate when conducting an experiment to publish a conclusion, not that the experiment will constitute proof (or an established scientific theory), but as a conclusion to the study that others can try to confirm or refute.

I also mention in the paper that it would be beneficial to conduct this experiment using different type systems for the reasons that you stated above.

The argument against static typing that I was testing didn't mention any particular type system nor any particular dynamically typed language, it was a general argument that stated that unit testing obviated static typing. Because the argument was so general and absolute I felt that any static type system that could be shown to expose bugs that were not caught by unit testing would be enough to refute the argument. I was not trying to prove that any type system would catch bugs not found by any unit tested software. The paper also points out that I'm trying to see whether unit testing obviates static typing in practice, in theory you could implement a poor mans type checker as unit tests, but my experiment was focused on whether in practice unit testing obviates static typing.

Finally I believe that my conclusion in the paper was at least a bit more modest than that of the blog post. The lack of apparent modesty in the blog post was caused more by a lack of ability on my part to accurately summarize than an inflated sense of accomplishment and self importance.

Re: Experiment: Unit testing isn't enough; You need static types, too

#257

Post by efarrer » Thu, Jun 21, 2012, 5:25 AM UTC

kingkilr wrote: post #21

If you can convert a program from one language to another (which is non-trivially different), in the time it takes to complete your masters, I'm pretty sure it wasn't a very interesting program. Further, the quality of the developers is going to play a large role in how effective any tool (and make no mistake, static typing is a tool) is. This is not intended as a disparaging remark to the authors, but in the 30 seco…

It seems like you're two criticisms are as follows: 1. The sample size is too small. 2. The quality of the Python code isn't that great.

I (the author) would like to address both of these. First of all I completely agree that more research needs to be done. I mention this in the paper. I have provided a data point not a proof. It took me a couple months of several hours a day to the do the translation, I hope more people translate more programs to see if the results hold in the face of a larger data set. Second I agree the quality of the Python code isn't that great. I wanted to see whether unit testing obviated static typing in practice. In order to avoid selection bias I choose the projects at random. I picked the first four projects that were I believe that my methodology, nor my conclusions are flawed but that all should remember that a single experiment does not make a scientific proof. I hope that others will try to replicate this experiment on many more code bases. If they do it will be interesting to see the results.

Re: Experiment: Unit testing isn't enough; You need static types, too

#258

Post by jaylevitt » Thu, Jun 21, 2012, 5:27 AM UTC

zopa wrote: post #246

x1 wrote: post #207

Earlier quoted context omitted.

> In the rest of my program, those functions would be the only way to talk to the database. There's your guarantee. Honest question. Take these pseudo sql calls: //Bad Person username = "lastname'; drop table user--" //Good Programmer query = "select * from users where name like %[username]%"; input = {"username":"frank"}; result = execute(query,input); //Bad Programmer query = "select * from users where name like '%…

> "Right now I see no difference between the good and bad" You're building a new query string each time you create a Query object, and concatenating the string onto that. With that approach, each time you build a Query object you have a fresh opportunity to mess up. So you're right that there's no difference between your to cases. Let's drop my off-the-cuff example and look at how a real library, postgresql-simple, h…

Sure, and Rails offers a similar syntax:

  User.select('x').where('name like ?', username)

But if your language allows literal string interpolation (as Ruby does), what prevents you from doing this:

query conn ("select x from users where name like #{username}")

How do type-safe languages prevent this?

Re: Experiment: Unit testing isn't enough; You need static types, too

#259

Post by efarrer » Thu, Jun 21, 2012, 5:31 AM UTC

timruffles wrote: post #2

I wish more of these open questions in coding (and everything) had people making such a good effort to research them! There never is a simple answer, or they'd not be an open question. Really inspiring, painstaking piece of work.

Thanks for the compliment, one of my goals is to put the science back in computer science.

Re: Experiment: Unit testing isn't enough; You need static types, too

#260

Post by efarrer » Thu, Jun 21, 2012, 5:44 AM UTC

njharman wrote: post #112

> "frequently cited claim by proponents of dynamically typed programming languages that static typing was not needed for detecting bugs in programs" Who says that? There's trade offs, multitudes, in choosing paradigm / language. It's never so black and white (except for academics, and twits who like to argue more than code) I did not read paper, don't have time for 60 pages of pointlessness.

I agree there are generally trade offs, that's essentially the results of my "60 pages of pointlessness" paper. If you did have time to read the paper you would notice a reference to this book http://my.safaribooksonline.com/book/software-engineering-an... that is an agument for unit testing instead of static typing. I think we need to use the scientific method in computer science and not just base our ideas of intuition, belief or absolutes like "It's never so black and white".