Study of 49 programmers: static type system had no effect on development time • Hacker News Board

Re: Study of 49 programmers: static type system had no effect on development time

#131

Post by nuncanada » Fri, Jun 08, 2012, 1:25 PM UTC

edanm wrote: post #57

fusiongyro wrote: post #32

The study may or may not be flawed, but what's really interesting to me is the reaction. We need more science in our computer science, which means more experiments and more results like this. We should also be open to the truth that we use the tools we like because we like them rather than because they're technically superior, even though we pimp them ad nauseum as though they are. I once read an article about a tech…

I think your point is important, and well made. The only problem is, people have been doing these experiments for 30 years, and do you know what the net effect it's had on the world of programmers: none at all. Saying "no no, this time really listen to this study " seems to be having no effect. There are a lot of causes for this, not least of all the things you mention (nobody cares about science, people like or disl…

Not at all, that study was not well conducted, it compared people just learning to program with people that already had experience, in a time when programming was much different from what it is today... Just go read the damn paper...

Re: Study of 49 programmers: static type system had no effect on development time

#132

Post by seunosewa » Fri, Jun 08, 2012, 1:45 PM UTC

kaffeinecoma wrote: post #129

I wonder whether they are actually measuring the right thing here. I think the real value to static typing is not so much the initial development of small (tiny, in this case) programs, but rather large systems that have long lifetimes and maintenance periods involving multiple developers. I just started diving into Ruby. I'm about a month into it, after having spent over a decade doing Java and other static language…

Depends on the IDE. I use Python with Eclipse. Auto-completion works quite perfectly. Refactoring too.

Re: Study of 49 programmers: static type system had no effect on development time

#133

Post by silverlake » Fri, Jun 08, 2012, 1:55 PM UTC

fusiongyro wrote: post #32

The study may or may not be flawed, but what's really interesting to me is the reaction. We need more science in our computer science, which means more experiments and more results like this. We should also be open to the truth that we use the tools we like because we like them rather than because they're technically superior, even though we pimp them ad nauseum as though they are. I once read an article about a tech…

No study can control all the variables enough to convince a fanboy that his favorite language isn't the greatest ever. However, the current state of PL research is as close to astrology as you can get. Why are we working on type systems and modules and concurrency primitives et. al. without a scrap of evidence that any of it contributes to programmer productivity? There's no science there.

In fact, everyone here should flip this around and ask: can you design a practical experiment to compare productivity of dynamic vs. static languages? Will others find it convincing? Probably not. PL advocates are no different than religious missionaries. They have no objective proof of any of their claims. And both will murder the natives if they don't convert.

edit: Fowler's view here http://martinfowler.com/bliki/CannotMeasureProductivity.html

Re: Study of 49 programmers: static type system had no effect on development time

#134

Post by barik » Fri, Jun 08, 2012, 2:04 PM UTC

dalke wrote: post #89

barik wrote: post #72

Earlier quoted context omitted.

"In fact, one of the few studies that people actually believe is the "some programmers are 10x better" study, and that was actually fairly well conducted - many students were given identical tasks, and a fairly large amount of time to do them." Can you by chance point me to this paper? I'd like to add it to my paper collection, since most of the studies I've seen concerning programmer variability use members of the w…

http://forums.construx.com/blogs/stevemcc/archive/2011/01/09...

I'm not sure how that link helps me. I've already seen many of those studies. Which one satisfies, "students were given identical tasks, and a fairly large amount of time to do them"?

That is the specific study I am searching for to add to my list of papers. Did you give me this link because you were referring to Humphrey (A Discipline for Software Engineering), or something else? I can track down Humphrey, but it will take me a few days, since it's a physical book.

Re: Study of 49 programmers: static type system had no effect on development time

#135

Post by Peaker » Fri, Jun 08, 2012, 2:05 PM UTC

kaffeinecoma wrote: post #129

I wonder whether they are actually measuring the right thing here. I think the real value to static typing is not so much the initial development of small (tiny, in this case) programs, but rather large systems that have long lifetimes and maintenance periods involving multiple developers. I just started diving into Ruby. I'm about a month into it, after having spent over a decade doing Java and other static language…

> The productivity boost for new development in Ruby is very real; I converted a 10K LoC java program to It's important not to conflate the benefits of Ruby here with benefits of dynamic typing.

Haskell is statically typed but can be as concise as Ruby (sometimes more concise, sometimes less).

Re: Study of 49 programmers: static type system had no effect on development time

#136

Post by takluyver » Fri, Jun 08, 2012, 2:11 PM UTC

Let's summarise what the study actually found. The researchers go in with the expectation that static languages will have advantages in their study. They designed a brand new language - 'Purity' - in both dynamically and statically typed variants, so there was no difference in documentation or other language features. The only difference is that one forces you to declare variable types, and checks them at compile time rather than at run time.

They take 49 students and train them in the language - 16 hours for the dynamic version, 18 for the static, because of "the additional effort for teaching the static type system". The students have 27 hours to complete two related tasks. All activity was logged, so they could look at the time from a test run failing with type errors to a successful test run.

- Scanner task: overall, students using the dynamically typed language were significantly quicker to complete a scanner that passed the tests - on average, about a third faster. Within that, the time spent debugging type errors doesn't differ between the groups.

- Parser task (builds on the scanner): The implementations are tested after coding has finished, showing no significant difference in success between the languages. In both groups, about half the students didn't manage a meaningful parser at all. In this part, students with dynamic typing spent significantly less time debugging type errors.

Re: Study of 49 programmers: static type system had no effect on development time

#137

Post by dbecker » Fri, Jun 08, 2012, 2:18 PM UTC

Paradigma11 wrote: post #52

takeoutweight wrote: post #46

Absence of evidence is not evidence of absence.

But it is, at least from a bayesian perspective.

Not at all.

Even from a Bayesian perspective, having a small sample that is too small to reach strong conclusions does not shift your posterior towards 0.

Re: Study of 49 programmers: static type system had no effect on development time

#138

Post by mjallday » Fri, Jun 08, 2012, 2:27 PM UTC

seunosewa wrote: post #132

kaffeinecoma wrote: post #129

I wonder whether they are actually measuring the right thing here. I think the real value to static typing is not so much the initial development of small (tiny, in this case) programs, but rather large systems that have long lifetimes and maintenance periods involving multiple developers. I just started diving into Ruby. I'm about a month into it, after having spent over a decade doing Java and other static language…

Depends on the IDE. I use Python with Eclipse. Auto-completion works quite perfectly. Refactoring too.

I disagree. There's simply no way that the IDE can know everything about a variable without running the code and I don't believe that eclipse does this.

It's ok at refactoring but it breaks down all the time for me, renaming unrelated variables or missing cases. I don't believe it can handle kwargs for example (no computer in front of me, feel free to prove me wrong)

Re: Study of 49 programmers: static type system had no effect on development time

#139

Post by dbecker » Fri, Jun 08, 2012, 2:28 PM UTC

The author appears to misunderstand statistical significance when he makes claims such as "the existence of a static type system has neither a positive nor a negative impact on ... development time."

The mean development time for dynamically typed languages is about 30% shorter in his first experiment, and it is about 20% shorter in the second experiment.

Statistical significance test determines how likely this difference would be if the difference development time in a large population was 0... It doesn't mean that the true effect actually is 0... it's just addressing whether we can completely rule that out.

A more accurate summary of his results would be "Dynamic type systems allowed 20% to 30% faster development in my experiments, but I had too few participants to reject equality."

Re: Study of 49 programmers: static type system had no effect on development time

#140

Post by papsosouid » Fri, Jun 08, 2012, 2:29 PM UTC

This study starts with a bizarre premise. I've never heard anyone, ever suggest that statically typed languages make initially writing a program faster. Very much the opposite, java apologists claim that their slower development time is offset by easier maintenance and less bugs, explicitly acknowledging that initial development is slower in a java-esque static language.

Of course, the static typing proponents you find on HN are generally not advocating incredibly limited type systems like java or the one used in this "study". I find it hard to believe that a language with such a limited type system is representative of languages with useful type systems like ocaml or haskell.