A comparison of programming languages in economics • Hacker News Board

Re: A comparison of programming languages in economics

#41

Post by Alupis » Thu, Jul 03, 2014, 1:13 AM UTC

k2enemy wrote: post #9

bradfordarner wrote: post #6

Earlier quoted context omitted.

It is unfortunate that the authors of this paper didn't reach out to experts in each programming language in order to write high-quality code against which they could have run the analysis. This seems like the perfect use-case for open-source: academic study, subject-matter experts in one area who are stepping outside their areas of expertise and an interesting subject matter. They should have crowd-sourced the code…

From the paper... Second, to make the comparison as unbiased as possible, we coded the same algorithm in each language without adapting it to the peculiarities of each language (which could reflect more about our knowledge of each language than of its objective virtues). They have invited others to fork the code on github and write the fastest possible for each language.

^^ This is not "unbiased"... in their attempt to make it unbiased, they introduced a huge amount of bias.

They wrote the code for each language based on prior knowledge about each language, and/or from likely googling how-to's... far from performant code with any language.

A mild understanding of each language could yield dramatic performance improvements over not understanding it at all.

Re: A comparison of programming languages in economics

#42

Post by Alupis » Thu, Jul 03, 2014, 1:14 AM UTC

georgewfraser wrote: post #39

There are two glaring problems with this paper: 1. The algorithm is just a bunch of arrays and loops in one giant block. This is the easiest possible case for compilers that optimize through specialization. Even a bad specializing compiler will be good at this but will blow up when faced with a big piece of code with lots of functions and data structures. 2. The runtimes are only ~2s, which is not long enough for the…

> EDIT: I stand corrected. I downloaded his examples off Github and ran them the same way as the paper. I get 1.87s for C, and for Java... also exactly 1.87s.

Your code probably got JIT'ed... which essentially makes your java code into native C... So... it's no surprise they perform around or exactly the same (given the JVM is allowed time to warm up, discover "hot spots", and then perform the JITing)

Re: A comparison of programming languages in economics

#43

Post by georgewfraser » Thu, Jul 03, 2014, 1:18 AM UTC

Alupis wrote: post #42

georgewfraser wrote: post #39

There are two glaring problems with this paper: 1. The algorithm is just a bunch of arrays and loops in one giant block. This is the easiest possible case for compilers that optimize through specialization. Even a bad specializing compiler will be good at this but will blow up when faced with a big piece of code with lots of functions and data structures. 2. The runtimes are only ~2s, which is not long enough for the…

> EDIT: I stand corrected. I downloaded his examples off Github and ran them the same way as the paper. I get 1.87s for C, and for Java... also exactly 1.87s. Your code probably got JIT'ed... which essentially makes your java code into native C... So... it's no surprise they perform around or exactly the same (given the JVM is allowed time to warm up, discover "hot spots", and then perform the JITing)

True though oddly I compiled and ran them exactly the same way he described in the paper -- one time through, no warm-up time.

Re: A comparison of programming languages in economics

#44

Post by Terr_ » Thu, Jul 03, 2014, 1:22 AM UTC

georgewfraser wrote: post #43

Alupis wrote: post #42

Earlier quoted context omitted.

> EDIT: I stand corrected. I downloaded his examples off Github and ran them the same way as the paper. I get 1.87s for C, and for Java... also exactly 1.87s. Your code probably got JIT'ed... which essentially makes your java code into native C... So... it's no surprise they perform around or exactly the same (given the JVM is allowed time to warm up, discover "hot spots", and then perform the JITing)

True though oddly I compiled and ran them exactly the same way he described in the paper -- one time through, no warm-up time.

I think the command-line option -XX:+PrintCompilation should show you if/when the JIT kicks in for a method.

Re: A comparison of programming languages in economics

#45

Post by Alupis » Thu, Jul 03, 2014, 1:25 AM UTC

thinkpad20 wrote: post #30

MrBuddyCasino wrote: post #13

In case anyone missed the Java results, I do think it is worth mentioning that it is very competitive: "Our fourth result is that Java imposes a speed penalty of 110 to 169 percent. Given the similarity between Java and C++ syntax, there does not seem to be an obvious advantage for choosing Java unless portability across platforms or the wide availability of Java programmers is an important factor."

There are legions of reasons to choose Java over C++, or vice-versa, none of which have anything to do with syntax. Is this a strawman on the part of the authors, or are they unaware of the legion of things that make C++ a far more complex and dangerous language than Java? Or are those things irrelevant to the needs of economists?

It is a very un-scientific study, done by non-programmers with little knowledge about programming in general. To them, syntax may seem like a big deal... just because they are ignorant to the languages other advantages or disadvantages.

As you noted, there are a plethora of reasons to choose either Java or C++...

Re: A comparison of programming languages in economics

#46

Post by Alupis » Thu, Jul 03, 2014, 1:30 AM UTC

georgewfraser wrote: post #43

Alupis wrote: post #42

Earlier quoted context omitted.

> EDIT: I stand corrected. I downloaded his examples off Github and ran them the same way as the paper. I get 1.87s for C, and for Java... also exactly 1.87s. Your code probably got JIT'ed... which essentially makes your java code into native C... So... it's no surprise they perform around or exactly the same (given the JVM is allowed time to warm up, discover "hot spots", and then perform the JITing)

True though oddly I compiled and ran them exactly the same way he described in the paper -- one time through, no warm-up time.

The sample code is probably too predictable and the JVM was able to optimize it well.

In any event... the "myth" that Java is slow is bogus :)

Re: A comparison of programming languages in economics

#47

Post by taeric » Thu, Jul 03, 2014, 2:19 AM UTC

Alupis wrote: post #40

briandh wrote: post #32

Earlier quoted context omitted.

>The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version One of they authors addresses this in the comments on the OP: > As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled…

The "study" does not account for nuances in each language. JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed. The tests did not account for JVM warm-up time, nor allow the code to become "hot" before running their tests. They also executed their test code on poorly performing operating systems not designed to execute high performance code, ie. they ran their…

JIT'ed Java code is lacking a major facility that truly performant native code has, and that is cache locality. That is, it isn't enough that the code to perform the algorithm is "well tuned" if the data structures are not.

(Well, unless I am just wrong on the feats that JIT'ed code can perform nowdays.)

Re: A comparison of programming languages in economics

#48

Post by jeremyjh » Thu, Jul 03, 2014, 3:00 AM UTC

Alupis wrote: post #40

briandh wrote: post #32

Earlier quoted context omitted.

>The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version One of they authors addresses this in the comments on the OP: > As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled…

The "study" does not account for nuances in each language. JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed. The tests did not account for JVM warm-up time, nor allow the code to become "hot" before running their tests. They also executed their test code on poorly performing operating systems not designed to execute high performance code, ie. they ran their…

They also didn't account for the possiblity that a Java programmer would use an ArrayList instead of double[]. You can't JIT that away; JIT does not change the semantics of the Java code so you have to code Java in a sympathetic way. I can imagine this may be more idiomatic in numeric intensive fields though.

Re: A comparison of programming languages in economics

#49

Post by nine_k » Thu, Jul 03, 2014, 3:33 AM UTC

chuto wrote: post #36

bradfordarner wrote: post #6

Earlier quoted context omitted.

It is unfortunate that the authors of this paper didn't reach out to experts in each programming language in order to write high-quality code against which they could have run the analysis. This seems like the perfect use-case for open-source: academic study, subject-matter experts in one area who are stepping outside their areas of expertise and an interesting subject matter. They should have crowd-sourced the code…

It is also unfortunate that several of the language developers and related dev communities have come out against compilers (python and jython in particular). This is the (religious) downside to open source fundamentalism.

PyPy is a counterexample: it's a JIT compiler for Python, pretty efficient.

Compiling dynamic languages is not easy: their semantics of having everything dynamic and changeable at runtime resists it. Python is internally compiled into a pretty compact bytecode, but something like a method call is much more involved business in Python than even in C++, and is many times slower.

WRT fundamentalism: most sane open source communities are pretty pragmatic; Python's in particular.

Re: A comparison of programming languages in economics

#50

Post by maxlybbert » Thu, Jul 03, 2014, 3:53 AM UTC

Alupis wrote: post #40

briandh wrote: post #32

Earlier quoted context omitted.

>The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version One of they authors addresses this in the comments on the OP: > As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled…

The "study" does not account for nuances in each language. JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed. The tests did not account for JVM warm-up time, nor allow the code to become "hot" before running their tests. They also executed their test code on poorly performing operating systems not designed to execute high performance code, ie. they ran their…

The authors do have a comment on the Github project that code needs to be allowed to run for a while so the JIT can kick in, although I didn't see any statement of how they did that. The paper also mentions a decision to turn a loop body into a function to allow a JIT to optimize things.

But, generally, I'm not interested in whether "for" loops run faster in language X or language Y. I am interested in whether some languages encourage code styles that happen to be easily optimized (I'm also interested in whether some languages encourage code styles that are easily maintained, easily extensible, or easily run into a ditch).

This paper is meaningful to somebody, but it's clearly written by people who spend most of their thinking about things other than programming. That's not an insult; it's a simple fact that a lot of programs are written by people who are experts in other fields. I happen to work with a former physicist who has experience writing programs to run on one of the nation's largest supercomputers. The depressing thing (to me) is that while he knows how to write numerical programs that run close to a computer's theoretical maximum speed, that information is only well-known among a small group of people (namely people who have to show that if they're allowed to use a supercomputer, they won't make it run wasteful code). I wish it were shared more broadly.