The Great Ruby Shootout (July 2010) • Hacker News Board

Re: The Great Ruby Shootout (July 2010)

#41

Post by headius » Tue, Jul 20, 2010, 8:00 AM UTC

xpaulbettsx wrote: post #37

acangiano wrote: post #21

Earlier quoted context omitted.

> The Linux versus Windows numbers are a bit surprising to me. If I were to make a guess, I'd guess that the JVMs used were not identical (perhaps like Isaac Gouy mentions, one platform was 64-bit and the other wasn't) or some other detail altered the performance characteristics of the test. The same identical version was used: Java HotSpot(TM) 64-Bit Server VM 1.6.0_20. > But the performance drop does seem to be in…

I don't know that this theory totally holds water (Windows OS developer here, announcing obvious bias) - looking at the results, the one place Windows really gets nailed is the I/O test; I suspect there's some significant optimizations that could be done there, as I/O on Windows OS in general certainly isn't 4x slower than Linux.

If there's something we or the JVM could do to improve these numbers, I would love to talk to you about it. At this point, if the JVM developers haven't found the magic sauce, we JRuby guys probably won't either...but I'd really love for JRuby performance on Windows to match JRuby performance on Linux.

Re: The Great Ruby Shootout (July 2010)

#42

Post by headius » Tue, Jul 20, 2010, 8:00 AM UTC

donw wrote: post #36

headius wrote: post #19

I can comment on the JRuby results. For the base perf numbers, I'm not surprised. We've known that we're roughly on par with 1.9.2 for some time, and many of the benchmarks in question have started to reach a point of irreducible complexity (e.g. you can only slice and dice strings so fast). It's good to see JRuby remains at or near the front of the pack as far as performance, especially considering we've made no maj…

Tooting my own metaphorical horn a bit, but it was relatively easy to put together a Jetty-Rack interface for high-performance webapps. It's called Mizuno, and lives at http://github.com/matadon/mizuno Internally it uses Jetty's event-driven I/O, so performance is on a par with Thin and Passenger, at least on my MacBook. I'm a bit pressed for time at the moment, so if anybody wants to add in the cometd servlet (it's…

Sounds pretty cool :) I'd love to see some blog posts about it, but I doubt I'll have time to do so in the next couple weeks.

Re: The Great Ruby Shootout (July 2010)

#43

Post by xpaulbettsx » Tue, Jul 20, 2010, 7:05 PM UTC

headius wrote: post #41

xpaulbettsx wrote: post #37

Earlier quoted context omitted.

I don't know that this theory totally holds water (Windows OS developer here, announcing obvious bias) - looking at the results, the one place Windows really gets nailed is the I/O test; I suspect there's some significant optimizations that could be done there, as I/O on Windows OS in general certainly isn't 4x slower than Linux.

If there's something we or the JVM could do to improve these numbers, I would love to talk to you about it. At this point, if the JVM developers haven't found the magic sauce, we JRuby guys probably won't either...but I'd really love for JRuby performance on Windows to match JRuby performance on Linux.

I don't think that I can actively hack on the JVM, but I can tell you that XPerf is a great way to determine where you're spending your time. http://www.microsoftpdc.com/2009/CL16 is a really good intro video on the topic, it's a very powerful tool (though doesn't provide as much analysis as Instruments for example)

Re: The Great Ruby Shootout (July 2010)

#44

Post by leif » Wed, Jul 21, 2010, 3:25 PM UTC

acangiano wrote: post #35

leif wrote: post #34

Means and (especially) medians mean little to nothing when you're doing them over entirely different benchmarks. Why bother? A 3D plot would be a lot clearer.

> Means and (especially) medians mean little to nothing It's the other way around. Given the skewed nature of the data, the median is a better measure of the central tendency of the data set. But note that I'm not plotting the median only. I use a box plot which gives you a much better statistical picture than using a simple median (or mean). Nevertheless, summarizing different benchmarks has its limitations and box…

I disagree. If you write ten benchmarks, and they finish in the same order on all platforms, the median really doesn't give you any information apart from how the middle one did. At least the mean will be changed if the slowest benchmark for some reason is a lot slower on one platform. In this case, there was some variation, but for the most part it looks like the median is just one of 4 or 5 middling benchmarks in all cases, so you're really just removing data, if anything.

The box plot is a good choice here. What I would like to see is a graph plotting time against {VM} x {benchmark size} to see how each platform scales within a benchmark.

With the data you've got, apart from confusing me a little by including the mean and median, you did a good job presenting it. I'd love to see why some platforms do so much well than others in some cases (except JRuby which everyone knows is a memory hog), but this seems way out of scope for this article. :)

Re: The Great Ruby Shootout (July 2010)

#45

Post by messel » Sun, Jul 25, 2010, 5:24 PM UTC

headius wrote: post #27

messel wrote: post #24

The curve I dream of seeing: c++ slower than any good coverage flavor of Ruby.

I'd like to see that too. In JRuby, we may be able to get Ruby-to-Ruby calls to perform as well as Java calls, which would at least get that bottleneck out of the way. The remaining performance issues, however, are usually the rate at which objects can be allocated. In order to reduce that we may need a little JVM help (escape analysis that works well enough to actually eliminate allocations) and we may start to expl…

Charles, is there any chance virtual machines, JIT compilers can get "smart enough" to introduce native typing without requiring a hard decision by a developer. I'd prefer not to have to worry about types ever, and let the interpreter/compiler/optimizer slap them onto objects as needed to really crank numeric throughput.

Maybe I could tune software with profiling tools after the fact. But it feels like as soon as you start locking specific objects down it's a slippery slope.

I'm not very familiar with the depth you've gone to, but can a dynamic object have numeric features or be capable of substituting numeric handling for a limited time (virtual numerics) and then revert back to a sloppy untyped object?