Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Ruby Fast Yet?

isrubyfastyet.com

Post reply on HN
Print view View original

21–30 of 95 posts

Re: Is Ruby Fast Yet?

#21

Post by lukeschlather »

jarrett wrote: post #10
RyanZAG wrote: post #3

Not sure how comparing different versions of rails shows if its 'fast' yet. I'd think setting up some request target and seeing how far RoR is from that target with each version would be better. Or comparing RoR to other platforms is also a good bet, eg http://www.techempower.com/benchmarks/ I just don't really see how this answers the question other than 'ehh I think it looks a bit faster than it was awhile ago?' ED…

Amazon says their "m1.small" instance has 1.7 GiB of memory. That's a lot less than most current laptops. Maybe JRuby uses more memory in total, but can handle more concurrent requests per MB of memory used? That could explain the discrepancy if it's true, though it's just speculation on my part. Another theory is that the benchmarks are too application-specific. Rails is a complex framework with which you can build…

The thing is, those are all I/O bound operations and in the case of the databases most of the code you're benchmarking is native extensions, so I would expect the difference to be negligible.

Testing whole Rails apps really does seem like the best way to get good data. The interesting speedups are likely going to be in places like string matching / string copying and other places that don't involve network access.

Re: Is Ruby Fast Yet?

#23

Post by Freaky »

Jgrubb wrote: post #8

JRuby question -- Why is it that much slower? I'd have thought that part of the point was to take advantage of the JVM's built in optimization.

A lot of optimizations are off by default, or completely unavailable because he's using JDK 1.6. See https://github.com/jruby/jruby/wiki/PerformanceTuning

Startup time is slow because it needs to warm up the JIT. Same with Rubinius, which is mysteriously missing from most of these graphs.

Memory consumption partially depends on how much you tell the JVM to use - it'll run the garbage collector more or less aggressively to keep within that bound - default appears to be min(system memory/4, 1GB).

That's not to say it doesn't need more memory than MRI, but you should factor in the concurrent native threads allowing one process to serve many clients, and the precise compacting GC limiting memory fragmentation and associated heap bloat with long-running processes.

Re: Is Ruby Fast Yet?

#24

Post by mml »

Jgrubb wrote: post #8

JRuby question -- Why is it that much slower? I'd have thought that part of the point was to take advantage of the JVM's built in optimization.

My question as well. Every perf benchmark I've seen of jruby over the years, it's much faster than MRI (given time to boot & JIT anyway).

I think this guy's numbers are very suspect.

Re: Is Ruby Fast Yet?

#25

Post by sluukkonen »

Jgrubb wrote: post #8

JRuby question -- Why is it that much slower? I'd have thought that part of the point was to take advantage of the JVM's built in optimization.

Looks like it's running with the threadsafe mode off. That would probably give JRuby an edge, as it's able to take advantage of greater thread parallelism than MRI. No idea if he's letting the JVM warm up enough either.

Re: Is Ruby Fast Yet?

#27

Post by jrochkind1 »

sluukkonen wrote: post #25
Jgrubb wrote: post #8

JRuby question -- Why is it that much slower? I'd have thought that part of the point was to take advantage of the JVM's built in optimization.

Looks like it's running with the threadsafe mode off. That would probably give JRuby an edge, as it's able to take advantage of greater thread parallelism than MRI. No idea if he's letting the JVM warm up enough either.

Depending on how you are deploying the ruby app.

As far as I know, Passenger Enterprise and Puma are the only multi-platform ruby rack app servers that support multi-threaded concurrent request dispatch. There may be other jruby/java-only deployment scenarios that support concurrent request dispatch.

Without concurrent request dispatch, jruby's hypothetical greater thread parallelism is unlikely to effect requests-per-second benchmarks. Also, the benchmarking test would have to be willing to issue requests concurrently!

Re: Is Ruby Fast Yet?

#28

Post by jrochkind1 »

mikesilvis wrote: post #12
matthuggins wrote: post #4

Wouldn't it make more sense to test various features of Ruby for comparison rather than comparing an entire web framework? Given that, the URL should be israilsfastyet.com, or more appropriately howdoesrailsbehaveonvariousrubyversions.com

Agreed. This is much more of a test of the rails framework then ruby

Well, it's a comparison of different ruby versions, using Rails. That's pretty useful to people who use Rails, which is a lot of people.

Of course, it's not a real Rails app presumably, but just a sample/dummy one. So you may or may not see comparable speed-up with a real app. But it's probably suggestive of the upper limit of improvement you can get by switching ruby versions.

"Will I get a performance improvement from my Rails app by upgrading ruby" is a real question a lot of people will have. Micro-benchmarks of ruby wouldn't answer it as well as the OP does.

Re: Is Ruby Fast Yet?

#30

Post by jrochkind1 »

jarito wrote: post #19
timmorgan wrote: post #11

I'm surprised at how negative the comments are here. Graphing how different Ruby versions/implementations perform running Rails is very helpful! Glad to see that MRI continues to get faster with every version. Merry Christmas!

The negativity comes from the fact that this benchmark is useless and, at worst, misleading. True benchmarks are hard to get right. Running a bunch of stuff on a laptop isn't sufficient evidence of anything. Kudos to making the benchmark opensource at least, but even from a quick peak, I wouldn't trust this at all. As others have mentioned, JRuby shouldn't be this slow. Odds are the benchmark is throwing away the JIT…

I'm not sure I have ever seen a benchmark on HN that wasn't called misleading and useless in the comments.

But I agree it is hard to get benchmarks right -- I'm not even sure there is a 'right', as a benchmark will usually be based on certain assumptions that only sometimes apply. (Unless you are benchmarking your own actual production code, in which case you know it's assumptions are valid -- for your own code).

So I don't think that's a reason not to share benchmarks. But I _do_ think it's a reason to be really transparent about exactly how the benchmarks were done. Ideally all code should be available on github.

Then people can examine your assumptions (including ones you knew you were making and ones you didn't) and modelling, as well as fork or pull request to do benchmarks in some other way to better model the scenarios they care about.

Although also in my experience, you get LOTS of people telling you your benchmark is wrong because of X, Y or Z -- sometimes they are correct that it doesn't accurately capture something they are interested in, sometimes they are just confused -- but virtually nobody actually interested in writing code via fork or pull request to capture what they are interested in. Once you put out a benchmark, everyone in the peanut gallery thinks they are entitled to keep demanding you change it in various ways for them.)

No matter how much time you spend on it and thought you put into it, publishing a benchmark on HN is probably the easiest way to have lots of people telling you you're an idiot who did things wrong. It almost seems like HN collectively wants to discourage people from running benchmarks -- or from sharing what they found publicly.

But it's because doing good benchmarking is so hard -- well, that's exactly we need more people working on it, and sharing how they did it so others can improve upon it, etc, right?

Post reply on HN