I'm torn about this. On the one hand, while I've known my framework of choice (Rails) is slow, I didn't know how much slower it could be in the grand scheme of things. But on the other hand, I'm more shocked by the difference between EC2 and dedicated hardware (10x improvement with rails), and even 89 requests per second (20 query benchmark on EC2) is still a decent amount of traffic. (Plus this doesn't count any opt…
EC2 was a hard platform to test on, only because our i7 hardware would give us results fairly instantaneously, but we became impatient when we had to wait upwards of 10x as long for the data on the EC2 large instances. We're actually very interested in how the large/newer instances perform.
Web Framework Benchmarks
271–280 of 415 posts
Re: Web Framework Benchmarks
#272I still believe launching a usable web application next week is preferable to launching a really fast web application 3 months from now, if ever.
Re: Web Framework Benchmarks
#273As a Rails developer and admirer, this is eye-opening. I love the framework (and Ruby especially), but these numbers bear some serious consideration. 30-50x performance difference gets really... real, no? The standard refrain of "throw more hardware at it" must reconcile with the fact that a factor of 30-50x means real dollars for the same amount of load. Is the developer productivity really that much greater?
I fully respect the JVM family of languages as well. I just think that Mark Twain said it best when he said: "There are three kinds of lies: lies, damned lies, and statistics." It's not that the numbers aren't true, it's that they may not matter as much, and in the way, that we initially perceive them.
Performance is certainly something you should consider when selecting a language/framework, but it is not the only thing.
========================
You should undertake a detailed examination of these statistics before making any decisions.
Issue #1) The 30-50x performance difference only exists in a very limited scenario that you're unlikely to encounter in the real world.
Look carefully at the tests performed. The first test is an extraordinarily simple operation: take this string, serialize it, and send it to the client. This is the test in which we see massive differences:
Gemini vs Rails
25,264/687 (gemini/rails-ruby) = 36.774
25,264/871 (gemini/rails-jruby) = 29.000
Node.js vs Rails
10,541/687 (nodejs/rails-ruby) = 15.343
10,541/871 (nodejs/rails-jruby) = 12.102
That's a 37x performance win for Gemini, and 15x for Node.js.
Side note: You might be wondering why I didn't compare to the top performer, Netty. Netty is more like Rack. You build frameworks on top of Netty, not with Netty. As a Ruby dev, you could think of this in the same context of comparing Ruby on Rails with Rack; not a good comparison. Hence We won't compare Rails to Netty.
The error would be in extrapolating that a move to Gemeni or Node.js would give you a 37x or 15x performance increase in your application. To understand why this is an error, we jump down to the "Database access test (multiple queries)" benchmark.
Issue #2) Performance differences for one task doesn't always correlate proportionally with performance differences for all tasks.
In the multi-query database access test, we start to see the top JSON performers slow down significantly when compared to the slow down for Rails:
Gemini vs Rails
663/89 (gemini/rails-ruby) = 7.449
663/108 (gemini/rails-jruby) = 6.138
Node.js vs Rails
116/108 (nodejs-mysql-raw/rails-ruby) = 1.077
60/108 (nodejs-mysql/rails-ruby) = 0.555
In this scenario -- which is arguably much closer to the real world -- Ruby on Rails closes the gap and even beats some of the hip new kids.
But why? The in-depth answer to this question would require a lot of space, but the really, really short version is kind of a "what's the sound of one hand clapping" response: Ruby isn't actually all that slow.
To understand what the hell that means, check out this presentation from Alex Gaynor (of rdio/Topaz fame):
https://speakerdeck.com/alex/why-python-ruby-and-javascript-...
Ruby is just about as fast as C, provided you're comparing it to C that does exactly the same operations on the hardware as the Ruby code. Don't get me wrong, that's a HUGE provision. But it warrants close examination.
The real benefit of lower level languages like C is that they give you the flexibility to drill down in to your actual bare-metal operations and optimize the way the program executes on the hardware. As Alex points out, we don't currently have that level of flexibility in languages like Ruby (without dropping down to inline C), so we suffer a performance penalty.
This penalty is huge for simple tasks because they involve only a handful of operations that execute extremely quickly. As you add complexity, however, the benefits of micro-optimizations get lost in the vastness of the overall execution time.
Look at it like this. When Gemini hits 36,717 req/s in the JSON test, each request only lasts about 1.6 ms. This is only possible because of the simplicity of the operations being done on the hardware. Ruby loses big here because there is a lower boundary to the way you can optimize without dropping down to C.
gemini: 1.6 ms per request
rails-ruby: 87.3 ms per request
When we look at the multi-query database access test, we can see how the optimization at the low level gets lost in the sea of time taken to process the request.
gemini: 90.5 ms per request
rails-ruby: 674.2 ms per request
Granted, that is still over a 7x performance win for Gemini, but this is where the Ruby arguments about programmer efficiency come in to play. I don't know Gemini, so it may very well beat Rails in that comparison too. Ruby is getting more performant with every release though, so it's easier to justify on the basis of preference alone when we're this close.
Re: Web Framework Benchmarks
#274Utter crap. "Sadly Django provides no connection pooling and in fact closes and re-opens a connection for every request. All the other tests use pooling." But it's free, open-source software and we provide asynchronous database connection pooling for Postgres SQL: https://github.com/iiilx/django-psycopg2-pool
Obviously if they optimized each and every one of these benchmarks we would see different results, but it would take a massive amount of time to learn the ins and outs of each framework to the point where you can do so effectively. For one person do a benchmark over this many samples, they have to just go with the out of the box setup for each.
As our blog post suggests, where we are not experts we had to rely on the tutorials provided by each framework's authors to build a test setup. If a specific framework seems low on the list, it could be due to the fact that the best practices guides we found for getting set up were not correctly configured for production use.
Draw what conclusions you would like from this statement, but we did aim to be as fair and unbiased as possible.
Re: Web Framework Benchmarks
#275As a Rails developer and admirer, this is eye-opening. I love the framework (and Ruby especially), but these numbers bear some serious consideration. 30-50x performance difference gets really... real, no? The standard refrain of "throw more hardware at it" must reconcile with the fact that a factor of 30-50x means real dollars for the same amount of load. Is the developer productivity really that much greater?
Preface: This post is going to come across as a Rails apologist piece, but please read the entire thing before you reach a conclusion. Please also consider that you could apply these same arguments to just about any of the high-level language based frameworks on the list. I use Ruby on Rails in my comparisons, but I'm a huge fan of Node.js, Python/Django, and Go. I fully respect the JVM family of languages as well. I…
Re: Web Framework Benchmarks
#276Earlier quoted context omitted.
Why not both? Insisting on a very stable API can result in long times between releases, which can mean more people using tip. That's distinct from how stable tip is.
Given the frequent complaints that the previous stable release isn't very stable, I think trying to interpret it as "tip is super stable" is wishful thinking. Tip is currently less bad than stable. The fact that stable releases are not stable is a bad thing, not a good thing.
If stable means suitable for production, Go tips vastly improved performance, especially in regards to garbage collection, make it more suitable than 1.0.3 for large/high-scale applications in production.
Re: Web Framework Benchmarks
#277"Among the many factors to consider when choosing a web development framework, raw performance is easy to objectively measure." Oh really? Then why did Zed write such an angry rant about how you are doing it wrong? http://zedshaw.com/essays/programmer_stats.html Can we please see some standard deviations, at least?
What's wrong with that guy?
Really, I'm not making this up. He sounds like a jerk to work with.
Re: Web Framework Benchmarks
#278Earlier quoted context omitted.
Since these tests seem to be all about JSON serialization, it would've been interesting to see the tests with rails-api instead of the standard Rails stack: https://github.com/rails-api/rails-api What webserver were you using on JRuby? Was it Trinidad? Did you try Jetpack?
I concur, and Rails 4 may not be officially released yet but it's stable enough to run these tests against.
Re: Web Framework Benchmarks
#279Earlier quoted context omitted.
There are a few problems with your Play code that are causing it to be unnecessarily slow. First-- what you're really testing here is the Jackson library. A majority of the cycles used in your application are being burned in that toJson call of an array of objects. This isn't a fair test compared to the servlet implementation because you're calling Jackson against a map in the Play example, versus against a simple St…
Do you think such a configuration could outrun the Vert.x configuration they've posted? I'm not challenging you, I'm just genuinely curious! Because if Play+Akka can outrun Vert.x, then it would be an interesting game altogether...
Re: Web Framework Benchmarks
#280It would be very interesting to see OpenResty (Nginx + Lua) in there, since it's so different from other approaches.