Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Framework Benchmarks

techempower.com

Post reply on HN
Print view View original

161–170 of 415 posts

Re: Web Framework Benchmarks

#161

Post by kclay »

bhauer wrote: post #92
goodwink wrote: post #77

Earlier quoted context omitted.

Here's a quick version of the framework optimization index. Higher is better (ratio of framework performance to raw platform performance, multiplied by 100 for scale): Framework Framework Index Gemini 87.88 Vert.x 76.29 Express 68.85 Sinatra-Ruby 67.88 WebGo 51.08 Compojure 45.69 Rails-Ruby 31.75 Wicket 29.33 Rails-Jruby 20.09 Play 18.02 Sinatra-Jruby 15.96 Tapestry 13.57 Spring 13.48 Grails 7.11 Cake 1.17

It's funny you should put this together because in an earlier draft of this blog entry I had created a tongue-in-cheek unit to express the average cost per additional line of code. Based on our Cake PHP numbers, I wanted to describe PHP as having the highest average cost per line of code. But we dropped this because I felt it ultimately wasn't fair to say that based on the limited data we had and it could be easily i…

I'm pretty shocked that Play scored so low. One would think that being built on netty would put Play in a higher rank. Database access for sure needs to be in async block

Re: Web Framework Benchmarks

#162

Post by [deleted] »

bhauer wrote: post #92
goodwink wrote: post #77

Earlier quoted context omitted.

Here's a quick version of the framework optimization index. Higher is better (ratio of framework performance to raw platform performance, multiplied by 100 for scale): Framework Framework Index Gemini 87.88 Vert.x 76.29 Express 68.85 Sinatra-Ruby 67.88 WebGo 51.08 Compojure 45.69 Rails-Ruby 31.75 Wicket 29.33 Rails-Jruby 20.09 Play 18.02 Sinatra-Jruby 15.96 Tapestry 13.57 Spring 13.48 Grails 7.11 Cake 1.17

It's funny you should put this together because in an earlier draft of this blog entry I had created a tongue-in-cheek unit to express the average cost per additional line of code. Based on our Cake PHP numbers, I wanted to describe PHP as having the highest average cost per line of code. But we dropped this because I felt it ultimately wasn't fair to say that based on the limited data we had and it could be easily i…

[deleted]

Re: Web Framework Benchmarks

#163

Post by aliston »

xal wrote: post #135

What kind of server did you guys use for your rails test? Thin, Puma, Unicorn? Are you sure you ran it in production environment? Update: Looks like passenger in development mode. Good job you benchmarked a web server that no one uses wile reloading all code between requests. Update2: Ok it seems to run in production mode but still, passenger is not an idiomatic choice.

If that's true, it's not really a fair comparison. Running in development mode slows things considerably. They do have a question in the FAQ about that point:

"You configured framework x incorrectly, and that explains the numbers you're seeing." Whoops! Please let us know how we can fix it, or submit a Github pull request, so we can get it right."

Perhaps you/we should submit a pull request?

Re: Web Framework Benchmarks

#164

Post by andybak »

knappador wrote: post #112

Utter crap. "Sadly Django provides no connection pooling and in fact closes and re-opens a connection for every request. All the other tests use pooling." But it's free, open-source software and we provide asynchronous database connection pooling for Postgres SQL: https://github.com/iiilx/django-psycopg2-pool

Your comment is valid and would have been vastly improved without the first two words.

Re: Web Framework Benchmarks

#166

Post by awj »

daenz wrote: post #142
xal wrote: post #135

What kind of server did you guys use for your rails test? Thin, Puma, Unicorn? Are you sure you ran it in production environment? Update: Looks like passenger in development mode. Good job you benchmarked a web server that no one uses wile reloading all code between requests. Update2: Ok it seems to run in production mode but still, passenger is not an idiomatic choice.

I'm not familiar with Rails. Could you explain what you mean in layman terms? Why is code being reloaded on every request by default?

In development mode Rails will reload code on each request to pick up on changes you have made to your application. That way you can interact with the application to help verify that your code is working properly.

As of Rails 3.2 development mode watches for file changes and attempts to only reload those files, but it's still a significant performance issue.

By default all Rails applications start in development mode, so one gotcha of benchmarking Rails is that some people will forget to set the mode correctly. That said, from the setup code[1] (line 14) it looks like they were running passenger in production mode. The max pool size seems excessive, especially when running on large ec2 instances, but I'm not fully convinced that it's out of line.

[1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

Re: Web Framework Benchmarks

#167

Post by pfalls »

xal wrote: post #135

What kind of server did you guys use for your rails test? Thin, Puma, Unicorn? Are you sure you ran it in production environment? Update: Looks like passenger in development mode. Good job you benchmarked a web server that no one uses wile reloading all code between requests. Update2: Ok it seems to run in production mode but still, passenger is not an idiomatic choice.

We used Phusion Passenger, although we have plans to add additional servers (such as Unicorn). We tried to spend time with various server choices for all platforms, and for ruby, in our short test, Passenger won out against the others.

Our understanding is that when running Passenger, simply passing '-e production' to the command line is sufficient to run in production, but if that's incorrect, we'll gladly update the test.

Re: Web Framework Benchmarks

#168

Post by kainsavage »

pekk wrote: post #150
goodwink wrote: post #99

Earlier quoted context omitted.

Could you elaborate? Their website seems to indicate that they are a very polyglot shop, not someone pushing a JVM agenda: "On the back-end, we use Java, Ruby, Python, .NET, PHP and others based on what makes sense balancing server performance, scalability, hosting costs, development efficiency, and your internal development team's capabilities."

"We have included our in-house Java web framework, Gemini, in our tests. ... " Then you see results for some languages that are completely out of whack with most other such benchmarks (they themselves mention the weirdness of Sinatra vs. Rails, for example). Then you see on a couple platforms that more performant mainstream options have been excluded, for no good reason. Then if you look at the repo, there are deploy…

Feel free to issue a pull request to help our testing. We are not trying to push Java-based frameworks over any others and we believe we are being fair across the board. That being said, if there are "code mistakes... which go well beyond elementary incompetence", then we would love to correct and retest these.

Re: Web Framework Benchmarks

#169

Post by awj »

xal wrote: post #135

What kind of server did you guys use for your rails test? Thin, Puma, Unicorn? Are you sure you ran it in production environment? Update: Looks like passenger in development mode. Good job you benchmarked a web server that no one uses wile reloading all code between requests. Update2: Ok it seems to run in production mode but still, passenger is not an idiomatic choice.

This (from [1]) looks like passenger in production mode to me.

> rvm ruby-2.0.0-p0 do bundle exec passenger start -p 8080 -d -e production --pid-file=$HOME/FrameworkBenchmarks/rails/rails.pid --nginx-version=1.2.7 --max-pool-size=24

https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

Re: Web Framework Benchmarks

#170

Post by bhauer »

jfarmer wrote: post #132

What would be much more interesting than "peak responses per second," which is a weird metric to begin with, is the actual histogram of sampled throughputs. Or at the very least a box-and-whisker plot ( http://en.wikipedia.org/wiki/Box_plot ). Most folks who have run Rails at scale, for example, find that the untuned garbage collector in MRI (Ruby's default interpreter) introduces a large amount of variance, for exam…

We don't have a box plot, but we have line charts for all of the data (performance at multiple client concurrency levels). Just click the "All samples (line chart)" tab on the data panels.

(Note that the very first data panel in the intro is an image and doesn't have tabs.)

Post reply on HN