Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Framework Benchmarks Round 2

techempower.com

Post reply on HN
Print view View original

11–20 of 241 posts

Re: Web Framework Benchmarks Round 2

#11

Post by e12e »

bhauer wrote: post #2

This is our first follow up to last week's web framework benchmarks. Since last week, we have received dozens of comments, thoughts, questions, criticisms, and most importantly pull requests. This post shows data collected from a second run on EC2 and i7 hardware that started on Tuesday of this week. A third round with even more community contribution is already underway. Thanks especially to those who have contribut…

It's great that you're doing this, and listing stuff like standard deviation in the tables -- but I'd say your focus/interpretation of the data isn't quite right. At least provide the option to sort by standard deviation -- as that might well be more interesting than requests/second?

Maybe I'm just being mean because I was reminded of this essay by Zed Shaw earlier today (I was looking for his alluded rant on CC licenses, which I didn't find):

http://zedshaw.com/essays/programmer_stats.html

For instance, you state that:

  > In this week's tests, we have added latency tab (available using the rightmost tab at the top of this panel). On i7, we see that several frameworks are able to provide a response in under 10 milliseconds. Only Cake PHP requires more than 100 milliseconds.
Only cake php requires more than 100 milliseconds on average. But look at Django: Average around 60 ms, standard deviation around 90 ms (!). Not to mention a "worst" score of 1.4 seconds.

Re: Web Framework Benchmarks Round 2

#12

Post by voidlogic »

cmsimike wrote: post #9
voidlogic wrote: post #6

It is worth pointing out Go 1.0.3 not Go 1.1 beta is being used. I tested the code locally and on my machine 1.1 beta was about 20% "faster" (req/sec).

Really? We're definitely going to try that out!

In addition to the improvements from the improved scheduler, use of accept4 on Linux, better GC and code generation, Brad Fitzpatrick has been giving a lot of love to the "net/http" package. Here is a small example: https://plus.google.com/u/0/115863474911002159675/posts/L3o9...

Re: Web Framework Benchmarks Round 2

#13

Post by pfalls »

e12e wrote: post #11
bhauer wrote: post #2

This is our first follow up to last week's web framework benchmarks. Since last week, we have received dozens of comments, thoughts, questions, criticisms, and most importantly pull requests. This post shows data collected from a second run on EC2 and i7 hardware that started on Tuesday of this week. A third round with even more community contribution is already underway. Thanks especially to those who have contribut…

It's great that you're doing this, and listing stuff like standard deviation in the tables -- but I'd say your focus/interpretation of the data isn't quite right. At least provide the option to sort by standard deviation -- as that might well be more interesting than requests/second? Maybe I'm just being mean because I was reminded of this essay by Zed Shaw earlier today (I was looking for his alluded rant on CC lice…

Thanks for the suggestion, we're still in the early stages of getting the latency information incorporated, and having the ability to sort by the various metrics makes a lot of sense.

Re: Web Framework Benchmarks Round 2

#14

Post by Uchikoma »

Very interesting - again.

What I - again - find most interesting, is the large discrepancy in the 20-query test. Common wisdom seems to be that language performance is not important b/c every language hangs in the same IO/DB.

Re: Web Framework Benchmarks Round 2

#15

Post by TylerE »

pfalls wrote: post #10
coolsunglasses wrote: post #5

Go 1.1 beta is up, really need to get some results with tip instead of 1.0.3.

Go 1.1 is very high on our list, we'd love to get that into the tests as soon as we can. Of course, anyone in the Go community is welcome to issue a pull request that updated the version to 1.1. Edit: Just saw this pull request come in, so expect some Go 1.1 love in the next round. https://github.com/TechEmpower/FrameworkBenchmarks/pull/64

Don't think you'd need to change any code, just pull mercurial tip and recompile go and the app.

Re: Web Framework Benchmarks Round 2

#16

Post by giulianob »

bhauer wrote: post #2

This is our first follow up to last week's web framework benchmarks. Since last week, we have received dozens of comments, thoughts, questions, criticisms, and most importantly pull requests. This post shows data collected from a second run on EC2 and i7 hardware that started on Tuesday of this week. A third round with even more community contribution is already underway. Thanks especially to those who have contribut…

Did you guys turn on byte code caching for all the PHP frameworks? If not, then I recommend everyone ignore these benchmarks until that is completely done.

Re: Web Framework Benchmarks Round 2

#17

Post by nlh »

This is an awesome project, and thank you for the follow-up.

In particular, this is awesome because it's introduced me to some new frameworks that I hadn't even heard of (ie vert.x) that seem extremely interesting.

Keep it up and many thanks for this contribution!

Re: Web Framework Benchmarks Round 2

#18

Post by krg »

giulianob wrote: post #16
bhauer wrote: post #2

This is our first follow up to last week's web framework benchmarks. Since last week, we have received dozens of comments, thoughts, questions, criticisms, and most importantly pull requests. This post shows data collected from a second run on EC2 and i7 hardware that started on Tuesday of this week. A third round with even more community contribution is already underway. Thanks especially to those who have contribut…

Did you guys turn on byte code caching for all the PHP frameworks? If not, then I recommend everyone ignore these benchmarks until that is completely done.

We did. PHP 5.4.13 with APC, using PHP-FPM, running behind nginx.

Re: Web Framework Benchmarks Round 2

#19

Post by kainsavage »

giulianob wrote: post #16
bhauer wrote: post #2

This is our first follow up to last week's web framework benchmarks. Since last week, we have received dozens of comments, thoughts, questions, criticisms, and most importantly pull requests. This post shows data collected from a second run on EC2 and i7 hardware that started on Tuesday of this week. A third round with even more community contribution is already underway. Thanks especially to those who have contribut…

Did you guys turn on byte code caching for all the PHP frameworks? If not, then I recommend everyone ignore these benchmarks until that is completely done.

Yes, we are using APC for all the PHP tests in this round of benchmarking.

Re: Web Framework Benchmarks Round 2

#20

Post by voidlogic »

TylerE wrote: post #15
pfalls wrote: post #10

Earlier quoted context omitted.

Go 1.1 is very high on our list, we'd love to get that into the tests as soon as we can. Of course, anyone in the Go community is welcome to issue a pull request that updated the version to 1.1. Edit: Just saw this pull request come in, so expect some Go 1.1 love in the next round. https://github.com/TechEmpower/FrameworkBenchmarks/pull/64

Don't think you'd need to change any code, just pull mercurial tip and recompile go and the app.

  cd go/src
  hg pull
  hg update tip
  ./all.bash
Post reply on HN