Web Framework Benchmarks Round 4 • Hacker News Board

Re: Web Framework Benchmarks Round 4

#131

Post by tlarkworthy » Thu, May 02, 2013, 5:12 PM UTC

diroussel wrote: post #93

bhauer wrote: post #2

This is the most recent update to our ongoing project measuring the performance of web application platforms and frameworks. In this round we've received several more community-contributed tests in Perl, PHP, Python, Java, and JavaScript. Go is a comeback champion thanks to changes made by Brad Fitzpatrick [1] and others in the Go community. A new "Fortunes" test was also added (implemented in 17 of the frameworks) t…

The requests per second is importanct, but some frameworks seem to get high average throughput but at the expense of a few slow requests. Also when measuring latency, average and std dev are only revelent if the distribution is guassian in distrition. Which is unlikely. Better to show percentile based measurements. Like 90% of all requests served in 5ms, and 99% of requests served in 15ms. See Gil Tene's talk "How no…

"average and std dev are only revelent if the distribution is Gaussian in distrition"

technically not true. Knowledge of the second order moment (variance) lets you uniquely identify other distributions like Poisson, or uniform. Knowledge of even higher order moments lets you fit more complicated statistical models.

Low variance is good, regardless of underlying distribution.

Re: Web Framework Benchmarks Round 4

#132

Post by desireco42 » Thu, May 02, 2013, 5:12 PM UTC

While I really enjoy results, I still would prefer to see smaller number of frameworks with specific tasks that are common in web development, logging in (assume user and pass are provided), listing of something. These two would be way more meaningful then anything else to me.

Again, thank you very much for hard work. I think for some there are some revelations there, like that C framework.

Re: Web Framework Benchmarks Round 4

#133

Post by bhauer » Thu, May 02, 2013, 5:13 PM UTC

diroussel wrote: post #89

voidlogic wrote: post #50

Earlier quoted context omitted.

Linux amd64 is the dominate internet serving platform; supporting a non-free OS for the benefit of a single test entry seems dumb. I'm all for a Mono entry however.

I don't think it seems dumb. It would be a very useful comparison between platforms. Stackoverlfow runs on .Net and Windows servers and they say it's very performant. So why not compare with other frameworks on the same hardware.

Voidlogic is right that we are waiting to get a pull request that will include some .Net frameworks [1]. If you can help, it would be greatly appreciated. We do want to include .Net. We will test on Mono to start.

We also want to test on .Net's native Windows platform. But we need to work on the testing platform we've built in order to automate a Windows server in the same way we presently automate a Linux server.

[1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/15...

Re: Web Framework Benchmarks Round 4

#134

Post by pjmlp » Thu, May 02, 2013, 5:13 PM UTC

ddellacosta wrote: post #30

axlerunner wrote: post #5

Good to see Java is holding its own.

I don't particularly love Java itself, but it is quite impressive how Java and other JVM languages are pretty much ass-kicking at the top of the list. That seems to go for heavyweight stuff as well as simpler "micro" frameworks. I guess if anything it speaks to what a solid piece of software the JVM is.

> I guess if anything it speaks to what a solid piece of software the JVM is.

While true, it all depends which JVM one talks about, there are plenty to choose from, even native code compilers.

Re: Web Framework Benchmarks Round 4

#135

Post by pjmlp » Thu, May 02, 2013, 5:14 PM UTC

What about vibe.d? While D is listed, vibe.d seems to be missing.

Re: Web Framework Benchmarks Round 4

#136

Post by mrgoldenbrown » Thu, May 02, 2013, 5:15 PM UTC

phillmv wrote: post #118

dllthomas wrote: post #58

Earlier quoted context omitted.

I agree that performance shouldn't dominate the decision, but there's no reason not to be informed by it - it can wind up mattering.

Except benchmarking is really, really hard to get right, and these benchmarks aren't really testing anything that resembles a production app. For all non-trivial apps, by the time you get 100 req/sec your bottleneck is very likely going to be your database.

This is a good point. Especially if you only cared about how fast you can make your app. But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? 1 or 10? At certain scales for certain tasks, the hosting costs matter more than the development costs.

Re: Web Framework Benchmarks Round 4

#137

Post by bhauer » Thu, May 02, 2013, 5:15 PM UTC

desireco42 wrote: post #132

While I really enjoy results, I still would prefer to see smaller number of frameworks with specific tasks that are common in web development, logging in (assume user and pass are provided), listing of something. These two would be way more meaningful then anything else to me. Again, thank you very much for hard work. I think for some there are some revelations there, like that C framework.

We aim to add more tests in time.

Take a look at the new Fortunes test. That test lists rows from a database, sorts them, and then renders the list using a server-side template. It's only implemented in 17 of the 57 frameworks right now, but we hope to have better coverage on that in time as well.

Re: Web Framework Benchmarks Round 4

#138

Post by bdesimone » Thu, May 02, 2013, 5:16 PM UTC

I'd like to thank you for the effort, and genuine openness of this project. I hope you'll continue to expand these tests.

Re: Web Framework Benchmarks Round 4

#139

Post by shellac » Thu, May 02, 2013, 5:16 PM UTC

kainsavage wrote: post #107

dthakur wrote: post #90

I asked for this before. Could you please add error bars to your plots?

There is a column labeled 'errors' for this, but basically only Lift and Phreeze had errors.

OP means 'error bars' in the sense of giving and idea of the accuracy of the measured value (e.g. standard deviation, standard error).

Re: Web Framework Benchmarks Round 4

#140

Post by bhauer » Thu, May 02, 2013, 5:17 PM UTC

pjmlp wrote: post #135

What about vibe.d? While D is listed, vibe.d seems to be missing.

Good catch. I have "D" listed because I heard from the vibe.d community [1] that a pull request was incoming. We have not yet received it. Can you help the vibe.d guys get it submitted?

[1] http://forum.dlang.org/thread/urpqdftuofgwespkcdxg@forum.dla...