Web Framework Benchmarks Round 2 • Hacker News Board

Re: Web Framework Benchmarks Round 2

#81

Post by meritt » Fri, Apr 05, 2013, 4:30 PM UTC

krg wrote: post #37

keammo1 wrote: post #31

Earlier quoted context omitted.

Is there somewhere we can see the settings for APC? Thanks for the great work!

https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... See php.ini, php-fpm.conf.

Having max/min spare servers as the same amount is a bad idea. This is incurs a substantial amount of process swapping, as every single request php-fpm is going to try and ensure there are precisely 256 idle servers.

Re: Web Framework Benchmarks Round 2

#82

Post by alberth » Fri, Apr 05, 2013, 4:31 PM UTC

I wish Lua was tested in the benchmarks.

In particular, Lua-CJSON [1]

[1] http://www.kyne.com.au/~mark/software/lua-json-performance.h...

(I know, everyone wishes their favorite language was represented and not all can be tested. I'm just posting this in case the OP does a "Round 3")

EDIT: fix typo

Re: Web Framework Benchmarks Round 2

#83

Post by jcroll » Fri, Apr 05, 2013, 4:31 PM UTC

jimmytucson wrote: post #79

Is anyone else alarmed to see raw PHP spanking Flask, Sinatra, and other Python/Ruby frameworks? If not, can anyone explain in layman's terms why that should be the case?

Why shouldn't that be the case? PHP has been designed from the get go to be super fast and 5.4 introduced a bunch of optimizations that have sped it up even more so. Ruby was designed for programmer happiness, PHP was designed for performance and getting sh*t done.

Re: Web Framework Benchmarks Round 2

#84

Post by bsnyder788 » Fri, Apr 05, 2013, 4:32 PM UTC

I think that the play-java one would likely benefit greatly from using raw jdbc rather than the ebean ORM. In fact my best guess is this is why the scala/java versions are so far apart. In past projects, scrapping ebean always led to large performance improvements in throughput, etc.

Re: Web Framework Benchmarks Round 2

#85

Post by kainsavage » Fri, Apr 05, 2013, 4:33 PM UTC

mike-- wrote: post #73

kainsavage wrote: post #64

Earlier quoted context omitted.

We have at least one Erlang test that came in yesterday and will be included in the next round of benchmarks.

thanks.

Don't thank us, we didn't write the code for it - one of you (the community) wrote it and submitted a pull request. We just ensured that the test would work in our benchmarks, then ran it! We are happy to do it, and super appreciative of the support from you guys!

Re: Web Framework Benchmarks Round 2

#86

Post by don_draper » Fri, Apr 05, 2013, 4:33 PM UTC

I code in Ruby, Python and Groovy/Grails. I'm amazed that Grails performs so much better than the others.

Re: Web Framework Benchmarks Round 2

#87

Post by thatthatis » Fri, Apr 05, 2013, 4:33 PM UTC

This is still a json API comparison, not a full framework comparison.

In the first test it sees how fast frameworks can serialize json and in the second test it sees how fast frameworks can access the database AND serialize json.

An inefficient json library and a framework is doomed across all tests. There are a lot of things frameworks do beyond serving json. These benchmarks are presented as representative of broad use when really they are only representative of the frameworks if used as a json api.

A raw HTML hello world, and removing the json encoding step in the db test would go a long way towards getting results that can mean what the authors seem to want them to mean.

There's nothing wrong with a json API benchmark, in fact its quite valuable in a lot of cases. But if that's what this is intended to be, the authors should say so.

Re: Web Framework Benchmarks Round 2

#88

Post by Zak » Fri, Apr 05, 2013, 4:33 PM UTC

A note about Compojure: they're using Korma for database access. Korma is an abstraction layer that's more Clojure-friendly than using SQL directly, but lighter than an ORM. Using clojure.java.jdbc might produce higher performance.

Re: Web Framework Benchmarks Round 2

#89

Post by kainsavage » Fri, Apr 05, 2013, 4:34 PM UTC

alberth wrote: post #82

I wish Lua was tested in the benchmarks. In particular, Lua-CJSON [1] [1] http://www.kyne.com.au/~mark/software/lua-json-performance.h... (I know, everyone wishes their favorite language was represented and not all can be tested. I'm just posting this in case the OP does a "Round 3") EDIT: fix typo

Personally, I really like Lua and I would love to see a pull request with a Lua framework test!

Re: Web Framework Benchmarks Round 2

#90

Post by kalmar » Fri, Apr 05, 2013, 4:35 PM UTC

kainsavage wrote: post #45

voidlogic wrote: post #41

It might be interesting to track average and peak server side memory usage during these tests. For example, if framework A is only 80% the "speed" of framework B, but uses 1/4 as much memory, for some users this might be a win for A.

This is definitely something we want to include in the tests, but it is difficult to manage a fair way of monitoring. Outside of the actual benchmarks, where we kept everything as fair as possible, we ran individual tests to make sure they worked prior to the benchmarking and would routinely have htop running at the same time to get an idea of what that sort of data looks like. It IS very interesting and is definitel…

One idea that jumps to mind is putting the running frameworks in an LXC container (or similar) and monitoring the memory usage of that. Not sure how accurate that is, but it's one avenue. It also still might not be fair because some runtimes can let their heap get pretty big despite not actually needing the memory. You'd want a way to differentiate those from the ones with big heaps that would choke with some memory pressure.