Web Framework Benchmarks

>And let us simply draw the curtain of charity over the Cake PHP results.

Something about that phrase "Let us simply draw the curtain of charity..." resulted in an immediate spittake. Coffee everywhere.

Really depends on which kind of app you're working on. My main work app is 99% cached content so it would probably work just fine with almost anything. Developer time is certainly the biggest expense in my case so high-level it is.

> This exercise aims to provide a "baseline" for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application's performance can only get worse.

I disagree with the implication here (that this is a good point for comparison because "real-world application's performance can only get worse."). Yes it can only get worse but how much worse (per unit of "features") is both significant and unaddressed.

This isn't the best example but look at the gap between the top and bottom of the scale in the Database access test (single query) and Database access test (multiple queries) charts: In the first, Gemini is ~340x faster than Cake, in the second, only ~23x faster. There is still a big gap but it closed by an order of magnitude once you stepped past the most trivial possible DB access test.

So nodejs or php-raw is faster than cake at a single DB access, but what about when you create a real world scenario with authentication, requirement to be able to update features faster (i.e. use an ORM), env. portability requirement, etc.? It seems to me this would look like a little slower, a little slower, a little slower in the {raw} versions, and already included, already included, already included in Rails or Cake. The full featured frameworks take a lot of their performance penalty up-front, with less of a hit as features are added (maybe? :P).

My point is that it's not reasonable to assume that hackernews-benchmarks will actually reflect production use. That said I think the article is cool, and agree that it's good to keep framework authors' feet to the fire regarding performance!

I used Go's benchmarking tool to compare raw routing performance of various frameworks. The handlers all return a simple "Hello World" string. Here are the results:

  PASS
  Benchmark_Routes           100000	     13945 ns/op
  Benchmark_Pat              500000	      6068 ns/op
  Benchmark_GorillaHandler   200000	     11042 ns/op
  Benchmark_Webgo            100000	     26350 ns/op
  ok  	github.com/bradrydzewski/routes/bench	12.605s

I then ran the same benchmark, but this time I modified the handler to serialize a struct to JSON and write to the response. Here are the results:

  Benchmark_Routes              100000	     21446 ns/op
  Benchmark_Pat                 100000	     14130 ns/op
  Benchmark_GorillaHandler      100000	     17735 ns/op
  Benchmark_Webgo                50000	     33726 ns/op
  ok  	github.com/bradrydzewski/routes/bench	9.805s

In the first test, Pat is almost twice as a fast as the Gorilla framework. In the second test, when we added a bit more logic to the handler (marshaling a struct to JSON), Pat was only about 18% faster than Gorilla. In fact, it turns out it takes longer to serialize to JSON (8000ns) than it does for Pat to route and serve the request (6000ns).

Now, imagine I created a third benchmark that did something more complex, like executing a database query and serving the results using the html/template package. There would be a negligible difference in performance across frameworks because routing is not going to be your bottleneck.

I would personally choose my framework not just based on performance, but also based on productivity. One that can help me write code that is easier to test and easier to maintain in the long run.

Maybe you should've had the controllers execute raw SQL for a better comparison. I see you executing a regular query in Rails whereas your Java servlet is using prepared statements.