Frameworks Round 6

#1

Post by amarsahinovic » Tue, Jul 02, 2013, 6:23 PM UTC

Re: Frameworks Round 6

#2

Post by bhauer » Tue, Jul 02, 2013, 6:29 PM UTC

Thanks for posting this, amarsahinovic.

This is the latest round of our ongoing project to benchmark the performance of fundamental services provided by modern web application frameworks. With the continued contributions of the developer community, the number of frameworks and platforms included is now 74! Round 6 also adds the often-requested plaintext test with HTTP pipelining (thanks in large part to William Glozer of Wrk fame).

The results web site also now includes a "framework overhead" tab that allows you to compare a framework versus its underlying platform. This was originally conceived by a Hacker News user named goodwink [1].

Like the round before, Round 6 gave us some technical challenges, including a period where we were unable to collect any latency data. Today, some problems persist, such as the Dart test (yes, there is server-side Dart) failing to complete for most of the tests. However, rather than continue to defer Round 6, we've decided it's better to move forward and use the round-over-round progression to iterate, fix, and improve. No problem that exists today is set in stone.

To that end, as always, I would love to hear any feedback, advice, questions, criticism, and especially contributions. And we hope you find this information helpful!

[1] https://news.ycombinator.com/item?id=5454970

Re: Frameworks Round 6

#3

Post by simonz05 » Tue, Jul 02, 2013, 6:35 PM UTC

bhauer wrote: post #2

Thanks for posting this, amarsahinovic. This is the latest round of our ongoing project to benchmark the performance of fundamental services provided by modern web application frameworks. With the continued contributions of the developer community, the number of frameworks and platforms included is now 74! Round 6 also adds the often-requested plaintext test with HTTP pipelining (thanks in large part to William Gloze…

Just want to say thanks for doing these tests. I think this is the first time we have such a thorough overview of the basic performance of web frameworks.

Re: Frameworks Round 6

#4

Post by columbo » Tue, Jul 02, 2013, 6:58 PM UTC

I'm really surprised at the performance of C#... honest no-flame question: how is possible it that Python is beating C#? I've been out of the .NET world for several years, almost a decade, but I always assumed they would be at Java level performance.

edit add: This is really fantastic work

Re: Frameworks Round 6

#5

Post by bhauer » Tue, Jul 02, 2013, 7:03 PM UTC

columbo wrote: post #4

I'm really surprised at the performance of C#... honest no-flame question: how is possible it that Python is beating C#? I've been out of the .NET world for several years, almost a decade, but I always assumed they would be at Java level performance. edit add: This is really fantastic work

Some caveats to point out:

* Round 6 does not yet use SQL Server. A contributor has provided SQL Server test implementations, but we don't expect to include that until Round ~7 because it will still require some additional work. It is conceivable that the .NET MySQL driver is especially poor.

* We do not yet have a test that is expressly heavy computation with no database connectivity. I suspect that C# and .NET would perform reasonably well on such a test (isolating it from its database connectivity).

* We do not yet have Windows i7 tests. But performance would likely scale about the same as seen between Linux EC2 to Linux i7.

All that said, it's also possible that the ASP.NET implementations of our tests could benefit from additional tuning. We have had a couple subject matter experts give contributions but we'd be happy to receive more.

Re: Frameworks Round 6

#6

Post by iaskwhy » Tue, Jul 02, 2013, 7:05 PM UTC

columbo wrote: post #4

I'm really surprised at the performance of C#... honest no-flame question: how is possible it that Python is beating C#? I've been out of the .NET world for several years, almost a decade, but I always assumed they would be at Java level performance. edit add: This is really fantastic work

I guess you need to change the Hardware to "Win" to see it shining.

Re: Frameworks Round 6

#7

Post by deelowe » Tue, Jul 02, 2013, 7:09 PM UTC

Thanks for continuing to do these. As far as I know, no one else is doing something like this in such a comprehensive way.

Re: Frameworks Round 6

#8

Post by rip747 » Tue, Jul 02, 2013, 7:10 PM UTC

i would really like to see how CFML engines (Railo, Adobe ColdFusion) and frameworks (FW/1, CFWheels, ColdBox) stack up in this comparison.

FD: I contribute to CFWheels

Re: Frameworks Round 6

#9

Post by bhauer » Tue, Jul 02, 2013, 7:22 PM UTC

rip747 wrote: post #8

i would really like to see how CFML engines (Railo, Adobe ColdFusion) and frameworks (FW/1, CFWheels, ColdBox) stack up in this comparison. FD: I contribute to CFWheels

We'd love to receive these test cases as pull requests if you have the time to contribute them. :)

https://github.com/TechEmpower/FrameworkBenchmarks

Re: Frameworks Round 6

#10

Post by jcampbell1 » Tue, Jul 02, 2013, 7:28 PM UTC

bhauer wrote: post #2

Thanks for posting this, amarsahinovic. This is the latest round of our ongoing project to benchmark the performance of fundamental services provided by modern web application frameworks. With the continued contributions of the developer community, the number of frameworks and platforms included is now 74! Round 6 also adds the often-requested plaintext test with HTTP pipelining (thanks in large part to William Gloze…

Luminus appears to be a clojure framework, not a php framework.

Thanks again for posting these tests.