Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Framework Benchmarks Round 4

techempower.com

Post reply on HN
Print view View original

281–290 of 358 posts

Re: Web Framework Benchmarks Round 4

#281

Post by mark_l_watson »

riprock wrote: post #257

Compojure is looking really attractive ("modern" dynamic language with JVM performance backing it up). How is the ecosystem around Clojure web development?

The ecosystem around Clojure web development is in good shape. I have used Compojure and Hiccup (and Noir) for a large part of my web development in the last few years and it has been a happy experience.

I have had a little less joy experimenting with both Clojurescript and Ember.js (with Clojure back end services): I eventually get things working, but at a huge time cost over writing non-rich clients just using Hiccup.

Re: Web Framework Benchmarks Round 4

#282

Post by kbenson »

laumars wrote: post #226
kbenson wrote: post #175

Earlier quoted context omitted.

I don't consider the Go size pretty small. Mojolicious[1], Dancer[2] and Kelp[3] have set the bar for small code size for me. Not sure yet if there are smaller ones (note that there are no other files required for those apps, period) In the same vein, Lua's OpenResty[4] looks good, as do Tornado[5], Flask[6] and Bottle[7] (although you need to tease the raw/ORM methods apart to get an idea for the last two). And of c…

> I don't consider the Go size pretty small. Mojolicious[1], Dancer[2] and Kelp[3] have set the bar for small code size for me. You've basically just listed Perl 3 times though. Particularly when the guts of the code in all 3 of those examples was Perl's standard database interface (the same DBI you'd use for CGI Perl or even standalone .pl scripts). I do love Perl for the flexibility of it's syntax and how concise t…

> Yeah, the post started slightly different than it ended, and that's an artifact of that change. My next paragraph listed many more, and I tried to do multiple from each language that I looked at which had simple implementation.s

> Particularly when the guts of the code in all 3 of those examples was Perl's standard database interface (the same DBI you'd use for CGI Perl or even standalone .pl scripts).

The benchmark page clearly tags which implementations use raw SQL access and which use an ORM. These all happen to be using raw SQL. To my knowledge, none of them have a pre-bundled ORM, and I'm not sure whether the ORM tested implementations are only supposed to indicate the pre-shipped ORM.

> But for me the performance of Go won out

I wasn't trying to imply they competed on that metric, I just wanted to give some examples of much simpler implementations. What one considers small is obviously relative.

> I've not tried Mojolicious, Dancer nor Kelp so I couldn't comment on how they compare for performance.

They all look to be bottom-half of the full set of results, performance wise. Mojolicious quite a bit slower (relatively, they are all slow compared to Go) than the others, most likely because it uses it's own internal, pure Perl JSON module. There's was to fall back to the optimized C-based JSON::XS module, but I'm not sure whether that would keep with the spirit of the benchmarks.

Re: Web Framework Benchmarks Round 4

#283

Post by voidlogic »

apalmer wrote: post #244
nopal wrote: post #16

Earlier quoted context omitted.

Any reason you didn't test ASP.NET MVC or ASP Web Forms?

ASP.Net kind of put themselves out of the benchmark game here: Mono Issue #1, since the vast vast majority of ASP.Net websites run on windows a Mono performance test even if accurate is going to be of dubious value. Mono Issue #2, since Mono is nowhere near as polished as the Microsoft .Net implementation the numbers wont really be meaningful. Windows Issue #1, if you do the test on a different OS than every other te…

Care to backup your Mono claims with evidence?

I have a friend who went to a Mono talk at MS MIX where a Mono developer was speaking. The Mono developer said that while Mono is a little slower than .NET (and he was talking a couple percent) Mono often ends up being faster on the same hardware because the Linux system calls the runtime uses are faster. There were a few ROFLs in the audience.

I also agree with you that investing in a .NET (Windows) test isn't good bang for the buck here.

Re: Web Framework Benchmarks Round 4

#284

Post by pekk »

bhauer wrote: post #113
saosebastiao wrote: post #15

I love what you guys are doing. This is by far the most comprehensive (in terms of number of frameworks) comparison of web frameworks. I also am a big fan of the new filtering metadata. However, I'm starting to think that all of the advocates of various frameworks are now conspiring independently to make this comparison meaningless...any framework (except Cake for some reason) can be superoptimized towards a small se…

Hi saosebastiao, As Pat points out, we definitely look forward to implementing some more computationally-intense request types in the future. This round does include the first server-side template test. We'd like to hear the community's opinions about more tests. That said, I feel most of the frameworks' implementations of the existing tests are not cheating. Our objective in this project is to measure every framewor…

The logic you have previously posted on HN for these benchmarks is that they measure the minimum overhead available on the platform, so that you cannot get faster than the benchmarked numbers. If a framework is too slow, the framework-chooser can exclude it from consideration for because the resulting project just can't be any faster than the framework benchmark. Sounds reasonable.

Except now it is clear that you are refusing optimizations for some frameworks due to a vague, aesthetic judgement of 'stripped'. Which now means that you actually aren't measuring the minimum framework overhead. You are measuring the overhead of the defaults, or the overhead of not taking optimization seriously, with large amounts of performance left on the table. Worse, selectively applying optimizations means you are comparing one framework's defaults to another framework's minimum overhead. And since you have abandoned minimum overhead, it now makes very little sense about why we are measuring performance independent of normal first-resort tactics like caching (who is running Cake without caching?)

If you were going to do that, you should have benchmarked defaults right down the line and allowed a full, normal range of simple deployment optimizations. Instead we have selective optimization and totally unrealistic deploys, so it really indicates very little.

Re: Web Framework Benchmarks Round 4

#287

Post by laumars »

kbenson wrote: post #282
laumars wrote: post #226

Earlier quoted context omitted.

> I don't consider the Go size pretty small. Mojolicious[1], Dancer[2] and Kelp[3] have set the bar for small code size for me. You've basically just listed Perl 3 times though. Particularly when the guts of the code in all 3 of those examples was Perl's standard database interface (the same DBI you'd use for CGI Perl or even standalone .pl scripts). I do love Perl for the flexibility of it's syntax and how concise t…

> Yeah, the post started slightly different than it ended, and that's an artifact of that change. My next paragraph listed many more, and I tried to do multiple from each language that I looked at which had simple implementation.s > Particularly when the guts of the code in all 3 of those examples was Perl's standard database interface (the same DBI you'd use for CGI Perl or even standalone .pl scripts). The benchmar…

> The benchmark page clearly tags which implementations use raw SQL access and which use an ORM. These all happen to be using raw SQL. To my knowledge, none of them have a pre-bundled ORM, and I'm not sure whether the ORM tested implementations are only supposed to indicate the pre-shipped ORM.

You miss my point. All of those examples you gave used the same core database framework and as the test was primarily a database performance test, all those 3 examples were essentially the same core Perl code.

Whether it's ORM or raw SQL is completely besides the point (though since we're on the topic, Perl's DBI basically works the same as Go's - or rather that should be the other way around given their age).

>I wasn't trying to imply they competed on that metric

Again, you missed my point. I wasn't suggestion that you were comparing the performance of the two. I was commenting on why I switched away from Perl to Go.

> I just wanted to give some examples of much simpler implementations.

Except you didn't You gave AN example (singular). It was one language; Perl.

> They all look to be bottom-half of the full set of results, performance wise.

I wouldn't trust that kind of benchmark for comparisons of Perl frameworks as setting up a Perl environment isn't as simple as compiling a Go program. With Perl, you have a number of different ways you can hook the runtimes into the web server (CGI, Apache libs, etc), pure Perl and C libraries (which you also mentioned) that significantly affect both memory usage and runtime performance and a whole boatload of config ($ENVS in mod_perl, bespoke handlers, etc) that also affect performance.

The ironic thing with Perl is despite scripts in the language being some of the most portable code on the POSIX community, running performance critical Perl webapps leads to very unportable set ups. (which was the other reason I migrated my sites to Go).

This might sound critical, but I genuinely do love Perl. I'd say it was up there as one of my favourite languages (and over the years I've learn to develop in a great number of different languages). But sadly nothing in life is perfect.

Re: Web Framework Benchmarks Round 4

#288

Post by threeseed »

raylu wrote: post #192
peeters wrote: post #75

Earlier quoted context omitted.

Java gets that wrap from originally being slow to execute, and also having a huge up-front cost to spin up a VM. The first isn't true any more: the Java VM competes with native code on most benchmarks, and due to its ability to perform runtime optimizations, can occasionally outperform native code. The second doesn't matter at all for web servers. The cost of starting up the web server is tertiary to uptime and perfo…

I agree with everything you've said here, but I'd like to add something about startup time. If it takes you 5s to start up your server, that's a lot of time you've added to each development iteration. Make a change, restart the server, wait 5s, see if it works/check debug output.

    Make a change, restart the server, wait 5s, see if it works/check debug output.
Look at JRebel, Play or Nailgun.

Most of us aren't waiting 5 seconds before checking output. We just refresh the browser.

Re: Web Framework Benchmarks Round 4

#289

Post by jakejake »

camus wrote: post #259
jakejake wrote: post #200

Earlier quoted context omitted.

I would like to pile my thanks onto this list as well. I'm the author of Phreeze and I can say that I'm grateful that fairness is being encouraged. There is certainly glory in ranking well on any benchmark and I have to admit, as I was implementing the tests in Phreeze, I saw many opportunities to "cheat." For example, skipping the framework routing, not using the "proper" way to communicate between the layers, etc a…

Can you share with us the tuning you did with class loading for instance ? thanks for your comment.

Oh sure, nothing to complicated. Basically I just happened to notice that I loaded several classes that were not always needed. I was able to tune up the framework to load some of them on-demand instead.

One example is that the framework loaded an lot of MySQL classes whether or not you do a DB query. So, now I wait to initialize the DB stuff until after you make a call that requires it. Phreeze has always been lazy about opening the DB connection, but now it's even lazier and doesn't even load the classes until you need them!

There were some other utility-type classes like XML parsing and such that probably don't even get used much. So that is lazy loaded now too.

For a non-DB request I was able to get it down from about 37 files that loaded to around 20. For a DB request I think it's still around 30 files, but I definitely consider that a performance improvement. The benchmark led me to scrutinize what is being loaded so I think it has already improved the framework.

Post reply on HN