Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Framework Benchmarks

techempower.com

Post reply on HN
Print view View original

211–220 of 415 posts

Re: Web Framework Benchmarks

#211

Post by bhauer »

kclay wrote: post #161
bhauer wrote: post #92

Earlier quoted context omitted.

It's funny you should put this together because in an earlier draft of this blog entry I had created a tongue-in-cheek unit to express the average cost per additional line of code. Based on our Cake PHP numbers, I wanted to describe PHP as having the highest average cost per line of code. But we dropped this because I felt it ultimately wasn't fair to say that based on the limited data we had and it could be easily i…

I'm pretty shocked that Play scored so low. One would think that being built on netty would put Play in a higher rank. Database access for sure needs to be in async block

Agreed. We need to get the Play benchmark code updated to do the database queries in an asynchronous block. Accepting pull requests! :)

Re: Web Framework Benchmarks

#212

Post by goodwink »

jholla14 wrote: post #195
goodwink wrote: post #86

Earlier quoted context omitted.

These could probably be further broken down into micro-frameworks (like Express, Sinatra, Vert.x etc.) and large MVC frameworks (like Play and Rails). Gemini is sort of an outlier that doesn't really fit either category well, but the micro-frameworks have a fairly consistently higher framework optimization index than the large MVC frameworks which is as expected. Express and Sinatra really stand out as widely-used, v…

The play test was written totally incorrectly since it used blocking database calls. Since play is really just a layer on top of Netty it should perform nearly as well if correctly written.

I believe they're encouraging pull requests to fix that sort of thing. It will be interesting to see if it helps to that degree; I hope so!

Re: Web Framework Benchmarks

#213

Post by zerni »

jondot wrote: post #17

pfalls - amazingly, I spent the last 2 days of my holiday doing the same thing for a future open source project. I was just stumped when I saw you guys did the same (could have saved me a couple of days!) I wanted to find the leanest Web framework on any kind of platform; but the difference from your approach - I already knew the kind of code that would run on it. I tested: Go, Java (servlet, dropwizard), Scala (scal…

Scala really would have been nice. Especially a Scala/Akka/Spray kombo :)

I'm working with a setup like this and just love it!

Re: Web Framework Benchmarks

#215

Post by goodwink »

marcamillion wrote: post #185
goodwink wrote: post #21

One of the most interesting things this comparison brings out to me is not so much the differences between the various frameworks (although the differences between options on the same platform is definitely very useful information), but also the issue that few of us seem to think about these days: the cost of any of the frameworks above the bare standard library of the platform its hosted on. Theres a consistant, con…

Not quite sure I understand your point re: the cost of the frameworks, above the bare standard library. Do you mind breaking it down for me a bit please? Cost in dollars or cost in hardware utilization or some other cost?

If you have developers who for one reason or another prefer a given platform, then the most important performance comparisons are about how close various frameworks on that platform get to the performance of the platform itself.

Knowing how much I'm giving up in performance in order to get the features a given framework gives me is an important consideration. Also understanding when it's worthwhile to work outside the framework on the bare platform given the speedup I'll get versus the cost I'll incur by doing so is a very important optimization decision-making tool.

Re: Web Framework Benchmarks

#216

Post by ochoseis »

bhauer wrote: post #115
saym wrote: post #111

What I'd love to see paired with this data is a cost comparison. At what scale does performance of ruby/python/php become cost prohibitive? Twitter made the move from Ruby to Java some years back, did they ever post a comparison of their numbers before and after? Also, the difference between EC2 and local i7 hardware is glaringly obvious. At what scale does owning the server hardware become imperative? I know these q…

As you might imagine, during this exercise, we've had a lot of conversations about the points you raise. We have our own opinions, but we ultimately removed most of that content from the blog post because we didn't want it to be too editorial. We will be posting follow ups with some of our opinions. Some things are really difficult to answer in a vacuum. If you already have a competent devops staff, hosting your own…

I too was shocked at the deltas between Amazon and dedicated hardware. I think AWS runs on Xen, so I wonder if you could strike a more optimal performance/flexibility balance using lighter virtualization (LXC or OpenVZ) with either in-house hardware or another VPS provider.

Re: Web Framework Benchmarks

#217

Post by bhauer »

datasage wrote: post #140
jtreminio wrote: post #71

Earlier quoted context omitted.

The code says php-raw refers to using PDO (which all project should be using) vs using an ORM or ActiveRecord. I am completely stunned by the performance cost of using ORM/AR, and will be using this to shame our team lead into giving it up and going for raw queries.

Some of the ORM cost can be mitigated by using caching. In most cases this is essential in a production deployment.

Agreed. However, for our database tests we expressly wanted to stress-test the ORM and, where configurable, disabled caching.

We plan a subsequent test, time permitting, that enables caching.

Re: Web Framework Benchmarks

#218

Post by meritt »

bhauer wrote: post #73
chasing wrote: post #70

What's the difference between "php" and "php-raw" in some of the data? Maybe I'm still in my morning fog, but having trouble thinking of what "php-raw" might mean. Sigh.

Hi Chasing. We put a note about that suffix in the "Environment Details" section. The "raw" suffix means there is no ORM used. If there is no "raw" suffix, you can assume some type of ORM is used.

FYI on the PHP-DB-Raw approach: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

By default, PDO does not do true prepare()s, it just does string interpolation. You need to pass this parameter with the options:

    PDO::ATTR_EMULATE_PREPARES => false
And then you'll actually be using MySQL prepared statements. You'll see a noticeable performance improvement for large amounts of queries.

Re: Web Framework Benchmarks

#219

Post by mark242 »

kainsavage wrote: post #64
neya wrote: post #53

How about Lift? Btw, the play framework you tested is Java or Scala based? Either way, I'm shocked to see Play perform so slow comparatively. Although it's easily 10x faster than rails on most tests, I'm shocked to see Node.js faster than Play! (by 2x in most cases) Wow!! Maybe Node.js critics should start appreciating it after all..

It is probably worth noting that while we strive to make the tests as fair as possible, we followed the official tutorials for each framework when building out the tests but we fully expect there to be small instances where minor tweaks improve a given test. Given that I am no Play expert, it would be of great value to have one who is (and it sounds like you could lend a hand there) to check out the code on the githu…

There are a few problems with your Play code that are causing it to be unnecessarily slow.

First-- what you're really testing here is the Jackson library. A majority of the cycles used in your application are being burned in that toJson call of an array of objects. This isn't a fair test compared to the servlet implementation because you're calling Jackson against a map in the Play example, versus against a simple String in the servlet example.

Second-- you are running database calls serially, and those are blocking. Considering that you're using the more-or-less default Play/Akka configuration, there are only enough threads as you have available processor cores. I would start by increasing the parallelism-factor and parallelism-max to be higher, so you'll have more available threads. More importantly though, the database access should be wrapped in a Future, and you should be returning asynchronously. This should speed up the application by a huge amount.

Re: Web Framework Benchmarks

#220

Post by bhauer »

mixmastamyk wrote: post #205

Wasn't it agreed a long time ago that benchmarking against a VM was a bad idea?

Possibly, although we believe it represents precisely what we wanted to test: a realistic production environment.

However, we also tested on our physical i7 hardware. Did you happen to scroll down? :)

Post reply on HN