Web Framework Benchmarks Round 2 • Hacker News Board

Re: Web Framework Benchmarks Round 2

#191

Post by eranation » Fri, Apr 05, 2013, 7:48 PM UTC

bhauer wrote: post #2

This is our first follow up to last week's web framework benchmarks. Since last week, we have received dozens of comments, thoughts, questions, criticisms, and most importantly pull requests. This post shows data collected from a second run on EC2 and i7 hardware that started on Tuesday of this week. A third round with even more community contribution is already underway. Thanks especially to those who have contribut…

First time I'm actually happy for building something with Servlets (have a feeling I'm the only one here that uses it as the go to framework for hacking something quickly, I'm an old man), it's indeed dead annoying for REST services, but doable, and blazing fast. The question is - when does my productivity starts to be more important than performance (I think 99% of the lifetime of anything I'll ever build is the first and not the latter).

I'm very surprised to see Play-Scala is not in the same playing field as other JVM based frameworks, had a lot of hope for it, I hope TypeSafe will take that into consideration...

Node.js is the biggest surprise for me on the good side, I think it changes my plans a little as for what to learn next...

Thanks for a very valuable study, this is great

Re: Web Framework Benchmarks Round 2

#192

Post by superuser2 » Fri, Apr 05, 2013, 7:51 PM UTC

This is definitely motivation to learn another framework besides Rails and Django.

Compojure's syntax and design seem really attractive... is anyone using that in production? Would it be a reasonable choice for a "serious" web application?

Re: Web Framework Benchmarks Round 2

#193

Post by msgilligan » Fri, Apr 05, 2013, 8:00 PM UTC

pfg wrote: post #152

don_draper wrote: post #86

I code in Ruby, Python and Groovy/Grails. I'm amazed that Grails performs so much better than the others.

I guess that's one of the big advantages JVM languages have. Using stuff like Spring or Hibernate (like Grails does), you benefit from thousands of person-hours spent on tuning those libraries and you still get native Java performance. Obviously there's a price you have to pay for all the things Grails built on top of Spring/Hibernate (in the benchmark it looks like Grails' performance = Spring / 2), but in general,…

It would be interesting to see the results for Grails 2.2 (which uses Groovy 2 and Invoke Dynamic)

One of the nice things about Grails is that you can easily drop down to the Spring and/or JVM level when needed for performance.

Re: Web Framework Benchmarks Round 2

#194

Post by bhauer » Fri, Apr 05, 2013, 8:05 PM UTC

vanderZwan wrote: post #114

pfalls wrote: post #13

Earlier quoted context omitted.

Thanks for the suggestion, we're still in the early stages of getting the latency information incorporated, and having the ability to sort by the various metrics makes a lot of sense.

As one of the people complaining about statistics last week (and also, by coincidence, citing Zed's rant), I'm glad to see you are working on it and open to more ideas and improvements! Also, I like the "sportsmanlike benchmarking game between different communities" vibe I'm getting from all this. Would be nice if the community helps turn this into the de facto example of how to benchmark correctly. Now I'll just hav…

Go 1.1 is looking very strong! Pat (pfalls) just showed me some very preliminary numbers and we are extremely happy with them.

Thanks for the input and constructive criticism, vanderZwan. It has been very helpful to get feedback from yourself and everyone else. I too am particularly happy with the sportsmanlike competition vibe. You have no idea how fulfilling that is to us.

Re: Web Framework Benchmarks Round 2

#195

Post by abalone » Fri, Apr 05, 2013, 8:08 PM UTC

Interesting that the Play code optimizations had virtually no effect. Almost identical absolute scores between tests. Clearly something heavy's going on within the Play request handling framework to slow things down, not the code we see.

I was also surprised at the difference between the Java and Scala play test, since I thought they were supposed to be similar. But it looks like the approaches are quite different. The Java DB test uses the ebean ORM while Scala does not. The JSON code also looks a bit different, with the Java version surfacing more Jackson internals. I don't know if these are necessarily "wrong", but perhaps it's not as apples-to-apples a comparison as it could be.

Re: Web Framework Benchmarks Round 2

#196

Post by Zak » Fri, Apr 05, 2013, 8:10 PM UTC

superuser2 wrote: post #192

This is definitely motivation to learn another framework besides Rails and Django. Compojure's syntax and design seem really attractive... is anyone using that in production? Would it be a reasonable choice for a "serious" web application?

I've used it for client work that was in production and for a side-project that's in production, but private with few users. It has been reliable for me.

Note that Compojure is more of a routing library than a framework. It's similar to Flask or Sinatra. Clojure's philosophy includes aggressive separation of concerns such that full-stack frameworks aren't really "the Clojure way".

Re: Web Framework Benchmarks Round 2

#197

Post by bhauer » Fri, Apr 05, 2013, 8:14 PM UTC

errnoh wrote: post #166

DonnyV wrote: post #101

Earlier quoted context omitted.

Why no .NET C# and Mono?

Their repo at https://github.com/TechEmpower/FrameworkBenchmarks is open for pull requests I don't think they have anything against C#, probably no one just has committed C# version to be benched. (edit: whops, linked fork first)

Thanks for the link, errnoh. You're precisely right, we want to have C# and .Net tests included but didn't find the time to do so ourselves in the past week and have not yet received a pull request. I was not familiar with ServiceStack prior to feedback we received to last week's test, but from the looks of it, I'd personally like to see how it does.

Re: Web Framework Benchmarks Round 2

#198

Post by Zak » Fri, Apr 05, 2013, 8:15 PM UTC

jamesaguilar wrote: post #161

superuser2 wrote: post #106

Earlier quoted context omitted.

Performant, not scalable. You can throw money at Rails and it will handle the load, but it would cost a lot more.

If budget is finite, performant ~= scalable once you get to the point where you're actually trying to scale.

I think most people these days take scalability to mean the ability to handle more load by adding hardware without changing the architecture. That's quite different from how much load the same hardware can handle (performance).

Re: Web Framework Benchmarks Round 2

#199

Post by bhauer » Fri, Apr 05, 2013, 8:19 PM UTC

abalone wrote: post #195

Interesting that the Play code optimizations had virtually no effect. Almost identical absolute scores between tests. Clearly something heavy's going on within the Play request handling framework to slow things down, not the code we see. I was also surprised at the difference between the Java and Scala play test, since I thought they were supposed to be similar. But it looks like the approaches are quite different. T…

Agreed, and I would certainly like to see the Play tests (both Scala and Java) improve versus what we have measured so far. I would not rule out a configuration glitch in our deployment either. But on that front, I'm really hoping the Play experts can lend a hand. I think we've received a couple tips about the database connection pool size.

If the contributions we've seen so far are any indicator, we're going to need to get more clever with how to show and hide rows in our results tables! :) But I'd like to see a few more rows added to cover the various permutations of the ORM and JSON options, as you point out.

Re: Web Framework Benchmarks Round 2

#200

Post by alberich » Fri, Apr 05, 2013, 8:22 PM UTC

columbo wrote: post #94

milesf wrote: post #61

It's a useful data point to know the speed of frameworks, but what matters most to me is speed of development and whether or not I enjoy the process. Wish there was a way to benchmark those points.

>what matters most to me is speed of development and whether or not I enjoy the process. Matters most? There are 100x differences here. Take two companies writing the same application, one in Spring the other in Rails. Both have easy access to knowledgable people, both are industry standard. However the Spring application would be several orders of magnitude more scalable[1]. I'd consider that to be a more important…

Hm, I'm not sure a benchmark like these allows one to draw such conclusions.

The benchmarks, although good, are a best case scenario and of limited aspects of those frameworks. Real applications don't behave like this. What if on the average case such huge differences in performance becomes eclipsed by other factors?