Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Framework Benchmarks Round 4

techempower.com

Post reply on HN
Print view View original

301–310 of 358 posts

Re: Web Framework Benchmarks Round 4

#301

Post by bhauer »

cies wrote: post #300

Could anyone explain: why is gemini sooo much faster then the others in these tests? I believe that the way these test are setup slightly advantage gemini, and more broadly java. Since they do not measure memory usage, or tasks that make memory usage critical, which is something JVM sucks at.

Hi Cies,

Gemini is our in-house framework and there are two points to consider:

(a) We are obviously very familiar with Gemini and therefore know how to use it effectively. For example, we know that we prefer to deploy Gemini applications using the Caucho Resin application server because it has proven the quickest Java application server in our previous experience. Of course, the other Java Servlet-based frameworks also benefit from deployment on Resin in these tests.

(b) In our design of Gemini, we do keep an eye on performance. But as the data shows, there are faster options.

Although we included Gemini in these tests, we did so because we wanted to know how it stacked up against other frameworks that we routinely use on projects. See more information in response to an earlier question here: https://groups.google.com/d/msg/framework-benchmarks/p3PbUTg...

Incidentally, the memory usage profile for the Gemini test is fairly compact, in terms of used heap space within the JVM, as are most of the Java tests in this project. With no need to do so, we're not trimming the memory allocation for the JVM to a minimum in our tests. But if we did, as you point out, the tests we've implemented so far don't require much memory.

Re: Web Framework Benchmarks Round 4

#302

Post by JulienSchmidt »

thrownaway2424 wrote: post #155
DennisP wrote: post #80

Earlier quoted context omitted.

Because of your comment I checked the latency view and found Go doing quite well, sometimes in the second or third spot. Perhaps I'm missing your point?

Well, in the database access test with multiple queries, Go has an outrageous max latency of 19 seconds, and a stddev of 3x the mean, which is terrible.

This is probably caused by running all the queries in their own goroutine (if you don't know what this is just think of it as a thread - but much cheaper). This causes the queries to be handled in more or less random order. We'll fix this for Round 5 ;-)

Re: Web Framework Benchmarks Round 4

#303

Post by JoshGlazebrook »

hilem wrote: post #144
krg wrote: post #8

Earlier quoted context omitted.

Ruby 2.0.0-p0 http://www.techempower.com/benchmarks/#section=environment

though they're also using Rails 3.2.11, which not only isn't the latest 3.2.* release but doesn't take advantage of Ruby 2.0... it would be more interesting to see the latest Rails 3.2.* on Ruby 1.9.* along with the latest Rails 4 release candidate on Ruby 2.0.0-p0

So move it up 2 or 3 spots then...

Re: Web Framework Benchmarks Round 4

#305

Post by est »

dataminer wrote: post #125
mitchi wrote: post #18

http://openresty.org/ Looks interesting but at a quick glance it looks like you are programming in configuration files? I'm not sure I like the idea.

You can try Lapis http://leafo.net/lapis/ its a new framework built on openresty.

There's also http://luvit.io/, a nodejs core (libuv) but with LuaJIT2

Re: Web Framework Benchmarks Round 4

#306

Post by carbocation »

nickpresta wrote: post #267
camus wrote: post #230

Earlier quoted context omitted.

You'll be better off writing raw PHP without classes than using Go. Go performances are raw Go. As soon as you add any framework to the party , the perfs are falling. see the gorilla test which is a Go framework.

Where are the Gorilla benchmarks? I only see raw Go for all 4 benchmarks.

Gorilla is a toolkit, not a framework. It's nice, but it doesn't do all that much beyond the standard library. To me, Go+Gorilla vs raw Go is not similar to, say, a PHP framework vs raw PHP.

Re: Web Framework Benchmarks Round 4

#307

Post by nickpresta »

carbocation wrote: post #306
nickpresta wrote: post #267

Earlier quoted context omitted.

Where are the Gorilla benchmarks? I only see raw Go for all 4 benchmarks.

Gorilla is a toolkit, not a framework. It's nice, but it doesn't do all that much beyond the standard library. To me, Go+Gorilla vs raw Go is not similar to, say, a PHP framework vs raw PHP.

(I know what Gorilla is, I've contributed to it :D)

The OP, to which I was replying, said: "see the gorilla test which is a Go framework."

I asked about the Gorilla benchmark itself, since OP seemed to say that adding something like Gorilla would slow down the Go benchmarks, which I don't agree with.

Re: Web Framework Benchmarks Round 4

#308

Post by bhauer »

jakejake wrote: post #200
bhauer wrote: post #113

Earlier quoted context omitted.

Hi saosebastiao, As Pat points out, we definitely look forward to implementing some more computationally-intense request types in the future. This round does include the first server-side template test. We'd like to hear the community's opinions about more tests. That said, I feel most of the frameworks' implementations of the existing tests are not cheating. Our objective in this project is to measure every framewor…

I would like to pile my thanks onto this list as well. I'm the author of Phreeze and I can say that I'm grateful that fairness is being encouraged. There is certainly glory in ranking well on any benchmark and I have to admit, as I was implementing the tests in Phreeze, I saw many opportunities to "cheat." For example, skipping the framework routing, not using the "proper" way to communicate between the layers, etc a…

Jakejake, perhaps I've said it before, but it bears repeating: your reaction to and participation in this project has been precisely the kind we hoped it would see (but weren't sure we'd actually see in practice). Thank you very much for joining in and having fun with it. It sounds like you've been able to get some increased performance from your tuning, and I hope you don't mind us feeling a little bit of pride in having inspired that.

Some readers may feel we are attempting to paint some frameworks in a poor light. Yes, we do have favorites, but we are absolutely intent on keeping this open and fair. If we're doing something wrong, help us fix it! A pull request is very happily received.

When I read reactions of that sort, I selfishly want to point the author to Jakejake's comments to demonstrate how awesome it is to see a framework improving. Speaking of that, I want to eventually have the ability to show performance over time (e.g., compare Round 1 to Round X) as a potentially interesting illustration of a framework's intent to improve performance.

Also, thanks for the idea for a future test. That sounds like a good one.

Re: Web Framework Benchmarks Round 4

#309

Post by DennisP »

bsaul wrote: post #249
apalmer wrote: post #244

Earlier quoted context omitted.

ASP.Net kind of put themselves out of the benchmark game here: Mono Issue #1, since the vast vast majority of ASP.Net websites run on windows a Mono performance test even if accurate is going to be of dubious value. Mono Issue #2, since Mono is nowhere near as polished as the Microsoft .Net implementation the numbers wont really be meaningful. Windows Issue #1, if you do the test on a different OS than every other te…

I completely understand your point, but I think it's fair to say that most .Net code will run on Windows server, and that pretty much everything else will run on some kind of linux flavor. Just like you have a "keep the default framework setting" approach to help compare very different frameworks because that's how the majority of people will use them, you may very well assume that comparing frameworks on their prefe…

Exactly, and if there is such a performance boost, having the numbers would help you figure out whether hardware savings would be worth the licensing cost.

Re: Web Framework Benchmarks Round 4

#310

Post by bhauer »

iknight wrote: post #189
platz wrote: post #43

Earlier quoted context omitted.

http://www.techempower.com/benchmarks/#section=motivation "As with the previous question, we'd love to. We have heard tentative word from a reader/contributor that a pull request may be incoming soon that will include several .NET frameworks on Mono, which we assume will be as easy to include as any other pull request. One challenge we face is that the test infrastructure we've built assumes a Linux deployment that w…

Can you share who the Mono contributor is?

Sure. We now have two pending issues, one of which is a pull request. If you can weigh in and perhaps contribute some review time to the PR in particular, that would be very appreciated!

https://github.com/TechEmpower/FrameworkBenchmarks/pull/239

https://github.com/TechEmpower/FrameworkBenchmarks/issues/15...

Post reply on HN