Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Framework Benchmarks Round 4

techempower.com

Post reply on HN
Print view View original

201–210 of 358 posts

Re: Web Framework Benchmarks Round 4

#201

Post by henk53 »

exabrial wrote: post #45
rallison wrote: post #12

For those new to these benchmarks, all of the source code is up on github: https://github.com/TechEmpower/FrameworkBenchmarks We welcome all pull requests, suggestions and criticisms.

Any chance you could add JEE6? The two major JSF 2.1 implementations MyFaces and Mojarra are both missing. I don't expect awesome performance numbers, but since JSF2.1 is the 'official' (sigh) web framework of JEE6, it would be interesting how awful they are compared to some of these other languages. I guess though if this is only testing JSON serialization, it may not make sense. Perhaps adding JAX-RS implementation…

>it would be interesting how awful they are compared to some of these other languages.

The World Wide Wait benchmark, which tested quite a lot, showed quite favorable numbers for both JSF implementations.

Re: Web Framework Benchmarks Round 4

#202

Post by desireco42 »

oberhamsi wrote: post #198
desireco42 wrote: post #194

I have one more question, I see a number of really fat frameworks on top, if rails is faster, then it must be because it is heavily optimized and people who made it are super smart, but for example codeigniter is above slim, kohana is above fuel, like twice the speed for Fortunes test. This isn't what I would expect at all, I would also expect Rails to be below php frameworks for speed alone. Did you use orm when it…

the source is available. as a first guess: if an ORM is the typical way to do it in an env then it was used.

yeah, my bad, should look into the source :)

Re: Web Framework Benchmarks Round 4

#203

Post by rdtsc »

Very nice presentation!

One thing I am wondering is "what about concurrency level"?

Just because a server can handle 10x the number of requests when doing a single request a time for 1000 requests, doesn't necessarily mean it can also handle those 1000 request at 10x performance when they all come in at once or in a short time period.

I saw some tests have "256 concurrency" does that mean they are sending 256 request concurrently? I want to see them play more with those numbers. Why not have 1024 or more. Then also play with the number of available CPUs and see which frameworks can auto-scale based on that. Some that can process sequential requests fast might fall face down when faced with slightly increased concurrency, in that respect these benchmarks are a bit misleading.

On the other hand it is good to see latency. That is a important. Now latency vs level of concurrency would also be interesting.

Re: Web Framework Benchmarks Round 4

#204

Post by desireco42 »

desireco42 wrote: post #194

I have one more question, I see a number of really fat frameworks on top, if rails is faster, then it must be because it is heavily optimized and people who made it are super smart, but for example codeigniter is above slim, kohana is above fuel, like twice the speed for Fortunes test. This isn't what I would expect at all, I would also expect Rails to be below php frameworks for speed alone. Did you use orm when it…

Ok to answer my question, it is raw query passing to db. I know this is easier for test to be made, but it is not as realistic. Having said that, I really appreciate work you are doing, let me repeat that for 100 times, I don't want to sound ungrateful, but this is still not the benchmark that can be used to compare frameworks.

Re: Web Framework Benchmarks Round 4

#205

Post by henk53 »

raylu wrote: post #192
peeters wrote: post #75

Earlier quoted context omitted.

Java gets that wrap from originally being slow to execute, and also having a huge up-front cost to spin up a VM. The first isn't true any more: the Java VM competes with native code on most benchmarks, and due to its ability to perform runtime optimizations, can occasionally outperform native code. The second doesn't matter at all for web servers. The cost of starting up the web server is tertiary to uptime and perfo…

I agree with everything you've said here, but I'd like to add something about startup time. If it takes you 5s to start up your server, that's a lot of time you've added to each development iteration. Make a change, restart the server, wait 5s, see if it works/check debug output.

JBoss EAP 6.0.1 does a cold start in 1.5 seconds on my 2.93Ghz i7. TomEE is some 2 seconds.

If even that's too much, there's JRebel which does full hot reloading of pretty much every piece of code you change.

Re: Web Framework Benchmarks Round 4

#206

Post by dragonwriter »

candybar wrote: post #188
dragonwriter wrote: post #182

Earlier quoted context omitted.

> Many people choose Ruby and figure, given that premature optimization is the root of all evil, they'll optimize later if needed. True. > That's like choosing between a farm tractor or a ferrari - and figuring if the tractor doesn't perform up to snuff, we'll add a spoiler Its really not like that at all, because programming languages aren't like vehicles. Particularly, with Ruby, on typical method of optimization i…

I keep hearing about this, but do people really rewrite performance-critical parts of their web apps in C? Even if it happens to be part of some third-party library? And maintain a fork? What if that performance-critical part is dependent on other parts in a non-trivial way? It seems that an unanticipated replacement of some core functionality with a C library may involve a major rewrite and most Ruby teams may not h…

> I keep hearing about this, but do people really rewrite performance-critical parts of their web apps in C?

Certainly they do it for Ruby apps in general. I don't think its all that common for it to be a high-value proposition for web apps.

> Even if it happens to be part of some third-party library? And maintain a fork?

If its an open-source third-party library that tends to get used in a way that is performance-critical, upstream will probably accept moving bottlenecks to (portable) C and maintaining the API, so its unlikely that you'll need to take up responsibility for a fork.

> What if that performance-critical part is dependent on other parts in a non-trivial way?

If the call pattern is such that they are not part of the performance critical part themsellves, then the performance critical part calls them through the regular conventions for calling Ruby from C.

If the call patter is such that they are part of the performance critical piece, well, I think the answer is obvious.

> It seems that an unanticipated replacement of some core functionality with a C library may involve a major rewrite

It might, but in the meantime you've got working code.

> and most Ruby teams may not have the expertise to do a good job maintaining a C code base any way.

If the team determines it needs expertise in a particular area that it doesn't currently have, then it should either develop that expertise or bring in people that have it. That's true whether its particular domain expertise (e.g., building messaging systems) or particular technology expertise (e.g., C). That's part of the normal development of a team.

Re: Web Framework Benchmarks Round 4

#207

Post by henk53 »

jbverschoor wrote: post #197
raylu wrote: post #192

Earlier quoted context omitted.

I agree with everything you've said here, but I'd like to add something about startup time. If it takes you 5s to start up your server, that's a lot of time you've added to each development iteration. Make a change, restart the server, wait 5s, see if it works/check debug output.

Ruby or Python are even worse (slow / big) than java these days. I'm a ruby guy btw..

Java is not that bad at all these days. For years now every iteration of Java EE has become lighter (with respect to the programming model and the startup time of servers).

Re: Web Framework Benchmarks Round 4

#209

Post by camus »

desireco42 wrote: post #194

I have one more question, I see a number of really fat frameworks on top, if rails is faster, then it must be because it is heavily optimized and people who made it are super smart, but for example codeigniter is above slim, kohana is above fuel, like twice the speed for Fortunes test. This isn't what I would expect at all, I would also expect Rails to be below php frameworks for speed alone. Did you use orm when it…

IMHO, .

it is stated in the benchmark when a framework test uses an ORM or "raw" pdo/whatever for db request.

Things like Doctrine are elegant and smart ,but let's face it,they are so slow. PHP is not JAVA. Hibernate may be fast on JAVA but Doctrine is hardly (fast)...

Symfony Laravele and Silex share the same http-kernel & event-dispatcher. Laravele and Silex however can use closures for controllers and filters/middleware, Maybe that's why there are faster.

Classes are expensive in PHP , since PHP is not OO centric and classes are merely a add-on. Bootstraping Symfony means creating an insane number of objects. There are things that could be done about it. I'm sure PHP frameworks are so slow because of the abuse of class hierarchy.

Re: Web Framework Benchmarks Round 4

#210

Post by oberhamsi »

ricardobeat wrote: post #191
oberhamsi wrote: post #161

I implemented the Ringo app for this benchmark and of course ran it against Node and a couple of others to see how we would perform in this neighborhood before I opened the pull request. And since that day I've been wondering: why does NodeJs (=V8 JS engine in C) talking to MongoDB have higher response times and latency than Ringo (=Rhino JS engine on JVM) talking to MySQL. The only thing where Node beats us JVM guys…

Node favours concurrency over raw speed; calls deferred with process.nextTick and callbacks end up costing time, in exchange for better concurrency. I think a blocking driver could leave Ringo in the dust, but it would be useless.

I will test that. Should the latency stay more constant with higher concurrency? Or what am I searching for?

Looking at the res/seq we got from round4. In order of concurrency (8, 16, 32, 64, 128, 256):

nodejs (mongodb raw)

12,541 22,073 26,761 26,461 28,316 28,856

ringojs (mysql raw)

13,190 23,556 27,758 30,697 31,382 31,997

both look like they got room to grow

Post reply on HN