Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Framework Benchmarks Round 4

techempower.com

Post reply on HN
Print view View original

291–300 of 358 posts

Re: Web Framework Benchmarks Round 4

#291

Post by apalmer »

voidlogic wrote: post #283
apalmer wrote: post #244

Earlier quoted context omitted.

ASP.Net kind of put themselves out of the benchmark game here: Mono Issue #1, since the vast vast majority of ASP.Net websites run on windows a Mono performance test even if accurate is going to be of dubious value. Mono Issue #2, since Mono is nowhere near as polished as the Microsoft .Net implementation the numbers wont really be meaningful. Windows Issue #1, if you do the test on a different OS than every other te…

Care to backup your Mono claims with evidence? I have a friend who went to a Mono talk at MS MIX where a Mono developer was speaking. The Mono developer said that while Mono is a little slower than .NET (and he was talking a couple percent) Mono often ends up being faster on the same hardware because the Linux system calls the runtime uses are faster. There were a few ROFLs in the audience. I also agree with you that…

I don't know that mono is guaranteed to be slower or not, but definitely a lot fewer man hours have gone into polishing mono as compared to the amount of effort spent to polish the .net framework on windows.

Further, even if Mono itself is as fast as the .net framework, IIS the web server is going to be totally different performance characteristics from whatever webserver you are using on linux.

I am glad you showed me the error of my ways because I would have guess .net was somewhat faster than mono, but it goes to show even more comparing across operating systems is meaningless...

now i would love to get my hands on the numbers dont get me wrong, just saying if i was running the project I wouldnt go through the amount of effort required to get the .net results.

Re: Web Framework Benchmarks Round 4

#292

Post by bhauer »

bilbo0s wrote: post #263
bhauer wrote: post #253

Earlier quoted context omitted.

Hello bilbo0s. Thanks for taking the time to dig in and provide some feedback. As much as possible, we want each test to be representative of idiomatic production-grade usage of the framework or platform. Furthermore, we have solicited contributions from fans of frameworks and the frameworks' authors. A side objective is that the code double as an example of how best to use the framework or platform. All of this mean…

"..The multiple query test has a client-provided count of queries, so in most Java cases, we create a fixed-size array to hold the results fetched from the database. I wrote the Servlet and Gemini tests, so I can confirm that behavior in those tests..." I agree, that approach would be best. I just was unsure why you didn't do it in Vert.x. "...it is our understanding that idiomatic Vert.x usage encourages the use of…

You're right, this is fascinating stuff. Like I said, we had not yet received a pull request for the Vert.x test, but presumably we will get one before Round 5? :)

We had not previously understood that there was no need for thread-safe behavior within a Vert.x handler. Removing that (apparently fictional) requirement allows us to use just a simple array. Out of curiosity, can you point me to where you found confirmation that handlers do not require thread safety?

Thanks again for your feedback!

Edit: spot checking Vert.x with a simple array does not appear to affect performance to a measurable degree.

Re: Web Framework Benchmarks Round 4

#293

Post by pekk »

candybar wrote: post #219
dragonwriter wrote: post #206

Earlier quoted context omitted.

> I keep hearing about this, but do people really rewrite performance-critical parts of their web apps in C? Certainly they do it for Ruby apps in general. I don't think its all that common for it to be a high-value proposition for web apps. > Even if it happens to be part of some third-party library? And maintain a fork? If its an open-source third-party library that tends to get used in a way that is performance-cr…

I guess my point is that it seems disingenuous to point out "you can write that bit in C" as a way to mitigate performance problems with Ruby, when in practice it's so costly compared to available alternatives (throw more hardware, write manually optimized Ruby, switch to a faster language/runtime) that almost no one does it. How much of Rails is written in C? It's like proposing compiler extensions/patches as a way…

If you cannot write one small part of the app in C due to the difficulty or time consumed, then how much better is it for you to write everything in Java from the beginning? Java does not really substitute for Ruby in the same niche.

Re: Web Framework Benchmarks Round 4

#294

Post by SkyMarshal »

krg wrote: post #185
SkyMarshal wrote: post #179

Earlier quoted context omitted.

Depends on whether any Lifters have optimized it for these tests or not, which, after skimming the github repo and commit history, does not appear to be the case. Same with any of the other frameworks - some have been optimized by their fans, others are running in default configs. Techempower should add a filter to show only frameworks that have been optimized.

At this point, most of the tests were contributed/improved by the community. Not all have been reviewed by experts in that framework, and I agree that an "expert reviewed" marker would be nice. That said, we have tried to not run anything in the "default" configs, but rather the "production deployment" configs if we could find documentation on that. Unfortunately there is a huge variability across frameworks in how g…

Have you guys reached out directly to the mailing lists/google groups/github issues/etc of all the frameworks you're testing, asking for experts to take a look?

I know some of these communities don't really frequent HN and Reddit, but they all frequent their mailing lists.

Re: Web Framework Benchmarks Round 4

#295

Post by bhauer »

pekk wrote: post #284
bhauer wrote: post #113

Earlier quoted context omitted.

Hi saosebastiao, As Pat points out, we definitely look forward to implementing some more computationally-intense request types in the future. This round does include the first server-side template test. We'd like to hear the community's opinions about more tests. That said, I feel most of the frameworks' implementations of the existing tests are not cheating. Our objective in this project is to measure every framewor…

The logic you have previously posted on HN for these benchmarks is that they measure the minimum overhead available on the platform, so that you cannot get faster than the benchmarked numbers. If a framework is too slow, the framework-chooser can exclude it from consideration for because the resulting project just can't be any faster than the framework benchmark. Sounds reasonable. Except now it is clear that you are…

Hi Pekk,

I'm not sure where you get the impression that we are refusing tuned tests (what we call "Stripped" tests). We have accepted two of those and would accept further tests of that nature. An implementation of course still needs to work and meet the obligations of the test scenario. For example, each row must be fetched from the database individually and the response must be serialized JSON. We did "reject" one test that fetched all 20 rows using a WHERE IN clause, but that implementation was quickly reconfigured by the submitter to match our specification.

We are expressly not including reverse proxy caches in these tests. We're not benchmarking the performance of the nginx proxy cache, Apache HTTPD's proxy cache, Varnish, or anything similar. You can find such benchmarks elsewhere. We are benchmarking the performance of the application framework for requests that do reach the application server. The tests are intended to be a viable minimum stand-in for application functionality in order to fulfill requests that, for whatever reason, reach your application server.

If the scenario is difficult to conceive, imagine your site cannot leverage a proxy cache because every request is providing private user information.

To be clear: none of the frameworks are being tested with a front-end cache.

Also presently, none of the tests use a back-end cache either, but future tests will include tests of back-end in-memory and near-memory caches.

Re: Web Framework Benchmarks Round 4

#296

Post by pkroll »

pfalls wrote: post #99
niggler wrote: post #88

How are multiple cores handled? For example, with node.js, the standard way to scale is to run multiple instances (but according to https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... only one instance is run). I wonder how the results would change if only one or two cores were enabled (using taskset or isolcpus)

We use the cluster module to handle multiple workers ( https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... ), so far we've received feedback that this is a valid approach.

For tests where you specify the number of cores, do you specify 4 or 8 on the i7?

Re: Web Framework Benchmarks Round 4

#297

Post by bhauer »

X4 wrote: post #290

I just wanted to say: Thank you! This is highly respected work (bookmarked) and deserves all the upvotes HN can give.

Thanks for the kind words, X4! With each round, we anticipate a spectrum of responses, but it's nevertheless a continuous surprise and honor to hear positive responses.

Re: Web Framework Benchmarks Round 4

#298

Post by dragonwriter »

candybar wrote: post #219
dragonwriter wrote: post #206

Earlier quoted context omitted.

> I keep hearing about this, but do people really rewrite performance-critical parts of their web apps in C? Certainly they do it for Ruby apps in general. I don't think its all that common for it to be a high-value proposition for web apps. > Even if it happens to be part of some third-party library? And maintain a fork? If its an open-source third-party library that tends to get used in a way that is performance-cr…

I guess my point is that it seems disingenuous to point out "you can write that bit in C" as a way to mitigate performance problems with Ruby, when in practice it's so costly compared to available alternatives (throw more hardware, write manually optimized Ruby, switch to a faster language/runtime) that almost no one does it. How much of Rails is written in C? It's like proposing compiler extensions/patches as a way…

> I guess my point is that it seems disingenuous to point out "you can write that bit in C" as a way to mitigate performance problems with Ruby, when in practice it's so costly compared to available alternatives

I don't think its costly compared to available alternatives; I think its generally an efficient alternative for the type of bottleneck that is actually related to implementation language efficieny. I think, for most typical web apps, the bottlenecks are only rarely of that type, so that's generally not where the effort is going to be spent, but for the ones that do have bottlenecks of that type, its quite appropriate a way of solving it.

> throw more hardware, write manually optimized Ruby, switch to a faster language/runtime

If writing manually optimized Ruby is an effective and cheaper solution, you aren't experiencing the class of bottlenecks that are related to implementation language efficiency. Switching languages or runtimes for a component is a proper subset of the work of switching languages or runtimes for a project, so the latter isn't going to be less costly than the former (it may, if language-related bottlenecks are pervasive, or if you have non-performance interests in the alternative language, have a bigger net payoff and be more cost effective, but it won't be less costly, and its inherently riskier to do all at once, since component-wise transition gives you a faster cycle time in terms of realizing value even if you end up doing a full replacement in the end.)

> And if you have a complex application that utilizes many of Ruby's idioms to deal with the complexity, it's extremely unlikely that you can simply replace parts of it with C libraries without reorganizing in such a way to increase complexity.

I disagree. Anything you can do in Ruby you can do in API-equivalent C that can still call out to the exact same Ruby code for the functions that aren't being moved into C, so there is no reason at all for the kind of reorganization you suggest, particularly if you are building with loosely-coupled components in the first place.

If you are building a complex app and its all tightly coupled, you've got a big maintainability nightmare no matter what language you're using, and that has nothing to do with Ruby.

Re: Web Framework Benchmarks Round 4

#299

Post by bhauer »

pkroll wrote: post #296
pfalls wrote: post #99

Earlier quoted context omitted.

We use the cluster module to handle multiple workers ( https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... ), so far we've received feedback that this is a valid approach.

For tests where you specify the number of cores, do you specify 4 or 8 on the i7?

8 since the i7 is four physical cores with hyper-threading.

Re: Web Framework Benchmarks Round 4

#300

Post by cies »

Could anyone explain: why is gemini sooo much faster then the others in these tests?

I believe that the way these test are setup slightly advantage gemini, and more broadly java. Since they do not measure memory usage, or tasks that make memory usage critical, which is something JVM sucks at.

Post reply on HN