Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Framework Benchmarks Round 4

techempower.com

Post reply on HN
Print view View original

251–260 of 358 posts

Re: Web Framework Benchmarks Round 4

#251

Post by apalmer »

dragonwriter wrote: post #182
zinxq wrote: post #178

There should be no surprise that interpreted, dynamic languages are utterly out-gunned as compared to compiled (JIT'd or otherwise) languages. It's inherent to the system - every little thing you do costs more. Many people choose Ruby and figure, given that premature optimization is the root of all evil, they'll optimize later if needed. That's like choosing between a farm tractor or a ferrari - and figuring if the t…

> Many people choose Ruby and figure, given that premature optimization is the root of all evil, they'll optimize later if needed. True. > That's like choosing between a farm tractor or a ferrari - and figuring if the tractor doesn't perform up to snuff, we'll add a spoiler Its really not like that at all, because programming languages aren't like vehicles. Particularly, with Ruby, on typical method of optimization i…

Seems a big part of this is is a lot of the proponents thought the comparison was really a Ford vs a Corvette, I dont think a lot of people were really internalizing exactly the ramifications of the order(s) of magnitude difference in performance. Which is why this kinda benchmark is pretty helpful.

Re: Web Framework Benchmarks Round 4

#252

Post by redtuesday »

fein wrote: post #35
showerst wrote: post #25

Earlier quoted context omitted.

Not that PHP is particularly fast, but I notice that Raw PHP does about middle of the pack, but all of the ORM versions seem to do terribly. Interesting stuff. I've usually had pretty good luck with Slim, I'll have to try a version with & without redbean and see how big a difference it makes.

If their FAQ's page is saying what I think it does, then they aren't testing with memcache or opcode caching. Also no love for Yii.

Someone mentioned the benchmark in the Yii forum a while ago [1], but it seems nobody wanted to submit a pull request for Yii.

[1] http://www.yiiframework.com/forum/index.php/topic/42168-fram...

Re: Web Framework Benchmarks Round 4

#253

Post by bhauer »

bilbo0s wrote: post #243

Wow... some of these tests are still pretty severely hobbled. Is there some reason that you use built in json serialization for some frameworks and not others? There is also a lot of heterogeneity in the implementation of the multiple queries test. For instance, even if I only look at... say... java frameworks, you seem to implement the exact same feature in very different ways between platforms. For instance, for se…

Hello bilbo0s.

Thanks for taking the time to dig in and provide some feedback. As much as possible, we want each test to be representative of idiomatic production-grade usage of the framework or platform. Furthermore, we have solicited contributions from fans of frameworks and the frameworks' authors. A side objective is that the code double as an example of how best to use the framework or platform.

All of this means we fully expect that the implementation approaches will vary significantly.

The multiple query test has a client-provided count of queries, so in most Java cases, we create a fixed-size array to hold the results fetched from the database. I wrote the Servlet and Gemini tests, so I can confirm that behavior in those tests.

We are not Vert.x experts and we have not yet received a community contribution for the Vert.x test. However, it is our understanding that idiomatic Vert.x usage encourages the use of asynchronous queries. The question then is: how do we collect the results into a single List in a threadsafe manner? Is your JsonArray alternative threadsafe? Admittedly, using a CopyOnWriteArrayList gave us pause, but we are not (yet) aware of a better alternative.

The Onion test was contributed by a reader and admittedly its compliance with the specification we've created is perhaps a bit dubious. We want a JSON serializer to process an in-memory object into JSON. I'm not certain if the Onion implementation matches that expectation, but the test implementation nevertheless seemed sufficiently idiomatic for his platform.

We're certainly open to more opinions on that matter.

Re: Web Framework Benchmarks Round 4

#254

Post by [deleted] »

cunac wrote: post #214
raylu wrote: post #192

Earlier quoted context omitted.

I agree with everything you've said here, but I'd like to add something about startup time. If it takes you 5s to start up your server, that's a lot of time you've added to each development iteration. Make a change, restart the server, wait 5s, see if it works/check debug output.

typically you don't stop/start server during development cycle, hot deployment works 99% of cases

[deleted]

Re: Web Framework Benchmarks Round 4

#255

Post by exabrial »

raylu wrote: post #192
peeters wrote: post #75

Earlier quoted context omitted.

Java gets that wrap from originally being slow to execute, and also having a huge up-front cost to spin up a VM. The first isn't true any more: the Java VM competes with native code on most benchmarks, and due to its ability to perform runtime optimizations, can occasionally outperform native code. The second doesn't matter at all for web servers. The cost of starting up the web server is tertiary to uptime and perfo…

I agree with everything you've said here, but I'd like to add something about startup time. If it takes you 5s to start up your server, that's a lot of time you've added to each development iteration. Make a change, restart the server, wait 5s, see if it works/check debug output.

If you're continuously restarting, you're doing it wrong. That argument became invalid about 4 years ago I think around the time Eclipse Helios was released.

Now-a-days, starting a Tomcat or TomEE JVM in debug mode with Eclipse gives you the ability to hot swap probably 95% of your changes. It doesn't supporting adding completely new functions or changing declared fields. JRebel does support this though.

As a matter of fact, if you're in a stack frame and you pause the execution pointer with a breakpoint, you can completely change the code of the function and the JVM will discard the current stack frame and then restart the functional call. Essentially, you can rewrite your code, while it's executing, without losing your stack.

Re: Web Framework Benchmarks Round 4

#256

Post by redtuesday »

qingu wrote: post #26

Thanks for doing these extensive benchmarking tests. It would be really helpful to see a more complex example that includes user authentication. Aside from the benchmarks it's also a really good starting point to compare the code in different languages and get a first impression of a framework. On a side-note, I'd really like to know why so few start-ups seem to be using Spring. It could be just a wrong impression .…

Maybe that's me, but I think it's easier to learn typical web frameworks like Rails, Django etc. than spring. On top of that the xml config sucks (at least in my opinion - though I used spring the last time around 2007, maybe it's not as bad as back then).

Re: Web Framework Benchmarks Round 4

#258

Post by redtuesday »

krg wrote: post #114
ihsw wrote: post #95

It's very depressing to see Symfony2 at the bottom of these lists. Although there are quite a few performance optimizations that can be done to improve this, there are few excuses for such poor performance by default.

If you have improvements to the test (that would be realistic for a production deployment) then please submit a pull request. We definitely want to show each framework doing the best it can do. A few frameworks have a "stripped" version (just Django and Rails so far) to try to show the best that can be achieved when typical functionality is stripped out. Essentially optimizing for this test, which is interesting even…

A stripped test is on the way it seems. One user submitted a raw test a few days ago which doesn't use doctrine.

Re: Web Framework Benchmarks Round 4

#259

Post by camus »

jakejake wrote: post #200
bhauer wrote: post #113

Earlier quoted context omitted.

Hi saosebastiao, As Pat points out, we definitely look forward to implementing some more computationally-intense request types in the future. This round does include the first server-side template test. We'd like to hear the community's opinions about more tests. That said, I feel most of the frameworks' implementations of the existing tests are not cheating. Our objective in this project is to measure every framewor…

I would like to pile my thanks onto this list as well. I'm the author of Phreeze and I can say that I'm grateful that fairness is being encouraged. There is certainly glory in ranking well on any benchmark and I have to admit, as I was implementing the tests in Phreeze, I saw many opportunities to "cheat." For example, skipping the framework routing, not using the "proper" way to communicate between the layers, etc a…

Can you share with us the tuning you did with class loading for instance ? thanks for your comment.

Re: Web Framework Benchmarks Round 4

#260

Post by rdtsc »

oberhamsi wrote: post #210
ricardobeat wrote: post #191

Earlier quoted context omitted.

Node favours concurrency over raw speed; calls deferred with process.nextTick and callbacks end up costing time, in exchange for better concurrency. I think a blocking driver could leave Ringo in the dust, but it would be useless.

I will test that. Should the latency stay more constant with higher concurrency? Or what am I searching for? Looking at the res/seq we got from round4. In order of concurrency (8, 16, 32, 64, 128, 256): nodejs (mongodb raw) 12,541 22,073 26,761 26,461 28,316 28,856 ringojs (mysql raw) 13,190 23,556 27,758 30,697 31,382 31,997 both look like they got room to grow

I think 256 is too low. It should at least start at 256 and go to 1024 perhaps. Then also try multiple CPUs 1 to 4 to see how it scales across.
Post reply on HN