111–120 of 241 posts
I find the lack of Erlang frameworks disappointing.
A note about Compojure: they're using Korma for database access. Korma is an abstraction layer that's more Clojure-friendly than using SQL directly, but lighter than an ORM. Using clojure.java.jdbc might produce higher performance.
Is anyone else alarmed to see raw PHP spanking Flask, Sinatra, and other Python/Ruby frameworks? If not, can anyone explain in layman's terms why that should be the case?
Why shouldn't that be the case? PHP has been designed from the get go to be super fast and 5.4 introduced a bunch of optimizations that have sped it up even more so. Ruby was designed for programmer happiness, PHP was designed for performance and getting sh*t done.
Earlier quoted context omitted.
It's great that you're doing this, and listing stuff like standard deviation in the tables -- but I'd say your focus/interpretation of the data isn't quite right. At least provide the option to sort by standard deviation -- as that might well be more interesting than requests/second? Maybe I'm just being mean because I was reminded of this essay by Zed Shaw earlier today (I was looking for his alluded rant on CC lice…
Thanks for the suggestion, we're still in the early stages of getting the latency information incorporated, and having the ability to sort by the various metrics makes a lot of sense.
Also, I like the "sportsmanlike benchmarking game between different communities" vibe I'm getting from all this.
Would be nice if the community helps turn this into the de facto example of how to benchmark correctly.
Now I'll just have to wait and see how Go 1.1 compares ;).
Is anyone else alarmed to see raw PHP spanking Flask, Sinatra, and other Python/Ruby frameworks? If not, can anyone explain in layman's terms why that should be the case?
> Flask I don't know about Sinatra or anything else but I can give you the reason why Flask out of the box performs like it does. Unlike PHP Flask is threadsafe and uses locks and thread or greenlet local storage. This level of indirection adds a basic performance penalty when executed on a single threaded server like gunicorn. There are ways to change that if you think that becomes a performance problem by replacing…
As an amateur hacker and after 275 days of reading Hacker News I feel I can now navigate in the sea of client side Javascript frameworks. With the introduction of this data on web framework performance, I now have another set of choices that I am completely unqualified to comprehend. My first impression is that this data shows about 3 levels of web frameworks. At the bottom (slowest) we have Django's and Rails and ma…
I find the lack of Erlang frameworks disappointing.
As an amateur hacker and after 275 days of reading Hacker News I feel I can now navigate in the sea of client side Javascript frameworks. With the introduction of this data on web framework performance, I now have another set of choices that I am completely unqualified to comprehend. My first impression is that this data shows about 3 levels of web frameworks. At the bottom (slowest) we have Django's and Rails and ma…
In general, I echo ConstantineXVI's comment. However, I'm also curious to know where the definite line is between the bottom half and the top 5 on the table. At what point does a team go "We're getting hammered and our backend is the problem. We have to switch to X or Y because our API on Z has reached its limit."
You add new servers and they get crushed. You scrutinize every line of code and fix algorithms and cache whatever is possible and you still can't keep up. You look at your full stack configuration and tune settings. You are afraid of growth because it will bring the site down. THEN you know it's time to consider switching frameworks. And even then I would try to find the core of the problem and rewrite that one piece.
Rewriting an app in a new framework can kill a company. Be leery of starting over. I personally know of one company who started over some 4+ years ago because the old app was too hard to maintain and only started rolling out the new app last year to extremely poor reception (even with about 1/2 the features of the old app). The reception was so bad that they had to stop rolling it out until it was fixed. They could have easily spent a year improving the old app and would be miles ahead.
That's not to say that all problems are due to the framework either. I've had web servers go unresponsive for 20+ minutes because apache went into the swap of death because KeepAlive was set to 15s and MaxClients was set to a value that would exceed available RAM. The quickest solution was to cycle the box. This was 10 years ago though and I think I had a total of 1GB of ram to work with.
Earlier quoted context omitted.
This is definitely something we want to include in the tests, but it is difficult to manage a fair way of monitoring. Outside of the actual benchmarks, where we kept everything as fair as possible, we ran individual tests to make sure they worked prior to the benchmarking and would routinely have htop running at the same time to get an idea of what that sort of data looks like. It IS very interesting and is definitel…
One idea that jumps to mind is putting the running frameworks in an LXC container (or similar) and monitoring the memory usage of that. Not sure how accurate that is, but it's one avenue. It also still might not be fair because some runtimes can let their heap get pretty big despite not actually needing the memory. You'd want a way to differentiate those from the ones with big heaps that would choke with some memory…
I think I said something similar on the first round, but for the Python tests, I'd be curious to see alternate runtimes and/or concurrency strategies: pypy, gevent, and probably the confluence of the two (though it requires quite a bit more hacking, since gevent doesn't officially support pypy).
Yes, I've submitted a pull request with these changes.