It is interesting to watch the latency numbers from the table(in a separate tab). I suppose latency is more important to most users than the peak throughput, considering that production applications almost never hit peak throughput and are often supplied with extra hardware before utilisation gets there.
Web Framework Benchmarks
21–30 of 116 posts
Re: Web Framework Benchmarks
#22I've been following these benchmarks for some time, and am always shocked that Spring does so poorly (it's 7% here). I haven't had any performance issues with Spring in production, so these benchmarks are puzzling. Are the other frameworks really that much faster in practice?
Yep I feel the same about Elixir and Phoenix. It's way down but again, the "way down" is 20k requests per second. Not too shabby. The biggest project I've worked on had 50 requests per second, and that was an E-Commerce site that brought in 9 figures a year in revenue.
I'm sure IoT workflows are mainly where you start to see these more insane RPS numbers. But still, this just tells me there's so much great choice out there for pretty much any platform you want to stick with.
I'd love to see another metric that normalizes between Cloud and Physical, like $ per request. We don't know how much the cloud server cost vs the physical, do we? I mean it's a 10x difference in performance, is it a 10x difference in price?
Update: Azure D2v3 instances (used for the cloud benchmark) are about $55 a month. ($660 a year).
Just the Xeon in the physical costs $1,500. The full server probably costs $5k? So assuming a server has a life of 3 years, you're looking at the Azure instance being $1,980, the Dell being maybe $5k? So it's about 10x performance for 3x the price.
Someone please fix my math and fill in the gaps :)
Re: Web Framework Benchmarks
#23Earlier quoted context omitted.
I'm not sure how many companies use C, C++ for Web development. In my opinion C/C++ wont be good idea to use for most. It is about choosing the best ration b/w tool speed and developer speed(productivity). There are very few companies which requires million per second request. For me Java, C#, Go, Node provide the best compromise b/w speed and productivity.
Vertx in Java is basically the fastest across all the benchmarks, putting it in the same speed/productivity bucket as node doesn’t make much sense...
I was talking about the compromise, if node is slower it provides better productivity.
Re: Web Framework Benchmarks
#24Earlier quoted context omitted.
> Anything other than C or Java really Rust, C#, and Go also seem to fit the bill, according to these benchmarks. C# particularly has rocketed up since the last time I looked in depth at these. Pretty impressive work from the .NET Core folks, I'd say.
For Rust, I'm interested to see what the future holds once the community coalesces around an async IO interface. Additionally, when procedural macros stabilizes (for real), the possibilities for ergonomic frameworks in Rust become endless. Rocket has been really interesting to work with, but it only works on nightly since it relies on experimental codegen/procedural macro features. Once procedural macros stabilizes,…
Speaking of which, a few hours ago Futures were added to the standard library, so the next nightly will have them in. We're quite close to that standardization!
Re: Web Framework Benchmarks
#25Is 'Fortunes' a test involving the quote generating program? Also, is the difference between 'Single Query' and 'Multiple Queries' concurrency? I always look at these but I have trouble understanding the test cases.
The best resource to answer your questions is the Requirements for the various test types [1]. But in brief: * Yes, Fortunes is named after the Unix tool of the same name. In our case, it's a test that executes a query of all rows in a table, adds an additional item, sorts the values in the application code, escapes the values as a XSS countermeasure, and then renders them using a server-side templating library. * Ye…
Thank you
Re: Web Framework Benchmarks
#26How different are the various frameworks here?
Are they all running with SSL? Do they keep track of session cookies? Do they have Content Security Policy headers? etc.. There's probably a million different feature configuration differences between each framework.
Re: Web Framework Benchmarks
#27I wrote the blog entry for this round [1], and I would agree with ksec (elsewhere in these comments) in recommending that readers start there since it includes the highlights of this round. I apologize ahead of time for my humor. It's still a bit early out here in California, but if anyone has any questions, I will try to answer as I can! [1] https://www.techempower.com/blog/2018/06/06/framework-benchm...
Do you have any plans to add memory usage to the benchmarks? It adds an extra dimension and sometimes even shows if there's a problem with a specific implementation (e.g. in the benchmark game it's often a hint for a program that can be optimized if the ratio compared to other languages is off).
Re: Web Framework Benchmarks
#28How different are the various frameworks here? Are they all running with SSL? Do they keep track of session cookies? Do they have Content Security Policy headers? etc.. There's probably a million different feature configuration differences between each framework.
> How different are the various frameworks here?
Quite different. We have included test implementations for frameworks that span 26 computer languages. There are an infinite variety of opinions about how to do things in computer programming in general, and our project is no exception.
And on the other hand, fairly similar. Tests should stick to the requirements [1], which are designed to be permissive but sufficiently clear on the expected work load of each test type. The principal goal is that test implementations should be realistic, and we will mark those the community believes are not realistic as "stripped" implementations.
> Are they all running with SSL?
No, not yet. But planned future test types would include SSL/TLS.
> Do they keep track of session cookies?
Generally no. These are intended to exercise anonymous requests. But a future test type could include session management.
> Do they have Content Security Policy headers?
No, that is not a requirement of our tests. We have specified which headers we expect. Others are optional.
> There's probably a million different feature configuration differences between each framework.
Yes. And it can be challenging at times to get all of these opinions to fit into the same box. For example, we've to-date kept SQLite implementations out of the project since those would not incur network costs. And other times the box has to be reshaped a bit to make room for new consensus opinions about what is suitable for "production." We recently made a decision to allow for innovative features in the Postgres protocol that are—in a manner of speaking—analogous to automatic pipelining. That conversation is still ongoing on our discussion forum.
Re: Web Framework Benchmarks
#29I wrote the blog entry for this round [1], and I would agree with ksec (elsewhere in these comments) in recommending that readers start there since it includes the highlights of this round. I apologize ahead of time for my humor. It's still a bit early out here in California, but if anyone has any questions, I will try to answer as I can! [1] https://www.techempower.com/blog/2018/06/06/framework-benchm...
Do you have any plans to add memory usage to the benchmarks? It adds an extra dimension and sometimes even shows if there's a problem with a specific implementation (e.g. in the benchmark game it's often a hint for a program that can be optimized if the ratio compared to other languages is off).
Yes! In fact, I just responded to a similar question over at the Rust Subreddit thread about the same topic.
We do in fact capture dstat data while executing the tests but as of today do not render these in any way. You can find raw CSV output from dstat at our logs server. For example at [1] are stats for Grizzly while measuring the "json" test type.
I will create an issue at the project's GitHub repo to begin a conversation about which stats to render. I'd like to select a meaningful bite-size set of stats that we can render into a table view. Basically another tab in the results view similar to the latency tab.
[1] http://tfb-logs.techempower.com/round-16/final/grizzly/json/...
Re: Web Framework Benchmarks
#30It looks like the bottleneck here is the database driver (or possibly ORM when used). The plaintext test shows a very different picture:
https://www.techempower.com/benchmarks/#section=#section=dat...
PS: As a side note, I do not like golang, my focus has been on python and now rust, but I am very impressed by both the plaintext and the DB test!