Techempower Web Framework Benchmarks Round 10

The main benefit is this allows rough comparison to a ton of other frameworks. Just running ab against your one server setup gives you one RPS/latency result on one hardware setup - that's good to know as an absolute metric, but tells you very little about your performance relative to other frameworks.

This project gives you RPS/latency metrics for many frameworks, on a few hardware setups. This enables a rough comparison of "how does my framework perform relative to all these other well-known or established frameworks". Naturally, the comparison is not perfect - there are a ton of reasons that measuring just requests/sec and latency doesn't allow complete comparison between two frameworks. However, once you accept that it is basically impossible to fully compare any two frameworks using just quantitative methods and these numbers should inform your choice of framework (instead of totally control your choice of framework), we can talk about why it's valuable.

Want to run a low-cost server in language X that you happen to love? This project can provide guidance about which frameworks written in language X are performing the best. Want to ensure your service can support 50k requests per second without loosing latency? This project can provide latency numbers for you to examine that let you know which frameworks appear to maintain acceptable latency even under high load.

If you wanted to, you could re-create this project by running ab against 100+ frameworks - that's the cornerstone of what is happening here. Granted, we currently use https://github.com/wg/wrk instead of ab, but the principle is the same - start up framework, run load generation, capture result data. Most of the codebase is dedicated to ensuring that these 100+ frameworks don't interfere with each other, setting up pseudo-production environments with separate server/database/load generation servers, and other concerns that have to be addressed.

Over time, this project has started collect more statistics than just requests/second and latency, which makes it more valuable than just running ab. As more metrics are added and more frameworks are added, this becomes a really valuable project for understanding how frameworks perform relative to one another.

Or PyParallel, when they include Windows in the next round: https://speakerdeck.com/trent/pyparallel-pycon-2015-language...

Consistently orders of magnitude faster than everything else out there in the Python landscape.

An error occurs, which is logged to stderr, but the benchmark logs don't capture stderr so it's hard to know what's happening. (Or maybe Play redirects stderr to a log file?)

The test passes in the preview runs, in the TechEmpower continuous integration tests and in the EC2 tests so it's probably some transient error that only occurred in the final bare metal test. Maybe there's a race condition in the Play 2 test scripts which only shows up sometimes.

I've spent a fair bit of time maintaining the Play 2 benchmark tests so it's very frustrating to get no result on the final test. Oh well!