Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Framework Benchmarks Round 4

techempower.com

Post reply on HN
Print view View original

321–330 of 358 posts

Re: Web Framework Benchmarks Round 4

#321

Post by tensor »

bjhoops1 wrote: post #246
axlerunner wrote: post #33

Earlier quoted context omitted.

I got on the Node thing for awhile too, but went back to Java/JBoss/Tomcat/Spring. If you are skilled with the Java stack it is hard to beat for performance and breadth. If you are not, well, I admit the learning curve is steep.

I definitely agree for large applications. But for quickly spinning up a few light service endpoints, Node can't be beat. Especially if you are using JSON-based persistence like MongoDB or CouchDB, using JSON all the way from database to the client is a huge win. I get tired of writing lots of JAXB POJOs to map my JSON objects to and from, especially early on in development when those definitions change rapidly. That…

You should look into clojure/compojure. You can get higher performance and write even less code.

Re: Web Framework Benchmarks Round 4

#322

Post by bilbo0s »

bhauer wrote: post #292
bilbo0s wrote: post #263

Earlier quoted context omitted.

"..The multiple query test has a client-provided count of queries, so in most Java cases, we create a fixed-size array to hold the results fetched from the database. I wrote the Servlet and Gemini tests, so I can confirm that behavior in those tests..." I agree, that approach would be best. I just was unsure why you didn't do it in Vert.x. "...it is our understanding that idiomatic Vert.x usage encourages the use of…

You're right, this is fascinating stuff. Like I said, we had not yet received a pull request for the Vert.x test, but presumably we will get one before Round 5? :) We had not previously understood that there was no need for thread-safe behavior within a Vert.x handler. Removing that (apparently fictional) requirement allows us to use just a simple array. Out of curiosity, can you point me to where you found confirmat…

Sorry (for the tardy response)... minor emergency came up.

Anyway, the relevant part of the manual is:

http://vertx.io/manual.html#concurrency

Also discovered you are running the equivalent of a single threaded environment anyway. (8 workers on an 8 core machine)

Any reason for that???

Maybe more confusion about Vert.x concurrency??? If so, here is the relevant documentation on worker verticles:

http://vertx.io/manual.html#worker-verticles

Long story short, you dedicated the entire machine (8 cores) to the equivalent of database connection management, (the persistor). Very little of the machine, (whatever is context switched in effectively), is dedicated to request handling. Try something a bit more fair... like 4 workers and 4 web request handlers.

Also, I was going through the Node.js tests and I had a question... do you guys do any clustering for the Node tests at all? Or are these results from the tests run on a single Node?

Sorry for all the questions, just want to make all of the tests do the same thing across all of the frameworks so that when you guys run it again, we can use that data here in a more meaningful fashion. For us, it is useful.

Re: Web Framework Benchmarks Round 4

#323

Post by rdtsc »

bhauer wrote: post #313
rdtsc wrote: post #260

Earlier quoted context omitted.

I think 256 is too low. It should at least start at 256 and go to 1024 perhaps. Then also try multiple CPUs 1 to 4 to see how it scales across.

Hi rdtsc, Until the project includes a WebSocket-enabled test or a test with forced idle time (e.g., waiting for an external service to provide a response), concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1]. Increasing the client-side concurrency level simply means that the front-end web server (or built-in web serve…

Hi bhauer,

> concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1].

Well websocket connections are becoming more and more popular. Maybe that's a different benchmark.

But the level of concurrency is pretty important. It basically tells the story of what happens to a "slashdotted" server. If nothing crazy like that happens than most servers might be ok, just maybe have a little higher latency. It is when shit hits the fan that different servers start separating from the herd. Some gracefully slow down, some scale smoothly across CPUs, some start throwing socket errors.

Who cares about these issues? Well anyone who becomes successful. If there are no visitors and no customers and only a GET request here and there every 10 minutes, then those places could really just use any server. A simple Perl or Ruby one will do. Now those that grow and see customers they will be interested in what happens in cases like that. There is a traffic spike at launch of new product so now there is a 200% increase in traffic for that one day and it tapers off.

Maybe we just come from a different background and that's why they focus is on different metrics.

> It doesn't make the server any faster at completing those requests.

But I am not sure what story does benchmarking the servers at an artificial level of concurrency tells us. Maybe it helps those that have a throttling/balancing proxy that always sets the number of connection to 256 at most and otherwise balances out the rest to other servers... And I am not sure if a the heuristic that "If it can handle 2456 requests/second with a single connection at a time" can be extrapolated and implies then it can "handle 2456 concurrent connections in a single second".

Re: Web Framework Benchmarks Round 4

#324

Post by pgsandstrom »

bhauer wrote: post #253
bilbo0s wrote: post #243

Wow... some of these tests are still pretty severely hobbled. Is there some reason that you use built in json serialization for some frameworks and not others? There is also a lot of heterogeneity in the implementation of the multiple queries test. For instance, even if I only look at... say... java frameworks, you seem to implement the exact same feature in very different ways between platforms. For instance, for se…

Hello bilbo0s. Thanks for taking the time to dig in and provide some feedback. As much as possible, we want each test to be representative of idiomatic production-grade usage of the framework or platform. Furthermore, we have solicited contributions from fans of frameworks and the frameworks' authors. A side objective is that the code double as an example of how best to use the framework or platform. All of this mean…

Concerning the CopyOnArrayList:

Unless the number of reads heavily outnumbers the numbers of writes, it is better to use something like Collections.synchronizedList(new ArrayList()) instead of CopyOnArrayList. The synchronized list does lock while reading or writing, but writing still becomes much faster.

Re: Web Framework Benchmarks Round 4

#325

Post by thrownaway2424 »

JulienSchmidt wrote: post #302
thrownaway2424 wrote: post #155

Earlier quoted context omitted.

Well, in the database access test with multiple queries, Go has an outrageous max latency of 19 seconds, and a stddev of 3x the mean, which is terrible.

This is probably caused by running all the queries in their own goroutine (if you don't know what this is just think of it as a thread - but much cheaper). This causes the queries to be handled in more or less random order. We'll fix this for Round 5 ;-)

You think so? I think it is probably cases by Go's single threaded stop the world garbage collector.

Re: Web Framework Benchmarks Round 4

#326

Post by chetzit »

ausjke wrote: post #304

Hold on, Symphony2 is that bad?? I use Drupal, which will use Symphony2, this sucks!

See, that's why these benchmarks do no good, people jump to quick conclusions. Also i suspect that it is not properly configured in OP case ( there is a ticket submitted https://github.com/TechEmpower/FrameworkBenchmarks/issues/23...). Also, Drupal will not use symfony2, it will use some of its components.

Re: Web Framework Benchmarks Round 4

#327

Post by diroussel »

mynegation wrote: post #247
tlarkworthy wrote: post #131

Earlier quoted context omitted.

"average and std dev are only revelent if the distribution is Gaussian in distrition" technically not true. Knowledge of the second order moment (variance) lets you uniquely identify other distributions like Poisson, or uniform. Knowledge of even higher order moments lets you fit more complicated statistical models. Low variance is good, regardless of underlying distribution.

Probably the statement should be that comparing mean and variance are only relevant if both metrics follow the same distribution. In the absence of distribution information (and it is usually absent in empirical tests like that) quantiles would help to do a better job at comparing performance.

Indeed. Often latency measurements are clumpy.

You might have one clump of fast responses when no GC occurs, another when some GC occurs, and a smaller clump where a stop-the-world full GC has occurred.

In such a case average is not meaningful.

Re: Web Framework Benchmarks Round 4

#328

Post by oberhamsi »

bjhoops1 wrote: post #242
oberhamsi wrote: post #47

Earlier quoted context omitted.

try our RingoJs: it's JS on the JVM. Scripting Java with JavaScript.

This, just in case noobs were not confused enough by Java/JavaScript? :)

confusing but accurate :)

The ability to script Java is one of Ringo's killer features - for this benchmark, for example, we dropped in two jars (JDBC myqlconnector & connection pooling from apachecommons) and glued them together with 10 LOC of JS.

Re: Web Framework Benchmarks Round 4

#329

Post by oberhamsi »

ricardobeat wrote: post #314
bhauer wrote: post #313

Earlier quoted context omitted.

Hi rdtsc, Until the project includes a WebSocket-enabled test or a test with forced idle time (e.g., waiting for an external service to provide a response), concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1]. Increasing the client-side concurrency level simply means that the front-end web server (or built-in web serve…

The point is benchmarking concurrency limits, not that it would improve raw performance.

ah! I didn't think of it like that. That would be interesting to see... at which concurrency level the environments crap out.

Re: Web Framework Benchmarks Round 4

#330

Post by diroussel »

bhauer wrote: post #122
diroussel wrote: post #93

Earlier quoted context omitted.

The requests per second is importanct, but some frameworks seem to get high average throughput but at the expense of a few slow requests. Also when measuring latency, average and std dev are only revelent if the distribution is guassian in distrition. Which is unlikely. Better to show percentile based measurements. Like 90% of all requests served in 5ms, and 99% of requests served in 15ms. See Gil Tene's talk "How no…

Hello diroussel, Thanks for the feedback! We started the project with WeigHTTP, then starting with Round 2 we switched to Wrk [1] at the advice of other readers. Wrk provides latency measurements consisting of average, standard deviation, and maximum. See the earlier conversation about standard deviation here: https://news.ycombinator.com/item?id=5455972 If we had distribution data available, we would aim to provide…

Ok, I've raise an enhancement for wrk https://github.com/wg/wrk/issues/31

Perhaps you could upvote or something?

Thanks for all the great work in these benchmarks. A useful resource.

Post reply on HN