Web Framework Benchmarks Round 4 • Hacker News Board

Re: Web Framework Benchmarks Round 4

#311

Post by bhauer » Fri, May 03, 2013, 2:17 AM UTC

egeozcan wrote: post #269

bhauer wrote: post #133

Earlier quoted context omitted.

Voidlogic is right that we are waiting to get a pull request that will include some .Net frameworks [1]. If you can help, it would be greatly appreciated. We do want to include .Net. We will test on Mono to start. We also want to test on .Net's native Windows platform. But we need to work on the testing platform we've built in order to automate a Windows server in the same way we presently automate a Linux server. [1…

Nancy[1] comes to mind. [1]: http://nancyfx.org/

I like the looks of that! Could I convince you to put together a test as a pull request? :)

Re: Web Framework Benchmarks Round 4

#312

Post by kbenson » Fri, May 03, 2013, 2:52 AM UTC

laumars wrote: post #287

kbenson wrote: post #282

Earlier quoted context omitted.

> Yeah, the post started slightly different than it ended, and that's an artifact of that change. My next paragraph listed many more, and I tried to do multiple from each language that I looked at which had simple implementation.s > Particularly when the guts of the code in all 3 of those examples was Perl's standard database interface (the same DBI you'd use for CGI Perl or even standalone .pl scripts). The benchmar…

> The benchmark page clearly tags which implementations use raw SQL access and which use an ORM. These all happen to be using raw SQL. To my knowledge, none of them have a pre-bundled ORM, and I'm not sure whether the ORM tested implementations are only supposed to indicate the pre-shipped ORM. You miss my point. All of those examples you gave used the same core database framework and as the test was primarily a data…

> You miss my point. All of those examples you gave used the same core database framework and as the test was primarily a database performance test, all those 3 examples were essentially the same core Perl code.

I think we are talking past each other. I listed a lot of frameworks, including three in python. I started with Perl, and added a whole bunch more. I could, and should, have presented them better.

Personally I think the fact they are using DBI is the inconsequential part. It takes up few lines of the example, and most of the other code is the specifics of the framework (although they are very similar, because they all Sinatra clones, to varying degrees). What do you expect to be different in a non-DB based test (I'm still unclear what point you are trying to make)? Their template systems are pretty simple to use as well.

> Again, you missed my point. I wasn't suggestion that you were comparing the performance of the two. I was commenting on why I switched away from Perl to Go.

That's fine, and a worthy conversation to have, I'm just trying to keep this on the topic of implementation size, since I think the performance side of the discussion is being handled well enough elsewhere.

> Except you didn't You gave AN example (singular). It was one language; Perl.

Actually I gave eight examples, three Perl, three Python, three lua and 1 Ruby. The fact there were three Perl implementations first, and listed by themselves is sort of an accident. I was really interested in how Mojolicious did, since that's my favorite at the moment, and then I checked the other Perl implementations, and then I looked for others that might be good examples. I intended for them to be taken all together, even if that's not how it seemed.

> With Perl, you have a number of different ways you can hook the runtimes into the web server (CGI, Apache libs, etc), pure Perl and C libraries ...

> he ironic thing with Perl is despite scripts in the language being some of the most portable code on the POSIX community, running performance critical Perl webapps leads to very unportable set ups.

How recent is the data this opinion is based on? My understanding is that now most (new) Perl web projects are using PSGI as a common back-end making it extremely portable, and often using pure-perl servers for performance. There's some evidence they can significantly beat mod_perl2[1].

> This might sound critical, but I genuinely do love Perl. I'd say it was up there as one of my favourite languages (and over the years I've learn to develop in a great number of different languages). But sadly nothing in life is perfect.

I was really, _really_ trying to not make it a Perl vs Go thing. It's obvious I do have a preference though. I'm glad you like Perl, it does seem to fit the mindset of certain people well, and even if they don't stick with it, they remember it fondly. :)

[1]: http://old.nabble.com/mod_perl2-vs-Starman-and-other-pure-pe...

Re: Web Framework Benchmarks Round 4

#313

Post by bhauer » Fri, May 03, 2013, 2:58 AM UTC

rdtsc wrote: post #260

oberhamsi wrote: post #210

Earlier quoted context omitted.

I will test that. Should the latency stay more constant with higher concurrency? Or what am I searching for? Looking at the res/seq we got from round4. In order of concurrency (8, 16, 32, 64, 128, 256): nodejs (mongodb raw) 12,541 22,073 26,761 26,461 28,316 28,856 ringojs (mysql raw) 13,190 23,556 27,758 30,697 31,382 31,997 both look like they got room to grow

I think 256 is too low. It should at least start at 256 and go to 1024 perhaps. Then also try multiple CPUs 1 to 4 to see how it scales across.

Hi rdtsc,

Until the project includes a WebSocket-enabled test or a test with forced idle time (e.g., waiting for an external service to provide a response), concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1].

Increasing the client-side concurrency level simply means that the front-end web server (or built-in web server's socket listener thread) needs to maintain a small queue of requests to hand off to the application server's worker threads. It doesn't make the server any faster at completing those requests. I've written some more about this at my personal blog [2].

[1] Caveat: Some frameworks appear to have locking or resource contention issues and do not saturate the CPU cores. We will attempt to capture CPU utilization stats in future rounds since this might be of interest to readers and framework maintainers. But increasing concurrency would not increase CPU utilization in these scenarios either.

[2] http://tiamat.tsotech.com/rps-vs-connections

Re: Web Framework Benchmarks Round 4

#314

Post by ricardobeat » Fri, May 03, 2013, 3:28 AM UTC

bhauer wrote: post #313

rdtsc wrote: post #260

Earlier quoted context omitted.

I think 256 is too low. It should at least start at 256 and go to 1024 perhaps. Then also try multiple CPUs 1 to 4 to see how it scales across.

Hi rdtsc, Until the project includes a WebSocket-enabled test or a test with forced idle time (e.g., waiting for an external service to provide a response), concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1]. Increasing the client-side concurrency level simply means that the front-end web server (or built-in web serve…

The point is benchmarking concurrency limits, not that it would improve raw performance.

Re: Web Framework Benchmarks Round 4

#315

Post by dorfsmay » Fri, May 03, 2013, 3:29 AM UTC

pfalls wrote: post #53

manulp wrote: post #46

Any chance of seing the python frameworks behind uwsgi instead of gunicorn?

Here is the issue for just that: https://github.com/TechEmpower/FrameworkBenchmarks/issues/11... If you're interested, we'd love to have some help getting this accomplished.

I look at that, I don't understand what the issue is?

Can you explain. I might be able to help, I've got some experience with uwsgi + bottle.

Re: Web Framework Benchmarks Round 4

#316

Post by kbenson » Fri, May 03, 2013, 3:39 AM UTC

cmircea wrote: post #241

EGreg wrote: post #215

Earlier quoted context omitted.

I agree, but what about the special case of hitting multiple shards and aggregating the results? Shouldn't the non-blocking win over the blocking?

Well depends exactly on the implementation. Some may issue queries in parallel and aggregate the results, blocking until everything is done. Others may run them sequentially, which is the simplest but slowest way.

Or if your data supports it, you shard based on an algorithm that is repeatable and cheap, and the client can compute where to look for data, if it exists.

Re: Web Framework Benchmarks Round 4

#317

Post by steveklabnik » Fri, May 03, 2013, 3:56 AM UTC

candybar wrote: post #219

dragonwriter wrote: post #206

Earlier quoted context omitted.

> I keep hearing about this, but do people really rewrite performance-critical parts of their web apps in C? Certainly they do it for Ruby apps in general. I don't think its all that common for it to be a high-value proposition for web apps. > Even if it happens to be part of some third-party library? And maintain a fork? If its an open-source third-party library that tends to get used in a way that is performance-cr…

I guess my point is that it seems disingenuous to point out "you can write that bit in C" as a way to mitigate performance problems with Ruby, when in practice it's so costly compared to available alternatives (throw more hardware, write manually optimized Ruby, switch to a faster language/runtime) that almost no one does it. How much of Rails is written in C? It's like proposing compiler extensions/patches as a way…

> How much of Rails is written in C?

None, on purpose. We want maximum portability, and so the Rails defaults are Ruby-only on purpose. Of course, it's easy to add gems that replace things that are written in C or Java, depending on what makes the most sense for your platform.

Re: Web Framework Benchmarks Round 4

#318

Post by bhauer » Fri, May 03, 2013, 4:01 AM UTC

ricardobeat wrote: post #314

bhauer wrote: post #313

Earlier quoted context omitted.

Hi rdtsc, Until the project includes a WebSocket-enabled test or a test with forced idle time (e.g., waiting for an external service to provide a response), concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1]. Increasing the client-side concurrency level simply means that the front-end web server (or built-in web serve…

The point is benchmarking concurrency limits, not that it would improve raw performance.

Ah, okay. That's interesting, I suppose. You basically are interested in seeing at what concurrency level the server starts spitting back 500-series responses (or simply doesn't provide a response). Basically, how many concurrent requests are needed before the server's inbound request queue overflows.

We could test that out at some point.

Re: Web Framework Benchmarks Round 4

#319

Post by carbocation » Fri, May 03, 2013, 4:02 AM UTC

nickpresta wrote: post #307

carbocation wrote: post #306

Earlier quoted context omitted.

Gorilla is a toolkit, not a framework. It's nice, but it doesn't do all that much beyond the standard library. To me, Go+Gorilla vs raw Go is not similar to, say, a PHP framework vs raw PHP.

(I know what Gorilla is, I've contributed to it :D) The OP, to which I was replying, said: "see the gorilla test which is a Go framework." I asked about the Gorilla benchmark itself, since OP seemed to say that adding something like Gorilla would slow down the Go benchmarks, which I don't agree with.

I accidentally replied to the wrong comment; this was intended to be a reply to the parent of your post.

But I also didn't realize that that person was saying that there was a gorilla test, so now I'm mostly confused.

Re: Web Framework Benchmarks Round 4

#320

Post by meric » Fri, May 03, 2013, 4:25 AM UTC

justincormack wrote: post #271

zinxq wrote: post #178

There should be no surprise that interpreted, dynamic languages are utterly out-gunned as compared to compiled (JIT'd or otherwise) languages. It's inherent to the system - every little thing you do costs more. Many people choose Ruby and figure, given that premature optimization is the root of all evil, they'll optimize later if needed. That's like choosing between a farm tractor or a ferrari - and figuring if the t…

Openresty is doing pretty well with Lua and despite the fact it uses LuaJIT it is actually interpreted and the JIT compiler is not used (that will change eventually). Ruby is just slow, as is PHP.

Can't wait to see Openresty benchmark with LuaJIT.