Understanding the code inside Tornado, the asynchronous web server • Hacker News Board

Re: Understanding the code inside Tornado, the asynchronous web server

#31

Post by _delirium » Sat, May 07, 2011, 11:30 PM UTC

barrkel wrote: post #29

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

Something I'd like people to understand, at a very deep level: event-based async programming, and blocking thread-based programming, are fundamentally the same. The problem of maintaining your state in between event triggers is explicit in the first style; it's implicit, in the thread stack and CPU instruction pointer, in the second style. In the second style, it's the OS which is running the event loop, and it dispa…

That's essentially the thesis of this 1978 paper, though they group evented programming under the broader "message-passing" style: http://www.sics.se/~adam/pt/duality78.pdf

Re: Understanding the code inside Tornado, the asynchronous web server

#32

Post by barrkel » Sat, May 07, 2011, 11:44 PM UTC

_delirium wrote: post #31

barrkel wrote: post #29

Earlier quoted context omitted.

Something I'd like people to understand, at a very deep level: event-based async programming, and blocking thread-based programming, are fundamentally the same. The problem of maintaining your state in between event triggers is explicit in the first style; it's implicit, in the thread stack and CPU instruction pointer, in the second style. In the second style, it's the OS which is running the event loop, and it dispa…

That's essentially the thesis of this 1978 paper, though they group evented programming under the broader "message-passing" style: http://www.sics.se/~adam/pt/duality78.pdf

Indeed. And the corollary is that every possible performance benefit of non-blocking code is in principle achievable with threaded code, though some of the scheduler may need to be implemented in userland, with more explicit hinting of things like lifetimes of local variables in stack frames, etc., to get there. The true benefit of event-oriented network programming may be simply in what it makes explicit vs implicit. By making the tracking of state awkward, it forces its minimization and consolidation. By a process of Worse is Better, it ends up being more performant for scaling despite (usually) being more awkward to program (especially if you have any kind of nesting or recursion structure in your protocol).

Re: Understanding the code inside Tornado, the asynchronous web server

#33

Post by rdtsc » Sun, May 08, 2011, 3:13 AM UTC

tptacek wrote: post #21

rdtsc wrote: post #19

Earlier quoted context omitted.

> On a nonblocking socket, the I/O operations you're talking about are simple u/k and k/u buffer copies. But because there is an IO mode where these copies are done by the kernel when data arrives not by the user process (after a kernel notification), there is now a differentiation between blocking and asynchronous. Asynchronous refers to IO readiness notifications, "blocking" refers to the copying of data (the actua…

I don't believe that "blocking" refers to the copying of data. I don't believe that a process can be "blocking" and "running". Where are you getting this from?

I guess "blocking" is overloaded here. To me it refers to the process of performing IO, but you think about process states from the point of view of the scheduler (as in RUNNING state vs BLOCKING (or IOWAIT)).

Yes, if aio_* system was never invented, we wouldn't be arguing, but because it exists it introduces a new possible way of doing IO. It is a general enough way of doing IO and at hardware level we have DMA but in the land of syscalls we have aio_.

Ok, just curious, what words would you use to describe what aio_ calls do?

Over the years many have thought of a way to bring that style of IO to networking. So far it works for disk IO very well. For example take a look at this benchmark from lighttpd for a sendfile:

http://blog.lighttpd.net/articles/2006/11/09/async-io-on-lin...

It looks like there is a consistent 50% improvement in speed when using aio with a 1MB block.

Wouldn't it be nice to have that kind of improvement for network sockets as well? I think it would be and there have been many attempts over the years but nothing good yet has happened.

Re: Understanding the code inside Tornado, the asynchronous web server

#34

Post by rdtsc » Sun, May 08, 2011, 3:17 AM UTC

tptacek wrote: post #23

rdtsc wrote: post #20

Earlier quoted context omitted.

I am not talking about in-kernel hardware mechanisms like DMA and such. Or about memory mapping a file. It is about the system call interface to perform IO. Yeah, I guess you can call it memory-mapped but I don't think that is quite accurate in this context.

The system call interface to perform I/O cannot be described as "memory mapped"; the read/write system calls are programmed I/O. The opposite of memory mapped. You're letting terminology get you into trouble. You should just let go of your definitions and make the broader point you're trying to make, which might still be salvageable.

I agree, "memory mapped" is the wrong term. I was just letting the original poster who called it this way keep that as a concession or a mental model. That was my mistake as it just confuses everyone even more.

Re: Understanding the code inside Tornado, the asynchronous web server

#35

Post by jrockway » Sun, May 08, 2011, 3:33 AM UTC

ankrgyl wrote: post #15

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

Even with Python (CGI backed by Apache), you can scale with threads. The issue of scaling with threads vs. events is a pretty hot debate, and I think the author sets up this kind of criticism by addressing it poorly. Personally, I fall into the event-driven camp, because it involves the operating system as little as possible (only file descriptors). The C10K link has a great overview of the threaded approach ( http:/…

Well, actually, it involves the operating system rather heavily. select, epoll, kqueue, etc., etc., are not user-space things, after all, and if the OS implementation is bad, you are going to run into trouble.

Re: Understanding the code inside Tornado, the asynchronous web server

#36

Post by jrockway » Sun, May 08, 2011, 3:35 AM UTC

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

But I'll bet that a node.js HTTP server handling 10,000 slow downloads is going to use a lot less memory than 10,000 Java+OS threads handling those same downloads.

Re: Understanding the code inside Tornado, the asynchronous web server

#37

Post by _iv38 » Sun, May 08, 2011, 3:37 AM UTC

There was a very good talk about this at #nodeconf this week by Tom from Joyent. The basic summary was that threaded network servers, like Apache, say, approach the speed problem by "pre-allocating" resources, a chunk for each thread/instance. The issue is that there are a finite number of resources on each machine (RAM mostly), and blocked threads are eating up a slice of those resources even when doing no work. This sets the upper bound of number of simultaneous connections that can be handled.

In an event based system the overhead for each connection is at least three orders of magnitude lower, sometimes four or five (hope I'm remembering this right). This translates into _considerable_ increase in number of connections that can be handled simultaneously, not quite the equivalent number of orders of magnitude due to other aspects of the system becoming more of the bottle-necks, but still dramatic.

This was a data-driven observation, not conjecture or assertion. I listened in during a break while the nodejs guys argued about approaches to getting TLS working better - they were worrying about the 1MB overhead for a TLS connection because that's a significant percentage of a connection handler. Think about that for a minute in the context of an apache threaded instance. 1MB matters? wow!

I'm new to this area and this was very interesting stuff and seemed related to the various discussions below about a threaded system can do what an event based system can do.

Re: Understanding the code inside Tornado, the asynchronous web server

#38

Post by tptacek » Sun, May 08, 2011, 4:58 AM UTC

rdtsc wrote: post #33

tptacek wrote: post #21

Earlier quoted context omitted.

I don't believe that "blocking" refers to the copying of data. I don't believe that a process can be "blocking" and "running". Where are you getting this from?

I guess "blocking" is overloaded here. To me it refers to the process of performing IO, but you think about process states from the point of view of the scheduler (as in RUNNING state vs BLOCKING (or IOWAIT)). Yes, if aio_* system was never invented, we wouldn't be arguing, but because it exists it introduces a new possible way of doing IO. It is a general enough way of doing IO and at hardware level we have DMA but…

It's not overloaded. Respectfully, you simply don't understand what it means.

Re: Understanding the code inside Tornado, the asynchronous web server

#39

Post by ntoshev » Sun, May 08, 2011, 6:32 AM UTC

cdavid wrote: post #25

ankrgyl wrote: post #15

Earlier quoted context omitted.

Even with Python (CGI backed by Apache), you can scale with threads. The issue of scaling with threads vs. events is a pretty hot debate, and I think the author sets up this kind of criticism by addressing it poorly. Personally, I fall into the event-driven camp, because it involves the operating system as little as possible (only file descriptors). The C10K link has a great overview of the threaded approach ( http:/…

Another issue with evented frameworks in python is that you have to ensure nothing is blocking anywhere in your code, which becomes harder the more complex your application becomes. People will often mention IO, database, etc... forgetting that it is also an issue if your request handler takes CPU for N ms (e.g. encoding a relatively large payload in json, etc...). everything needs to be written with async in mind. L…

Tornado devs recommend using blocking mysql drivers and disk I/O. These operations have relatively low and predictable latencies. You can run more Tornado processes than the number of cores is if CPU is underutilized.

Re: Understanding the code inside Tornado, the asynchronous web server

#40

Post by FooBarWidget » Sun, May 08, 2011, 9:23 AM UTC

rdtsc wrote: post #33

tptacek wrote: post #21

Earlier quoted context omitted.

I don't believe that "blocking" refers to the copying of data. I don't believe that a process can be "blocking" and "running". Where are you getting this from?

I guess "blocking" is overloaded here. To me it refers to the process of performing IO, but you think about process states from the point of view of the scheduler (as in RUNNING state vs BLOCKING (or IOWAIT)). Yes, if aio_* system was never invented, we wouldn't be arguing, but because it exists it introduces a new possible way of doing IO. It is a general enough way of doing IO and at hardware level we have DMA but…

It would be good to have more I/O API calls available that don't require as much copying but so far there simply aren't any such APIs that are mature and stable. POSIX AIO is only supported in a very limited way; on Linux it only works for files, not sockets. splice() only works between files/pipes and TCP sockets but not Unix domain sockets. Etc etc.