Understanding the code inside Tornado, the asynchronous web server • Hacker News Board

Re: Understanding the code inside Tornado, the asynchronous web server

#11

Post by tptacek » Sat, May 07, 2011, 5:48 PM UTC

rdtsc wrote: post #5

justincormack wrote: post #3

Earlier quoted context omitted.

No the network sockets are set to non block and will return E_AGAIN once you have emptied the kernel buffer. Thats still non blocking. I dont think he is using aio_* which Linux does not really implement usefully.

No. They would return E_AGAIN after the data has been read or written in blocking mode. If select comes back and says "read is ready on socket 4" then user code does read on socket 4 and gets (say) 4K of data followed by E_AGAIN. Getting that 4K of data happens in a blocking mode. To make it non-blocking you would provide a pointer to your user space buffer to the kernel and it would determine when IO is ready _and_…

A process blocks when it needs to be rescheduled, yielding back to the process scheduler pending I/O. That's what "block" means. It doesn't mean "all the time consumed by any operation performed by the process". For instance, a process doesn't "block" when a hash table lookup unexpectedly trends O(n) due to collisions.

On a nonblocking socket, the I/O operations you're talking about are simple u/k and k/u buffer copies.

I think between aio_+ and the C10k page, you may have gotten a bit scrambled. Whether you have an event loop or not, disk I/O is often transparently blocking, even when you try to set descriptors nonblocking. But I/O operations on a nonblocking socket don't wait the process. If there's data in the buffer, you get the data; if there isn't, you get the error.

Re: Understanding the code inside Tornado, the asynchronous web server

#12

Post by ZeSmith » Sat, May 07, 2011, 5:54 PM UTC

Reminds me of this: http://teddziuba.com/2009/09/twisted-vs-tornado-youre-both.h...

Re: Understanding the code inside Tornado, the asynchronous web server

#13

Post by jessedhillon » Sat, May 07, 2011, 6:23 PM UTC

rdtsc wrote: post #5

justincormack wrote: post #3

Earlier quoted context omitted.

No the network sockets are set to non block and will return E_AGAIN once you have emptied the kernel buffer. Thats still non blocking. I dont think he is using aio_* which Linux does not really implement usefully.

No. They would return E_AGAIN after the data has been read or written in blocking mode. If select comes back and says "read is ready on socket 4" then user code does read on socket 4 and gets (say) 4K of data followed by E_AGAIN. Getting that 4K of data happens in a blocking mode. To make it non-blocking you would provide a pointer to your user space buffer to the kernel and it would determine when IO is ready _and_…

I think you are confusing the definitions of non-blocking IO and memory-mapped IO.

Re: Understanding the code inside Tornado, the asynchronous web server

#14

Post by jparise » Sat, May 07, 2011, 6:30 PM UTC

Some of this is a bit dated. It looks like it was written in late 2009, judging by the first comment. For example, IO loop timers are now stored using a heapq[1] instead of a sorted list, and many of the other issues have been similarly addressed over the last year and a half.

Still, this is a good general overview of Tornado's internals.

[1] https://github.com/facebook/tornado/commit/b6c4d6d20196fa4fe...

Re: Understanding the code inside Tornado, the asynchronous web server

#15

Post by ankrgyl » Sat, May 07, 2011, 6:48 PM UTC

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

Even with Python (CGI backed by Apache), you can scale with threads. The issue of scaling with threads vs. events is a pretty hot debate, and I think the author sets up this kind of criticism by addressing it poorly. Personally, I fall into the event-driven camp, because it involves the operating system as little as possible (only file descriptors). The C10K link has a great overview of the threaded approach (http://www.kegel.com/c10k.html#threaded). Read through the rest of the page for a discussion on the pros/cons of other approaches.

Re: Understanding the code inside Tornado, the asynchronous web server

#16

Post by bdarnell » Sat, May 07, 2011, 7:08 PM UTC

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

Agreed. If requests per second is the metric you're optimizing for, then threads are a perfectly fine solution (see e.g. Paul Tyma's presentations about how threads are superior to event-driven approaches in Java). The strongest case for event-driven servers is for cases like long polling, where a connection will sit idle for a minute or more and the number of connected idle clients becomes an important metric.

Re: Understanding the code inside Tornado, the asynchronous web server

#17

Post by bdarnell » Sat, May 07, 2011, 7:13 PM UTC

jparise wrote: post #14

Some of this is a bit dated. It looks like it was written in late 2009, judging by the first comment. For example, IO loop timers are now stored using a heapq[1] instead of a sorted list, and many of the other issues have been similarly addressed over the last year and a half. Still, this is a good general overview of Tornado's internals. [1] https://github.com/facebook/tornado/commit/b6c4d6d20196fa4fe...

The heapq change was just last week and has not yet been in any release, so that's a perfectly reasonable omission. :) But yes, this is about a pre-1.0 release, so I would encourage anyone interested in exploring tornado internals to check out a more recent release.

Re: Understanding the code inside Tornado, the asynchronous web server

#18

Post by rdtsc » Sat, May 07, 2011, 7:14 PM UTC

haberman wrote: post #6

rdtsc wrote: post #5

Earlier quoted context omitted.

No. They would return E_AGAIN after the data has been read or written in blocking mode. If select comes back and says "read is ready on socket 4" then user code does read on socket 4 and gets (say) 4K of data followed by E_AGAIN. Getting that 4K of data happens in a blocking mode. To make it non-blocking you would provide a pointer to your user space buffer to the kernel and it would determine when IO is ready _and_…

The way you are using the term "blocking" is non-standard and not that useful. You are calling the syscall "blocking" just because it performs its work inline (in this case, copying a kernel buffer into a user-space buffer). By this definition, every syscall is blocking, even aio_read() because even aio_read() performs some work inline (namely enqueing a read request). In common usage, an I/O operation is considered…

> You are calling the syscall "blocking" just because it performs its work inline.

Yes because there is _another_ mode of IO where it doesn't have to do that. So the reason that is called "blocking" is because there is another mode of operation (albeit it is exotic) where the process does not block while performing IO -- the kernel reads/writes the data on behalf of the process. If this second mode of performing IO did not ever exist you could just interchange arbitrarily asynchronous and non-blocking as synonyms.

Since you didn't bother reading the link I posted here is the basic matrix of IO operations:

                    Synchronous            Asynchronous

    Blocking         read/write           select/[e]poll

 
    Non-Blocking    O_NONBLOCK+E_AGAIN      aio_*

There has been talk and various attempt at implementing aio_* style IO for network sockets on Linux, but nothing good so far.

> Even aio_read() performs some work inline (namely enqueing a read request).

Enqueuing the read request is not the same as actually copying gigabytes or terabytes of data from disk where the actual work is performed.

Re: Understanding the code inside Tornado, the asynchronous web server

#19

Post by rdtsc » Sat, May 07, 2011, 7:33 PM UTC

tptacek wrote: post #11

rdtsc wrote: post #5

Earlier quoted context omitted.

No. They would return E_AGAIN after the data has been read or written in blocking mode. If select comes back and says "read is ready on socket 4" then user code does read on socket 4 and gets (say) 4K of data followed by E_AGAIN. Getting that 4K of data happens in a blocking mode. To make it non-blocking you would provide a pointer to your user space buffer to the kernel and it would determine when IO is ready _and_…

A process blocks when it needs to be rescheduled, yielding back to the process scheduler pending I/O. That's what "block" means. It doesn't mean "all the time consumed by any operation performed by the process". For instance, a process doesn't "block" when a hash table lookup unexpectedly trends O(n) due to collisions. On a nonblocking socket, the I/O operations you're talking about are simple u/k and k/u buffer copi…

> On a nonblocking socket, the I/O operations you're talking about are simple u/k and k/u buffer copies.

But because there is an IO mode where these copies are done by the kernel when data arrives not by the user process (after a kernel notification), there is now a differentiation between blocking and asynchronous. Asynchronous refers to IO readiness notifications, "blocking" refers to the copying of data (the actual IO if you will).

> A process blocks when it needs to be rescheduled, yielding back to the process scheduler pending I/O. That's what "block" means.

However when your process is copying data it is in the running state and preventing other processes from running. It could be doing something else or it let other processes run in the meantime.

> The I/O operations you're talking about are simple u/k and k/u buffer copies.

That is true, however if the data comes in very small chunks very fast you are doing a lot of switching to user space and a lot of small context switches to, say copy 1K of data. When you could just request that the kernel fill up your 10MB buffer with data from a socket and tell you when it is ready. If there is anything I learned is to never just say "it is a simple copy". Today's memory is not very fast compared to CPU speeds and copying is not something to be taking lightly. It is one thing when looking at a toy example, another thing when dealing with realtime systems or large data sets.

Re: Understanding the code inside Tornado, the asynchronous web server

#20

Post by rdtsc » Sat, May 07, 2011, 7:36 PM UTC

jessedhillon wrote: post #13

rdtsc wrote: post #5

Earlier quoted context omitted.

No. They would return E_AGAIN after the data has been read or written in blocking mode. If select comes back and says "read is ready on socket 4" then user code does read on socket 4 and gets (say) 4K of data followed by E_AGAIN. Getting that 4K of data happens in a blocking mode. To make it non-blocking you would provide a pointer to your user space buffer to the kernel and it would determine when IO is ready _and_…

I think you are confusing the definitions of non-blocking IO and memory-mapped IO.

I am not talking about in-kernel hardware mechanisms like DMA and such. Or about memory mapping a file. It is about the system call interface to perform IO. Yeah, I guess you can call it memory-mapped but I don't think that is quite accurate in this context.