Understanding the code inside Tornado, the asynchronous web server • Hacker News Board

Re: Understanding the code inside Tornado, the asynchronous web server

#21

Post by tptacek » Sat, May 07, 2011, 7:39 PM UTC

rdtsc wrote: post #19

tptacek wrote: post #11

Earlier quoted context omitted.

A process blocks when it needs to be rescheduled, yielding back to the process scheduler pending I/O. That's what "block" means. It doesn't mean "all the time consumed by any operation performed by the process". For instance, a process doesn't "block" when a hash table lookup unexpectedly trends O(n) due to collisions. On a nonblocking socket, the I/O operations you're talking about are simple u/k and k/u buffer copi…

> On a nonblocking socket, the I/O operations you're talking about are simple u/k and k/u buffer copies. But because there is an IO mode where these copies are done by the kernel when data arrives not by the user process (after a kernel notification), there is now a differentiation between blocking and asynchronous. Asynchronous refers to IO readiness notifications, "blocking" refers to the copying of data (the actua…

I don't believe that "blocking" refers to the copying of data. I don't believe that a process can be "blocking" and "running". Where are you getting this from?

Re: Understanding the code inside Tornado, the asynchronous web server

#22

Post by tptacek » Sat, May 07, 2011, 7:44 PM UTC

rdtsc wrote: post #18

haberman wrote: post #6

Earlier quoted context omitted.

The way you are using the term "blocking" is non-standard and not that useful. You are calling the syscall "blocking" just because it performs its work inline (in this case, copying a kernel buffer into a user-space buffer). By this definition, every syscall is blocking, even aio_read() because even aio_read() performs some work inline (namely enqueing a read request). In common usage, an I/O operation is considered…

> You are calling the syscall "blocking" just because it performs its work inline. Yes because there is _another_ mode of IO where it doesn't have to do that. So the reason that is called "blocking" is because there is another mode of operation (albeit it is exotic) where the process does not block while performing IO -- the kernel reads/writes the data on behalf of the process. If this second mode of performing IO d…

I think the DeveloperWorks article you've cited is simply wrong, and that's sent you into a tailspin. Select and poll aren't "ways to implement asynchronous blocking I/O".

I think you'll find these links more helpful than the misleading one you've been using.

http://www.circlemud.org/~jelson/software/fusd/docs/node36.h...

http://people.freebsd.org/~hmp/stuff/docs/freebsd_kse.pdf

I could always be wrong about this stuff; maybe I'm the one with the broken semantics. But I'm pretty sure I'm not.

Re: Understanding the code inside Tornado, the asynchronous web server

#23

Post by tptacek » Sat, May 07, 2011, 7:49 PM UTC

rdtsc wrote: post #20

jessedhillon wrote: post #13

Earlier quoted context omitted.

I think you are confusing the definitions of non-blocking IO and memory-mapped IO.

I am not talking about in-kernel hardware mechanisms like DMA and such. Or about memory mapping a file. It is about the system call interface to perform IO. Yeah, I guess you can call it memory-mapped but I don't think that is quite accurate in this context.

The system call interface to perform I/O cannot be described as "memory mapped"; the read/write system calls are programmed I/O. The opposite of memory mapped. You're letting terminology get you into trouble. You should just let go of your definitions and make the broader point you're trying to make, which might still be salvageable.

Re: Understanding the code inside Tornado, the asynchronous web server

#24

Post by haberman » Sat, May 07, 2011, 8:01 PM UTC

rdtsc wrote: post #18

haberman wrote: post #6

Earlier quoted context omitted.

The way you are using the term "blocking" is non-standard and not that useful. You are calling the syscall "blocking" just because it performs its work inline (in this case, copying a kernel buffer into a user-space buffer). By this definition, every syscall is blocking, even aio_read() because even aio_read() performs some work inline (namely enqueing a read request). In common usage, an I/O operation is considered…

> You are calling the syscall "blocking" just because it performs its work inline. Yes because there is _another_ mode of IO where it doesn't have to do that. So the reason that is called "blocking" is because there is another mode of operation (albeit it is exotic) where the process does not block while performing IO -- the kernel reads/writes the data on behalf of the process. If this second mode of performing IO d…

> Since you didn't bother reading the link I posted here is the basic matrix of IO operations.

Actually I did. You are misinterpreting what it is saying. The reason it is calling the select() model "blocking" is because you block during the select(), not because the read() itself is blocking.

This is also non-standard usage: most people wouldn't refer to a select()-based loop as blocking I/O, because a program generally only calls select() when it has nothing else to do but service I/O, so the select() does not "block" the application.

Re: Understanding the code inside Tornado, the asynchronous web server

#25

Post by cdavid » Sat, May 07, 2011, 8:31 PM UTC

ankrgyl wrote: post #15

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

Even with Python (CGI backed by Apache), you can scale with threads. The issue of scaling with threads vs. events is a pretty hot debate, and I think the author sets up this kind of criticism by addressing it poorly. Personally, I fall into the event-driven camp, because it involves the operating system as little as possible (only file descriptors). The C10K link has a great overview of the threaded approach ( http:/…

Another issue with evented frameworks in python is that you have to ensure nothing is blocking anywhere in your code, which becomes harder the more complex your application becomes. People will often mention IO, database, etc... forgetting that it is also an issue if your request handler takes CPU for N ms (e.g. encoding a relatively large payload in json, etc...). everything needs to be written with async in mind.

Languages/frameworks where async IO is an implementation detail (like some web frameworks in haskell) don't have this issue.

Re: Understanding the code inside Tornado, the asynchronous web server

#26

Post by ZoFreX » Sat, May 07, 2011, 9:06 PM UTC

ankrgyl wrote: post #15

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

Even with Python (CGI backed by Apache), you can scale with threads. The issue of scaling with threads vs. events is a pretty hot debate, and I think the author sets up this kind of criticism by addressing it poorly. Personally, I fall into the event-driven camp, because it involves the operating system as little as possible (only file descriptors). The C10K link has a great overview of the threaded approach ( http:/…

I'm not entirely in either camp, I don't think event-driven programming is worse or even bad. I'm just sick of false arguments in either direction! (Especially as a Java developer who tried to learn nio only to find out that using lots of threads was fine all along... not a pleasant journey!)

Re: Understanding the code inside Tornado, the asynchronous web server

#27

Post by ZoFreX » Sat, May 07, 2011, 9:09 PM UTC

wildmXranat wrote: post #9

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

I agree with you mostly, and I think there is a misunderstanding of the terms. It's possible to serve large amounts of requests with threads as well as with event loops, but there is a distinction between the two on how they consume resources. The author is wrong in assuming threads will not scale, as they will, but a Java web framework using threads will eat up hundreds of megs under heavy load, while an event loop…

There are resource issues either way, in both cases you have to store state. With threads this can quickly become expensive if you keep the stack size for all those threads at the default amount (a whopping 48K if memory serves). I would say it's easier to use smaller amounts of memory with event-driven programming, but you still have to keep some data around per-client. Of course, if the situation is very asynchronous (seconds instead of milliseconds) then you don't necessarily need all of this hanging around in RAM.

Re: Understanding the code inside Tornado, the asynchronous web server

#28

Post by thejbf » Sat, May 07, 2011, 9:31 PM UTC

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

Yes, but I can still spawn thousands of event-based Scala actors on the same machine. The actual problem with author: He believes running long and costly operations on an HTTP request is OK which is never and ever right. If you need to wait 5 seconds to perform a task, make it asynchronous.

Re: Understanding the code inside Tornado, the asynchronous web server

#29

Post by barrkel » Sat, May 07, 2011, 10:51 PM UTC

ZoFreX wrote: post #8

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands…

Something I'd like people to understand, at a very deep level: event-based async programming, and blocking thread-based programming, are fundamentally the same. The problem of maintaining your state in between event triggers is explicit in the first style; it's implicit, in the thread stack and CPU instruction pointer, in the second style. In the second style, it's the OS which is running the event loop, and it dispatches events by resuming a continuation - a continuation that starts by "returning" from the blocking call.

Much of the translation from blocking style to event-based style is moving the work of dispatching and looping from kernel to userland. Other ancillary benefits, like reduced address space usage by blocked threads, are in principle also achievable in a threading model - e.g. by storing stack frames on the heap and being more aggressive about collecting them (assuming GC).

Other benefits of async - such as overlapping work - are also fairly trivially possible with threading, though less deterministic.

Re: Understanding the code inside Tornado, the asynchronous web server

#30

Post by barrkel » Sat, May 07, 2011, 11:00 PM UTC

rdtsc wrote: post #19

tptacek wrote: post #11

Earlier quoted context omitted.

A process blocks when it needs to be rescheduled, yielding back to the process scheduler pending I/O. That's what "block" means. It doesn't mean "all the time consumed by any operation performed by the process". For instance, a process doesn't "block" when a hash table lookup unexpectedly trends O(n) due to collisions. On a nonblocking socket, the I/O operations you're talking about are simple u/k and k/u buffer copi…

> On a nonblocking socket, the I/O operations you're talking about are simple u/k and k/u buffer copies. But because there is an IO mode where these copies are done by the kernel when data arrives not by the user process (after a kernel notification), there is now a differentiation between blocking and asynchronous. Asynchronous refers to IO readiness notifications, "blocking" refers to the copying of data (the actua…

A blocked process is a process which is not in a running or ready state in the kernel. The userland process entering the kernel on its stack isn't sufficient to change this state; nor is starting or finishing a memory copy operation, particularly one which is done on the CPU (things might be different if DMA were involved). Blocked, running, ready etc. are process scheduling concepts and are about sharing a limited resource, specifically the CPU. A process busy copying memory, but using the CPU, is not blocked, because another process may not be running on that CPU instead.