High performance services using coroutines
1–10 of 16 posts
Re: High performance services using coroutines
#2Writing high performance server software is a whole other world from writing services. In the first, data copies can cause unacceptable slow-downs of a fraction of a millisecond - and those copies may be hidden all the way down at the driver level. When writing services (which are just as often written in Ruby or Python) trading off a few milliseconds for safety is often a worthy thing.
I spend most of my time writing services, and when looking at a language like Go, there's a reason the default is pass-by-value. Passing around pointers to multiple co-routines is asking for a race condition, and in non-GC languages, null pointers. Services are rarely written with the precision of high-performance servers.
I don't envy the server writers (although I can see how it would be fun!). Giving up a few milliseconds per request to make sure my co-routines aren't sharing pointers is worthwhile, and I appreciate the safety that gives me. I'm sure someone will mention that Rust could give me the safety I was looking for in a non-GC language, but that's the point, isn't it - that by being able to game the system, you can gain a few precious microseconds here and there that enforced safety might cost you.
Re: High performance services using coroutines
#3The end implementation sounds a lot like... Erlang. With the exception that this solution avoids allocations that Erlang probably makes due to immutability.
Re: High performance services using coroutines
#4Writing high performance server software is a whole other world from writing services. In the first, data copies can cause unacceptable slow-downs of a fraction of a millisecond - and those copies may be hidden all the way down at the driver level. When writing services (which are just as often written in Ruby or Python) trading off a few milliseconds for safety is often a worthy thing. I spend most of my time writin…
> I'm sure someone will mention that Rust could give me
> the safety I was looking for in a non-GC language, but
> that's the point, isn't it
Re: High performance services using coroutines
#5>> If not, the coroutine will yield, and at the same time, be scheduled(migrated) to another set of threads responsible for the ‘slow’ requests(ones that need to block). Those will just schedule the coro in their own scheduler, and perform the read again and continue.
>> Alternatively, we could just yield the coro, waiting for a new special coroutine that would run in a background thread just executing readahead() and then re-scheduling(making runnable) the original coro, waiting for readahead.
Seems to me this scheme will ultimately be limited by the slower request. Performing fast and slow operations is essentially Little's Law[1] where the average time dominates. However if the slow/blocking reads where also async, I think you'd eventually be limited by io speed?
Re: High performance services using coroutines
#6This sounds very similar to warp, the Haskell web server that is known for its high-performance on many-core machines. The Web Application Interface (WAI) is also coroutine based (I believe): http://www.aosabook.org/en/posa/warp.html
Re: High performance services using coroutines
#7This sounds very similar to warp, the Haskell web server that is known for its high-performance on many-core machines. The Web Application Interface (WAI) is also coroutine based (I believe): http://www.aosabook.org/en/posa/warp.html
Sorry for the shameless plug but I just released HTTP server based on Lua coroutines (https://github.com/raksoras/luaw) that pretty much follows the exact design from the OP article.
Re: High performance services using coroutines
#8The end implementation sounds a lot like... Erlang. With the exception that this solution avoids allocations that Erlang probably makes due to immutability.
This is a C++ implementation. Erlang and other high level languages use Processes(in the case of Erlang) or other such constructs to encapsulate logic and those are indeed scheduled based on either cooperative/preemptive fashion.
The main goals of this system is to minimize potential for blocking or otherwise slow processing steps which can stall processing of other requests, especially if this happens for many such requests/second.
Specifically, the 'optimistic sequential execution' idea is that you run the request on the request it was accepted in, use mincore() to determine if will _likely_ stall if you read from it, and if so, either migrate the coro to a 'slow' coros threads list, or just keep on going (which should be the fast-path here).
This is a solution to a specific class of problems. The similarities with Erlang's Process scheduling is that a process is a lot like a coroutine (it encapsulates logic and data and it's scheduled by a user-space scheduler, and that there can be thousands of them in runnable state, and each has its own stack). But that's really the definition of stackless coroutines.
Re: High performance services using coroutines
#9>> If not, the coroutine will yield, and at the same time, be scheduled(migrated) to another set of threads responsible for the ‘slow’ requests(ones that need to block). Those will just schedule the coro in their own scheduler, and perform the read again and continue. >> Alternatively, we could just yield the coro, waiting for a new special coroutine that would run in a background thread just executing readahead() an…
I am familiar with Little's Law. The key concept is optimistic sequential execution. The overwheleming majority of requests do not need to page in cold-data or otherwise stall for too long. Also, in the case where they can stall to e.g process a lot of data in a loop, they can yield back to the scheduler, thereby being fair to all other accepted coroutines/requests and giving a chance to others to complete earlier.
In practice, at least based on my prototype and tests, this works extremely well. If about 10-15% of the requests may end up blocking the thread, while the read can read kernel-cache resident data and proceed, that's really huge in terms of performance gains.
The background threads that either become owners and responsible for running slow/blocking coros can rely on a work-strealing scheme to keep them all busy, and if you don't rely on them, thereby removing contention/stall from the 'fast' threads, you are going to eventually suffer stalls and latency spikes.
Re: High performance services using coroutines
#10This sounds very similar to warp, the Haskell web server that is known for its high-performance on many-core machines. The Web Application Interface (WAI) is also coroutine based (I believe): http://www.aosabook.org/en/posa/warp.html
Sorry for the shameless plug but I just released HTTP server based on Lua coroutines ( https://github.com/raksoras/luaw ) that pretty much follows the exact design from the OP article.
This is pretty cool. Thanks for sharing.