Earlier quoted context omitted.
More like * 2
Not for busy spinning threads. Source - spent a year performance tuning a matching engine.
Can you elaborate on why cores / 2 is the "ideal" number for throughput and latency?
Thread Pools in Nginx Boost Performance 9x (2015)
51–60 of 62 posts
Re: Thread Pools in Nginx Boost Performance 9x (2015)
#52It would be interesting to compare this with coroutines/fibers, and it would also be interesting to know how latency is affected. In my experience, thread pools introduce latency.
Trick is that nginx uses thread pools for non-blocking reading on the "fast" devices - you can't use coroutines/fibers, because on Linux is not reliably possible to read the regular file in a non-blocking way, as far as I'm aware, so that means that `read` in a single fiber would block entire thread until the data is fetched from the disk. In the storage engine I'm developing/maintaining, I've been using fibers, for…
You can move the the co-routine onto another thread. The go scheduler does this: https://morsmachine.dk/go-scheduler
There isn't really a "comparison" here. Co-routines/fibers could behave exactly like nginx does for IO or not at all. It all depends on the implementation.
Re: Thread Pools in Nginx Boost Performance 9x (2015)
#53It would be interesting to compare this with coroutines/fibers, and it would also be interesting to know how latency is affected. In my experience, thread pools introduce latency.
Trick is that nginx uses thread pools for non-blocking reading on the "fast" devices - you can't use coroutines/fibers, because on Linux is not reliably possible to read the regular file in a non-blocking way, as far as I'm aware, so that means that `read` in a single fiber would block entire thread until the data is fetched from the disk. In the storage engine I'm developing/maintaining, I've been using fibers, for…
A slightly off-topic question tinged with a touch of envy that you get to work on D.
Many HN readers are apprehensive of the use of D because it is garbage collected and conjecture that it is not appropriate for low latency and/or high throughput work loads. Correct me if I am wrong, but that is exactly the kind of load the sociomantic needs to address. I have thoroughly enjoyed the relevant Dconf videos but it would be great to hear some first hand account.
BTW is the move to D2 done ?
Re: Thread Pools in Nginx Boost Performance 9x (2015)
#54Earlier quoted context omitted.
Not for busy spinning threads. Source - spent a year performance tuning a matching engine.
Can you elaborate on why cores / 2 is the "ideal" number for throughput and latency?
Well HT is generally crap. In most cases if you go up to a random server, it is going to have HT. So cores / 2 will get you ballpark. You really should go in the bios and shut down HT if you are into latency at all. Then you can just do core or core - 2 threads.
So once you are there, the next step is to busy spin at a Thread Per Core (TPC). You have 10 cores, you find 10 (or more realistically 8-9 to leave the OS some spare cores to muck with) threads, and busy spin them at 100% doing work. You never let the linux scheduler touch them.
If this stuff interests you, there are some cool papers by lmax and the disruptor and gil tene that talk about this stuff..
The amount of work you can do with a single 10-20 core CPU is amazing.
Re: Thread Pools in Nginx Boost Performance 9x (2015)
#55Earlier quoted context omitted.
From what I can see, sendfile isn't really async, it still returns the number of bytes written or error status, and nginx probably needs to wait for that to e.g. log the results, handle short writes, or proceed with whatever next operation must be done to the socket (like closing it). No idea why aio_write needs offloading, though.
They say why in the article "The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks" They even go into how no interface has been surfaced yet that allows determining whether a file is in cache or not. Also how FreeBSD's aio interface doesn't have the same limitations
I never tried, but mmap and mincore could work (or not, as one of the many special cases in this area). I expect that's not a viable approach for this application, anyway :)
Re: Thread Pools in Nginx Boost Performance 9x (2015)
#56TLDR, the Linux non-blocking filesystem API requires you to bypass the OS file cache (!!!)
So the whole webserver serving clients might occasionally block waiting for disk if you're serving online videos or some other application where your whole website is too big to fit in the OS cache. But if you have multiple threads, then if one of them blocks, it isn't a big deal as the other threads will continue to unqueue requests.
> The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks
Re: Thread Pools in Nginx Boost Performance 9x (2015)
#57TLDR, the Linux non-blocking filesystem API requires you to bypass the OS file cache (!!!)
So the whole webserver serving clients might occasionally block waiting for disk if you're serving online videos or some other application where your whole website is too big to fit in the OS cache. But if you have multiple threads, then if one of them blocks, it isn't a big deal as the other threads will be able to take over.
> The asynchronous interface requires the O_DIRECT flag to be set on the file descriptor, which means that any access to the file will bypass the cache in memory and increase load on the hard disks
Re: Thread Pools in Nginx Boost Performance 9x (2015)
#58Re: Thread Pools in Nginx Boost Performance 9x (2015)
#59Earlier quoted context omitted.
Trick is that nginx uses thread pools for non-blocking reading on the "fast" devices - you can't use coroutines/fibers, because on Linux is not reliably possible to read the regular file in a non-blocking way, as far as I'm aware, so that means that `read` in a single fiber would block entire thread until the data is fetched from the disk. In the storage engine I'm developing/maintaining, I've been using fibers, for…
A slightly off-topic question tinged with a touch of envy that you get to work on D. Many HN readers are apprehensive of the use of D because it is garbage collected and conjecture that it is not appropriate for low latency and/or high throughput work loads. Correct me if I am wrong, but that is exactly the kind of load the sociomantic needs to address. I have thoroughly enjoyed the relevant Dconf videos but it would…
Oh, I'm aware this is a late reply, but it's better than nothing.
Yes, what you describe is exactly the kind of load we're addressing, and we're not getting GC to be in the way, simply by making usage of the reusable memory buffers, which are allocated in the first few requests, and then always reuse/recycled & fetched from the pool, so that we're not giving a chance for GC to kick in (as explained in the great blog post series, in D, GC mark&sweep will kick in only on allocations, completely deterministic: https://dlang.org/blog/category/gc/). Ocean provides some help here: https://github.com/sociomantic-tsunami/ocean/blob/v2.x.x/src... https://github.com/sociomantic-tsunami/ocean/blob/v2.x.x/src..., etc. This makes D completely suitable for these kinds of applications, and from my experience, none of D's features or the misfeatures is making such implementations hard.
D2 move is not yet completely done. We're still writing code that's compatible in both D1 and D2. For example, DHT node project (https://github.com/sociomantic-tsunami/dhtnode) and libraries on which it's depending on such as ocean (https://github.com/sociomantic-tsunami/ocean) and swarm (https://github.com/sociomantic-tsunami/swarm) are able to run in D2 with no performance penalties, but they still don't use D2-only constructs.
Re: Thread Pools in Nginx Boost Performance 9x (2015)
#60Earlier quoted context omitted.
Trick is that nginx uses thread pools for non-blocking reading on the "fast" devices - you can't use coroutines/fibers, because on Linux is not reliably possible to read the regular file in a non-blocking way, as far as I'm aware, so that means that `read` in a single fiber would block entire thread until the data is fetched from the disk. In the storage engine I'm developing/maintaining, I've been using fibers, for…
You can move the the co-routine onto another thread. The go scheduler does this: https://morsmachine.dk/go-scheduler There isn't really a "comparison" here. Co-routines/fibers could behave exactly like nginx does for IO or not at all. It all depends on the implementation.
Yes, sorry, I've missed the point that you can move them across the threads (was sidetracked by the usual notion of them running in the same thread).