OpenZFS: Reducing ARC Lock Contention
blog.delphix.com
OpenZFS: Reducing ARC Lock Contention
1–9 of 9 posts
Re: OpenZFS: Reducing ARC Lock Contention
#2I'm curious how much this affects the cache hit/miss rate. How slow do your disks have to be before this actually worsens performance?
Also, how big was the cache compared to the size of the test files => what was the probability of random cache hits to begin with?
Re: OpenZFS: Reducing ARC Lock Contention
#3While the single benchmark provided probably isn't statistically significant, I noticed that performance on system with 4 cores of less is worse with this patch.
If this is actually symptom of the patch, would it be possible to toggle lock behavior depending on available cores?
Re: OpenZFS: Reducing ARC Lock Contention
#4While the single benchmark provided probably isn't statistically significant, I noticed that performance on system with 4 cores of less is worse with this patch. If this is actually symptom of the patch, would it be possible to toggle lock behavior depending on available cores?
Currently, the number of sublists created is equal to the number of cores on the system.
While it'd be fairly trivial to extend the code to make it easy for an admin to define the number of sublists, I don't have any reason to believe that'd be a useful knob to export.
Re: OpenZFS: Reducing ARC Lock Contention
#5The batched log approach mentioned in the comments is also interesting. Possibly not directly applicable to cache page eviction policy where I expect the pressure to free up an old page to make space for data is more time-critical, but a useful model to keep in mind, alongside the probabilistic list sharding described in the article.
Re: OpenZFS: Reducing ARC Lock Contention
#6Is the difference between O(1) and O(n) where n=number of cores really that big?
Re: OpenZFS: Reducing ARC Lock Contention
#7The batched log approach mentioned in the comments is also interesting. Possibly not directly applicable to cache page eviction policy where I expect the pressure to free up an old page to make space for data is more time-critical, but a useful model to keep in mind, alongside the probabilistic list sharding described in the article.
That's certainly an interesting approach.
It's got a maximum latency problem though.
If for some reason the unlucky thread that acquires the lock cannot process the batch faster than other threads are adding.... well that ends poorly
Re: OpenZFS: Reducing ARC Lock Contention
#8The batched log approach mentioned in the comments is also interesting. Possibly not directly applicable to cache page eviction policy where I expect the pressure to free up an old page to make space for data is more time-critical, but a useful model to keep in mind, alongside the probabilistic list sharding described in the article.
That's certainly an interesting approach. It's got a maximum latency problem though. If for some reason the unlucky thread that acquires the lock cannot process the batch faster than other threads are adding.... well that ends poorly
That would require a write rate very atypical for a cache and easily mitigated, though largely an almost impossible scenario.
Re: OpenZFS: Reducing ARC Lock Contention
#9Is the difference between O(1) and O(n) where n=number of cores really that big?
Well, if the exact same semantics were required (i.e. always evict the oldest buffer), then it's really O(M*n) where M is the number of bytes to evict, and n being the number of sublists; each sublist would have to be checked for each buffer that evicted. But, I agree, if that was the only concern then the extra comparisons _probably_ wouldn't be enough to be concerned about.
The bigger issue is how to lock the sublists. It seems heavy handed to lock all sublists, find the oldest buffer, evict it, and then unlock all sublists. But if each is locked, checked, then unlocked; each sublists' oldest buffer can change while the eviction thread is still iterating the sublists to find the oldest buffer.
So, while the current solution is not very elegant, it is simple, requires minimal locking, and appears to work well enough.