Regarding profiling, it should be possible to get some insights using perf I would have thought (e.g. perf-timechart and some fancy flamegraph that also takes off-cpu time into account)?
Guide to Lock Convoys
11–15 of 15 posts
Re: Guide to Lock Convoys
#12> Maybe we can’t eliminate them, but that doesn’t mean there’s nothing we can do about them. Specifically for the request case, I recall seeing a talk suggestion that processing requests last-in-first-out resolves this failure mode as well as reduces the 99th percentile latency. The intuition is if a request is going to be late, might as well abandon it and process it later because it's already late instead of doing…
I know this isn't any kind of silver bullet, but if you have these kinds of time constraints, wouldn't building your solution on a real time OS make more sense?
Re: Guide to Lock Convoys
#13Why does the author say throughput tanks in the their first convoy example? Yes requests are being queued (delayed) but the server is still doing the same amount of work per “snap”. It’s not really a traffic shockwave, but instead more like adding cars to the beginning of the road.
Re: Guide to Lock Convoys
#14Why does the author say throughput tanks in the their first convoy example? Yes requests are being queued (delayed) but the server is still doing the same amount of work per “snap”. It’s not really a traffic shockwave, but instead more like adding cars to the beginning of the road.
Yeah, that part confused me too; the latency goes up, but the throughput stays the same. It looks like it's just a bad example, because the later example under "Forming a Lock Convoy" shows better understanding of the problem's dynamics. A better initial example would be fast food or retail workers serving a single queue of customers.
Re: Guide to Lock Convoys
#15Sort of reminds me of this post on Netflix Tomcat tuning https://netflixtechblog.com/tuning-tomcat-for-a-high-through...
Requests spikes cause the whole system to backup where appropriate upper limit tuning allows the servers to drop requests before they get overloaded and grind to a halt