Achieveing lower latencies with S3 object storage
1–10 of 24 posts
Re: Achieveing lower latencies with S3 object storage
#2> Roughly speaking, the latency of systems like object storage tend to have a lognormal distribution
I would dig into that. This might (or might not) be something you can do something about more directly.
That's not really an "organic" pattern, so I'd guess some retry/routing/robustness mechanism is not working the way it should. And, it might (or might not) be one you have control over and can fix.
To dig in, I might look at what's going on at the packet/ack level.
Re: Achieveing lower latencies with S3 object storage
#3The article posts a table of latency distributions, but the latencies are simulated based on the assumption that latencies are lognormal. I would be interested to read the article comparing the simulation to actual measurements.
The assumption that latencies are lognormal is a useful approximation but not really true. In reality you will see a lot of multi-modality (e.g. cache hits vs misses, internal timeouts). Requests for the same key can have correlated latency.
Re: Achieveing lower latencies with S3 object storage
#4The hedging strategies all seem to assume that latency for an object is an independent variable.
However, I would assume dependency?
Eg. if. a node holding a copy of the object is down and traffic needs to be re-routed to a slower node. Indifferently of how many requests I send, the latency will still be high?
(I am genuinly curious of this is the case)
Re: Achieveing lower latencies with S3 object storage
#5> Roughly speaking, the latency of systems like object storage tend to have a lognormal distribution I would dig into that. This might (or might not) be something you can do something about more directly. That's not really an "organic" pattern, so I'd guess some retry/routing/robustness mechanism is not working the way it should. And, it might (or might not) be one you have control over and can fix. To dig in, I migh…
I don't know what you mean by the word "organic", but I think lognormal distributions are very common and intuitive: whenever the true generative mechanism is “lots of tiny, independent percentage effects piling up,” you’ll see a log‑normal pattern.
Re: Achieveing lower latencies with S3 object storage
#6> Roughly speaking, the latency of systems like object storage tend to have a lognormal distribution I would dig into that. This might (or might not) be something you can do something about more directly. That's not really an "organic" pattern, so I'd guess some retry/routing/robustness mechanism is not working the way it should. And, it might (or might not) be one you have control over and can fix. To dig in, I migh…
How do you do that for an abstract service like S3? I see how you could do that for your own machines.
Re: Achieveing lower latencies with S3 object storage
#7The hedging strategies all seem to assume that latency for an object is an independent variable. However, I would assume dependency? Eg. if. a node holding a copy of the object is down and traffic needs to be re-routed to a slower node. Indifferently of how many requests I send, the latency will still be high? (I am genuinly curious of this is the case)
It’s not addressed directly but I do think the article implies you hope your request latencies are not correlated. It provides a strategy for helping to achieve that
> Try different endpoints. Depending on your setup, you may be able to hit different servers serving the same data. The less infrastructure they share with each other, the more likely it is that their latency won’t correlate.
Re: Achieveing lower latencies with S3 object storage
#8Lots of areas left for exploration.
Re: Achieveing lower latencies with S3 object storage
#9S3 is a bad choice if you need low latency to begin with.
Re: Achieveing lower latencies with S3 object storage
#10It is kinda of crazy how much work is done to mitigate the very high latency of S3 when we have NVMe SSDs with access latency of microseconds.