S3 is a bad choice if you need low latency to begin with.
Network-based storage is a bad choice if you need low latency, period. You’re not going to beat data locality.
Achieveing lower latencies with S3 object storage
21–24 of 24 posts
Re: Achieveing lower latencies with S3 object storage
#22The article posts a table of latency distributions, but the latencies are simulated based on the assumption that latencies are lognormal. I would be interested to read the article comparing the simulation to actual measurements. The assumption that latencies are lognormal is a useful approximation but not really true. In reality you will see a lot of multi-modality (e.g. cache hits vs misses, internal timeouts). Requ…
I think the distribution he uses is pretty close to the paper he links "Exploiting Cloud Object Storage for High-Performance Analytics" https://www.durner.dev/app/media/papers/anyblob-vldb23.pdf
Re: Achieveing lower latencies with S3 object storage
#23The very first sentence of this article contains an error: > Over the past 19 years (S3 was launched on March 14th 2006, as the first public AWS service), object storage has become the gold standard for storing large amounts of data in the cloud. While it’s true that S3 is the gold standard, it was not the first AWS service, which was in fact SQS in 2004.
author here - took that quote from this[1] blog post by an AWS VP/distinguished engineer, the use of "public service" might have some loosely defined meaning in this context.
[1] https://www.allthingsdistributed.com/2025/03/in-s3-simplicit...
Re: Achieveing lower latencies with S3 object storage
#24The very first sentence of this article contains an error: > Over the past 19 years (S3 was launched on March 14th 2006, as the first public AWS service), object storage has become the gold standard for storing large amounts of data in the cloud. While it’s true that S3 is the gold standard, it was not the first AWS service, which was in fact SQS in 2004.
author here - took that quote from this[1] blog post by an AWS VP/distinguished engineer, the use of "public service" might have some loosely defined meaning in this context. [1] https://www.allthingsdistributed.com/2025/03/in-s3-simplicit...
Interesting source - looks like it means “GA” service, rather than “public” per se. The SQS beta was also available to the public.