51–60 of 172 posts
I enjoyed this article but I think the answer to the headline is obvious: parallelism
That's like saying "how to get to the moon is obvious: traveling"
Data gets split into redundant copies, and is rebalanced in response to hot spots.
Everything in this article is the obvious answer you'd expect.
So is any of S3 powered by SSD's? I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.
I always assumed the really slow tiers were tape.
Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.
I was an SDE on the S3 Index team 10 years ago, but I doubt much of the core stack has changed. S3 is comprised primarily of layers of Java-based web services. The hot path (object get / put / list) are all served by synchronous API servers - no queues or workers. It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career. For a get call,…
Wait; how does that work, when a user is PUTting tons of objects concurrently into a bucket, and then LISTing the bucket during that? If the PUTs are all hitting different indexing-cluster nodes, then...?
(Or do you mean that there are queues/workers, but only outside the hot path; with hot-path requests emitting events that then get chewed through async to do things like cross-shard bucket metadata replication?)
Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).
And remember that there's a multiplication of iops for any individual client iop, whether you're using triplicate storare or erasure coding. S3 also has iop multiplication, which they solve with tons of HDDs.
For big object storage that's mostly streaming 4MB chunks, this is no big deal. If you have tons of small random reads and writes across many keys or a single big key, that's when you need to make sure your backing store can keep up.
Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).
Doing some light googling aside from Ceph being listed, there's one called Gluster as well. Hypes itself as "using common off-the-shelf hardware you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks." It's open source / free to boot. I have no direct experience with it myself however. https://www.gluster.org/
I used to keep a large cluster array with Gluster+ZFS (1.5PB), and I can’t say I was ever really that impressed with the performance. That said — I really didn’t have enough horizontal scaling to make it worthwhile from a performance aspect. For us, it was mainly used to make a union file system.
But, I can’t say I’d recommend it for anything new.
Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.
So is any of S3 powered by SSD's? I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.
It's assumed that the new S3 Express One Zone is backed by SSDs but I believe Amazon doesn't say so explicitly.
Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.
Microservices for days. I worked on lifecycle ~5 years ago and just the Standard -> Glacier transition path involved no fewer than 7 microservices. Just determining which of the 400 trillion keys are eligible for a lifecycle action (comparing each object's metadata against the lifecycle policy on the bucket) is a massive big data job. Always was a fun oncall when some bucket added a lifecycle rule that queued 1PB+ of…
Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.
I was an SDE on the S3 Index team 10 years ago, but I doubt much of the core stack has changed. S3 is comprised primarily of layers of Java-based web services. The hot path (object get / put / list) are all served by synchronous API servers - no queues or workers. It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career. For a get call,…