Does anyone know what is the technology stack of S3? Monolith or multiple services?
I assume would have lots of queues, caches and long running workers.
How AWS S3 serves 1 petabyte per second on top of slow HDDs
11–20 of 172 posts
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#12Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.
> conway’s law and how it shapes S3’s architecture (consisting of 300+ microservices)
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#13Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).
Lustre and ZFS can do similar speeds. However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.
Thanks, but I forgot to specify that I'm interested in S3-compatible servers only.
Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes.
So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast metadata store for Garage, the "special" device/small records store for the ZFS, the ZIL/SLOG device and so on.
Currently it's a bit of a Frankenstein's monster: using XFS+OpenCAS as the backing storage for an old version of MinIO (containerized to run as 5 instances), I'm looking to replace it with a simpler design and hopefully get a better performance.
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#14Earlier quoted context omitted.
Lustre and ZFS can do similar speeds. However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.
Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…
Do you know if some of these systems have components to periodically checksum the data at rest?
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#15Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.
Amazon biases towards Systems Oriented Architecture approach that is in the middle ground between monolith and microservices.
Biasing away from lots of small services in favour of larger ones that handle more of the work so that as much as possible you avoid the costs and latency of preparing, transmitting, receiving and processing requests.
I know S3 has changed since I was there nearly a decade ago, so this is outdated. Off the top of my head it used to be about a dozen main services at that time. A request to put an object would only touch a couple of services en route to disk, and similar on retrieval. There were a few services that handled fixity and data durability operations, the software on the storage servers themselves, and then stuff that maintained the mapping between object and storage.
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#16Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).
Doing some light googling aside from Ceph being listed, there's one called Gluster as well. Hypes itself as "using common off-the-shelf hardware you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks." It's open source / free to boot. I have no direct experience with it myself however. https://www.gluster.org/
I’ve used GlusterFS before because I was having tens of old PCs and it worked for me very well. It’s basically a PoC to see how it work than production though
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#17I enjoyed this article but I think the answer to the headline is obvious: parallelism
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#18[dead]
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#19Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).
SeaweedFS has evolved a lot the last few years, with RDMA support and EC.
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#20Earlier quoted context omitted.
Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…
Do you know if some of these systems have components to periodically checksum the data at rest?
ZFS/OpenZFS can do scrub and do block-level recovery. I'm not sure about Lustre, but since Petabyte sized storage is its natural habitat, there should be at least one way to handle that.