Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AWS S3 serves 1 petabyte per second on top of slow HDDs

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

21–30 of 172 posts

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#21

Post by bayindirh »

crabique wrote: post #13
bayindirh wrote: post #8

Earlier quoted context omitted.

Lustre and ZFS can do similar speeds. However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.

Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…

It might not be the most ideal solution, but did you consider installing TrueNAS on that thing?

TrueNAS can handle the OpenZFS (zRAID, Caches and Logs) part and you can deploy Garage or any other S3 gateway on top of it.

It can be an interesting experiment, and 80 disk server is not too big for a TrueNAS installation.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#22

Post by hnexamazon »

wg0 wrote: post #11

Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.

I was an SDE on the S3 Index team 10 years ago, but I doubt much of the core stack has changed.

S3 is comprised primarily of layers of Java-based web services. The hot path (object get / put / list) are all served by synchronous API servers - no queues or workers. It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career.

For a get call, you first hit a fleet of front-end HTTP API servers behind a set of load balancers. Partitioning is based on the key name prefixes, although I hear they’ve done work to decouple that recently. Your request is then sent to the Indexing fleet to find the mapping of your key name to an internal storage id. This is returned to the front end layer, which then calls the storage layer with the id to get the actual bits. It is a very straightforward multi-layer distributed system design for serving synchronous API responses at massive scale.

The only novel bit is all the backend communication uses a home-grown stripped-down HTTP variant, called STUMPY if I recall. It was a dumb idea to not just use HTTP but the service is ancient and originally built back when principal engineers were allowed to YOLO their own frameworks and protocols so now they are stuck with it. They might have done the massive lift to replace STUMPY with HTTP since my time.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#23

Post by js4ever »

hnexamazon wrote: post #22
wg0 wrote: post #11

Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.

I was an SDE on the S3 Index team 10 years ago, but I doubt much of the core stack has changed. S3 is comprised primarily of layers of Java-based web services. The hot path (object get / put / list) are all served by synchronous API servers - no queues or workers. It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career. For a get call,…

"It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career."

can you give some numbers? or at least ballpark?

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#24

Post by creiht »

crabique wrote: post #13
bayindirh wrote: post #8

Earlier quoted context omitted.

Lustre and ZFS can do similar speeds. However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.

Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…

It is probably worth noting that most of the listed storage systems (including S3) are designed to scale not only in hard drives, but horizontally across many servers in a distributed system. They really are not optimized for a single storage node use case. There are also other things to consider that can limit performance, like what does the storage back plane look like for those 80 HDDs, and how much throughput can you effectively push through that. Then there is the network connectivity that will also be a limiting factor.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#26

Post by hobs »

MDGeist wrote: post #10
nerdjon wrote: post #9

So is any of S3 powered by SSD's? I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.

I always assumed the really slow tiers were tape.

Not even the higher tiers of Glacier were tape afaict (at least when it was first created), just the observation that hard drives are much bigger than you can reasonably access in useful time.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#27

Post by hnexamazon »

js4ever wrote: post #23
hnexamazon wrote: post #22

Earlier quoted context omitted.

I was an SDE on the S3 Index team 10 years ago, but I doubt much of the core stack has changed. S3 is comprised primarily of layers of Java-based web services. The hot path (object get / put / list) are all served by synchronous API servers - no queues or workers. It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career. For a get call,…

"It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career." can you give some numbers? or at least ballpark?

Tens of thousands of TPS per node.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#28

Post by gostsamo »

littlesnugblood wrote: post #25
EwanToo wrote: post #2

I think a more interesting article on S3 is "Building and operating a pretty big storage system called S3" https://www.allthingsdistributed.com/2023/07/building-and-op...

[flagged]

Can you share some anecdotes?

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#29

Post by epistasis »

giancarlostoro wrote: post #5
crabique wrote: post #3

Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).

Doing some light googling aside from Ceph being listed, there's one called Gluster as well. Hypes itself as "using common off-the-shelf hardware you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks." It's open source / free to boot. I have no direct experience with it myself however. https://www.gluster.org/

A decade ago where I worked we used gluster for ~200TB of HDD for a shared file system on a SLURM compute cluster, as a much better clustered version of NFS. And we used ceph for its S3 interface (RadowGW) for tens of petabytes of back storage after the high IO stages of compute were finished. The ceph was all HDD though later we added some SSDs for a caching pool.

For single client performance, ceph beat the performance I get from S3 today for large file copies. Gluster had difficult to characterize performance, but our setup with big fast RAID arrays seems to still outperform what I see of AWS's luster as a service today for our use case of long sequential reads and writes.

We would occasionally try cephFS, the POSIX shared network filesystem, but it couldn't match our gluster performance for our workload. But also, we built the ceph long term storage to maximize TB/$, so it was at a disadvantage compared to our gluster install. Still, I never heard of cephFS being used anywhere despite it being the original goal in the papers back at UCSC. Keep an eye on CERN for news about one of the bigger ceph installs with public info.

I love both of the systems, and see ceph used everywhere today, but am surprised and happy to see that gluster is still around.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#30

Post by rubiquity »

nerdjon wrote: post #9

So is any of S3 powered by SSD's? I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.

> So is any of S3 powered by SSD's?

S3’s KeyMap Index uses SSDs. I also wouldn’t be surprised if at this point SSDs are somewhere along the read path for caching hot objects or in the new one zone product.

Post reply on HN