Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AWS S3 serves 1 petabyte per second on top of slow HDDs

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

61–70 of 172 posts

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#61

Post by timeinput »

dgllghr wrote: post #17

I enjoyed this article but I think the answer to the headline is obvious: parallelism

I generally don't think about storage I/O speed at that scale (I mean really who does?). I once used a RAID0 to store data to HDDs faster, but that was a long time ago.

I would have naively guessed an interesting caching system, and to some degree tiers of storage for hot vs cold objects.

It was obvious after I read the article that parallelism was a great choice, but I definitely hadn't considered the detailed scheme of S3, or the error correction it used. Parallelism is the one word summary, but the details made the article worth reading. I bet minio also has a similar scaling story: parallelism.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#62

Post by kerneltime »

crabique wrote: post #3

Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).

Apache Ozone has multiple 100+ petabyte clusters in production. The capacity is on HDDs and metadata is on SSDs. Updated docs (staging for new docs): https://kerneltime.github.io/ozone-site/

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#63

Post by toast0 »

crabique wrote: post #13
bayindirh wrote: post #8

Earlier quoted context omitted.

Lustre and ZFS can do similar speeds. However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.

Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…

If you can afford it, mirroring in some form is going to give you way better read perf than RAIDz. Using zfs mirrors is probably easiest but least flexible, zfs copies=2 with all devices as top level vdevs in a single zpool is not very unsafe, and something custom would be a lot of work but could get safety and flexibility if done right.

You're basically seek limited, and a read on a mirror is one seek, whereas a read on a RAIDz is one seek per device in the stripe. (Although if most of your objects are under the chunk size, you end up with more of mirroring than striping)

You lose on capacity though.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#64

Post by dang »

EwanToo wrote: post #2

I think a more interesting article on S3 is "Building and operating a pretty big storage system called S3" https://www.allthingsdistributed.com/2023/07/building-and-op...

Discussed at the time:

Building and operating a pretty big storage system called S3 - https://news.ycombinator.com/item?id=36894932 - July 2023 (160 comments)

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#65

Post by luhn »

temp0826 wrote: post #45
hobs wrote: post #26

Earlier quoted context omitted.

Not even the higher tiers of Glacier were tape afaict (at least when it was first created), just the observation that hard drives are much bigger than you can reasonably access in useful time.

In the early days when there were articles speculating on what Glacier was backed by, it was actually on crusty old S3 gear (and at the very beginning, it was just on S3 itself as a wrapper and a hand wavy price discount, eating the costs to get people to buy in to the idea!). Later on (2018 or so) they began moving to a home grown tape-based solution (at least for some tiers).

Do you have any sources for that? I'm really curious about Glacier's infrastructure and AWS has been notoriously tight-lipped about it. I haven't found anything better than informed speculation.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#66

Post by 0x457 »

timeinput wrote: post #61
dgllghr wrote: post #17

I enjoyed this article but I think the answer to the headline is obvious: parallelism

I generally don't think about storage I/O speed at that scale (I mean really who does?). I once used a RAID0 to store data to HDDs faster, but that was a long time ago. I would have naively guessed an interesting caching system, and to some degree tiers of storage for hot vs cold objects. It was obvious after I read the article that parallelism was a great choice, but I definitely hadn't considered the detailed schem…

My homelab servers all have raidz out of 3 nvme drives for this reason: higher parallelism without loosing redundancy.

> I would have naively guessed an interesting caching system, and to some degree tiers of storage for hot vs cold objects.

Caching in this scenario usually done outside of S3 in something like Cloudfront

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#67

Post by iamtedd »

luhn wrote: post #65
temp0826 wrote: post #45

Earlier quoted context omitted.

In the early days when there were articles speculating on what Glacier was backed by, it was actually on crusty old S3 gear (and at the very beginning, it was just on S3 itself as a wrapper and a hand wavy price discount, eating the costs to get people to buy in to the idea!). Later on (2018 or so) they began moving to a home grown tape-based solution (at least for some tiers).

Do you have any sources for that? I'm really curious about Glacier's infrastructure and AWS has been notoriously tight-lipped about it. I haven't found anything better than informed speculation.

My speculation: writes are to /dev/null, and the fact that reads are expensive and that you need to inventory your data before reading means Amazon is recreating your data from network transfer logs.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#68

Post by luhn »

iamtedd wrote: post #67
luhn wrote: post #65

Earlier quoted context omitted.

Do you have any sources for that? I'm really curious about Glacier's infrastructure and AWS has been notoriously tight-lipped about it. I haven't found anything better than informed speculation.

My speculation: writes are to /dev/null, and the fact that reads are expensive and that you need to inventory your data before reading means Amazon is recreating your data from network transfer logs.

Maybe they ask the NSA for a copy.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#69

Post by dekhn »

temp0826 wrote: post #45
hobs wrote: post #26

Earlier quoted context omitted.

Not even the higher tiers of Glacier were tape afaict (at least when it was first created), just the observation that hard drives are much bigger than you can reasonably access in useful time.

In the early days when there were articles speculating on what Glacier was backed by, it was actually on crusty old S3 gear (and at the very beginning, it was just on S3 itself as a wrapper and a hand wavy price discount, eating the costs to get people to buy in to the idea!). Later on (2018 or so) they began moving to a home grown tape-based solution (at least for some tiers).

That's surprising given how badly restoration worked (much more like tape than drives).

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#70

Post by epistasis »

crabique wrote: post #13
bayindirh wrote: post #8

Earlier quoted context omitted.

Lustre and ZFS can do similar speeds. However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.

Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…

It's great to see other people's working solutions, thanks. Can I ask if you have backup on something like this? In many systems it's possible to store some data on ingress or after processing, which serves as something that's rebuildable, even if it's not a true backup. I'm not familiar if your software layer has backup to off site as part of their system, for example, which would be a great feature.
Post reply on HN