Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AWS S3 serves 1 petabyte per second on top of slow HDDs

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

121–130 of 172 posts

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#121

Post by nh2 »

pjdesno wrote: post #88
crabique wrote: post #13

Earlier quoted context omitted.

Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…

Ceph's S3 protocol implementation is really good. Getting Ceph erasure coding set up properly on a big hard disk pool is a pain - you can tell that EC was shoehorned into a system that was totally designed around triple replication.

Coudl you eleborate what you mean by the last sentence?

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#122

Post by nh2 »

cullenking wrote: post #92
crabique wrote: post #3

Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).

We've been running a production ceph cluster for 11 years now, with only one full scheduled downtime for a major upgrade in all those years, across three different hardware generations. I wouldn't call it easy, but I also wouldn't call it hard. I used to run it with SSDs for radosgw indexes as well as a fast pool for some VMs, and harddrives for bulk object storage. Since i was only running 5 nodes with 10 drives eac…

We're using CephFS for a couple years, with some PBs of data on it (HDDs).

What performance issues and footguns do you have in mind?

I also like that CephFS has a performance benefits that doesn't seem to exist anywhere else: Automatic transparent Linux buffer caching, so that writes are extremely fast and local until you fsync() or other clients want to read, and repeat-reads or read-after-write are served from local RAM.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#123

Post by pjdesno »

enether wrote: post #6
giancarlostoro wrote: post #4

Earlier quoted context omitted.

Really nice read, thank you for that.

Author of the 2minutestreaming blog here. Good point! I'll add this as a reference at the end. I loved that piece. My goal was to be more concise and focus on the HDD aspect

Please fix the seek time numbers - they’re wildly in accurate; full platter seek is more like 20-25ms. I tried chasing down where the 8ms number came from a while back, and I think it applies to old sub-100GB 10K RPM high-speed drives from 25 or so years ago, which were purposely low density so they could swing the head faster and less precisely.

Check out the Olmez et al paper from MSST 2024 - I linked it above, but here it is again: https://www.msstconference.org/MSST-history/2024/Papers/msst...

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#124

Post by pjdesno »

nh2 wrote: post #121
pjdesno wrote: post #88

Earlier quoted context omitted.

Ceph's S3 protocol implementation is really good. Getting Ceph erasure coding set up properly on a big hard disk pool is a pain - you can tell that EC was shoehorned into a system that was totally designed around triple replication.

Coudl you eleborate what you mean by the last sentence?

Originally Ceph divided big objects into 4MB chunks, sending each chunk to an OSD server which replicated it to 2 more servers. 4MB was chosen because it was several drive rotations, so the seek+ rotational delay didn’t affect the throughput very much.

Now the first OSD splits it into k data chunks plus d parity chunks, so the disk write size isn’t 4MB, it’s 4MB/k, while the efficient write size has gone up 2x? 4x? since the original 4MB decision as drive transfer rates increase.

You can change this, but still the tuning is based on the size of the block to be coded, not the size of the chunks to be written to disk. (and you might have multiple pools with much different values of k)

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#125

Post by temp0826 »

luhn wrote: post #65
temp0826 wrote: post #45

Earlier quoted context omitted.

In the early days when there were articles speculating on what Glacier was backed by, it was actually on crusty old S3 gear (and at the very beginning, it was just on S3 itself as a wrapper and a hand wavy price discount, eating the costs to get people to buy in to the idea!). Later on (2018 or so) they began moving to a home grown tape-based solution (at least for some tiers).

Do you have any sources for that? I'm really curious about Glacier's infrastructure and AWS has been notoriously tight-lipped about it. I haven't found anything better than informed speculation.

Source is SWIM who worked there (doubt any of that stuff has been published)

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#126

Post by Demiurge »

derefr wrote: post #52
MDGeist wrote: post #10

Earlier quoted context omitted.

I always assumed the really slow tiers were tape.

My own assumption was always that the cold tiers are managed by a tape robot , but managing offlined HDDs rather than actual tapes.

There was an article or interview with one of the lead AWS engineers, and he said they use CDs or DVDs for cold glacier.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#127

Post by Dylan16807 »

christina97 wrote: post #108
toolslive wrote: post #76

Earlier quoted context omitted.

It's not really "redundant copies". It's erasure coding (ie, your data is the solution of an overdetermined system of equations).

That’s just fractional redundant copies.

And "fractional redundant copies" is way less obvious.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#128

Post by Dylan16807 »

pjdesno wrote: post #117
Pxtl wrote: post #109

Earlier quoted context omitted.

How much have hdd prices really fallen? AFAIK the incredible improvements in price per byte in HDD had slowed so much that they'll be eclipsed by SSDs in a few years.

Flash went from within 2x the price of DRAM in 2012 or so to maybe 40-50x cheaper today, driven somewhat by shrinking feature sizes, but mostly by the shift from SLC (1 bit/cell) to TLC (3 bits) and QLC (4 bits) and from planar to 300+ layer 3D flash. Flash is near the end of the “S-curve” of those technologies being rolled out. During that time HDD technology was pretty stagnant, with a mere 2x increase due to highe…

> mostly by the shift from SLC (1 bit/cell) to TLC (3 bits) and QLC (4 bits) and from planar to 300+ layer 3D flash

That "and" is doing a lot of work.

In 2012 most flash was MLC.

In 2025 most flash is TLC.

> During that time HDD technology was pretty stagnant, with a mere 2x increase due to higher platter count with the use of helium.

They've advanced slower than SSDs but it wasn't that slow. Between 2012 and 2025, excluding HAMR, sizes have improved from 4TB to 24TB and prices at the low end have improved from $50/TB to $12/TB.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#130

Post by hobs »

pjdesno wrote: post #120
hobs wrote: post #26

Earlier quoted context omitted.

Not even the higher tiers of Glacier were tape afaict (at least when it was first created), just the observation that hard drives are much bigger than you can reasonably access in useful time.

The “drain time” for a 30TB drive is probably between 36 and 48 hours. I don’t have one in my lab to test, or the patience to do so if I did.

Yep, did 20TB drives in my Unraid box and took about 2 days and some change to setup a clean sync between em all :)
Post reply on HN