Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AWS S3 serves 1 petabyte per second on top of slow HDDs

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

131–140 of 172 posts

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#131

Post by jasonwatkinspdx »

derefr wrote: post #52
MDGeist wrote: post #10

Earlier quoted context omitted.

I always assumed the really slow tiers were tape.

My own assumption was always that the cold tiers are managed by a tape robot , but managing offlined HDDs rather than actual tapes.

Yeah, I don't know about S3, but years back I talked a fair bit with someone that did storage stuff for HPC, and one thing he talked about is building huge JBOD arrays where only a handful of disks per rack would be spun up, basically pushing what could be done with scsi extenders or such. It wouldn't surprise me if they're doing something like that with batch scheduling the drive activations over a minutes to hours window.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#132

Post by zenmac »

crabique wrote: post #3

Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).

>Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).

What you mean by no EC?

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#133

Post by Dylan16807 »

Dylan16807 wrote: post #128
pjdesno wrote: post #117

Earlier quoted context omitted.

Flash went from within 2x the price of DRAM in 2012 or so to maybe 40-50x cheaper today, driven somewhat by shrinking feature sizes, but mostly by the shift from SLC (1 bit/cell) to TLC (3 bits) and QLC (4 bits) and from planar to 300+ layer 3D flash. Flash is near the end of the “S-curve” of those technologies being rolled out. During that time HDD technology was pretty stagnant, with a mere 2x increase due to highe…

> mostly by the shift from SLC (1 bit/cell) to TLC (3 bits) and QLC (4 bits) and from planar to 300+ layer 3D flash That "and" is doing a lot of work. In 2012 most flash was MLC. In 2025 most flash is TLC. > During that time HDD technology was pretty stagnant, with a mere 2x increase due to higher platter count with the use of helium. They've advanced slower than SSDs but it wasn't that slow. Between 2012 and 2025, e…

This is one of those times a downvote confuses me. I corrected some numbers. Was I accidentally rude? If I made a mistake on the numbers please give the right numbers.

If my first line was unclear: We might say the denser bits give us a 65% density improvement. And quick math shows that a 80-100x improvement is actually nine 65% improvements in a row. So the denser bits per cell aren't doing much, it's pretty much all process improvement.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#134

Post by ffsm8 »

DoctorOW wrote: post #57
ffsm8 wrote: post #50

Earlier quoted context omitted.

That's like saying "how to get to the moon is obvious: traveling"

Thank you for setting me up for this... It's not exactly rocket science.

Haha, good one!

I still feel like you're underselling the article however.

Is obviously ultimately parallelism, but parallelism is hard at scale - because things often don't scale - and incorrect parallelism can even make things slower. And it's not always obvious why something gets slower by parallelism.

As a dumb example, if you have a fictional HDD with one disk and one head, you have two straightforward options to optimize performance:

Make sure only one file is read at the same time (otherwise the disk will keep seeking back and forth)

Make sure the file is persisted in a way that you're only accessing one sector, never entering the situation in which it would seek back and forth.

Ofc, that can be dumped down to "parallelism", because this is inherently a question about how to parallelize... But it's also ignoring that that's what is being elaborated on: ways s3 used to enable parallelism

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#137

Post by jiggawatts »

rubiquity wrote: post #59
riknos314 wrote: post #46

Earlier quoted context omitted.

Microservices for days. I worked on lifecycle ~5 years ago and just the Standard -> Glacier transition path involved no fewer than 7 microservices. Just determining which of the 400 trillion keys are eligible for a lifecycle action (comparing each object's metadata against the lifecycle policy on the bucket) is a massive big data job. Always was a fun oncall when some bucket added a lifecycle rule that queued 1PB+ of…

I used to work on the backing service for S3's Index and the daily humps in our graphs from lifecycle running were immense!

I work on tiny systems now, but something I miss from "big" deployments is how smooth all of the metrics were! Any bump was a signal that really meant something.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#138

Post by jiggawatts »

charltones wrote: post #56
wg0 wrote: post #11

Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.

There's a pretty good talk on S3 under the hood from last year's re:Invent: https://www.youtube.com/watch?v=NXehLy7IiPM

"Pretty good" is hugely underselling this!

I was just looking for this video so I can send it to my coworkers as one of the best introductory videos into the basics of cloud computing concepts.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#139

Post by enether »

gregates wrote: post #72

A few factual inaccuracies in here that don't affect the general thrust. For example, the claim that S3 uses a 5:9 sharding scheme. In fact they use many different sharding schemes, and iirc 5:9 isn't one of them. The main reason being that a ratio of 1.8 physical bytes to 1 logical byte is awful for HDD costs. You can get that down significantly, and you get wider parallelism and better availability guarantees to bo…

See timestamp 42:20 at https://youtu.be/NXehLy7IiPM?si=QQEOMCt7kOBTMaGK

The way it’s worded makes me understand that’s what scheme they’re using. Curious to hear what you know

Post reply on HN