Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AWS S3 serves 1 petabyte per second on top of slow HDDs

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

81–90 of 172 posts

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#81

Post by MrDarcy »

timeinput wrote: post #61
dgllghr wrote: post #17

I enjoyed this article but I think the answer to the headline is obvious: parallelism

I generally don't think about storage I/O speed at that scale (I mean really who does?). I once used a RAID0 to store data to HDDs faster, but that was a long time ago. I would have naively guessed an interesting caching system, and to some degree tiers of storage for hot vs cold objects. It was obvious after I read the article that parallelism was a great choice, but I definitely hadn't considered the detailed schem…

If you’re curious about this at home, try Ceph in Proxmox.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#82

Post by chippiewill »

derefr wrote: post #52
MDGeist wrote: post #10

Earlier quoted context omitted.

I always assumed the really slow tiers were tape.

My own assumption was always that the cold tiers are managed by a tape robot , but managing offlined HDDs rather than actual tapes.

I think that's close to the truth. IIRC it's something like a massive cluster of machines that are effectively powered off 99% of the time with a careful sharding scheme where they're turned on and off in batches over a long period of time for periodic backup or restore of blobs.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#83

Post by everfrustrated »

temp0826 wrote: post #45
hobs wrote: post #26

Earlier quoted context omitted.

Not even the higher tiers of Glacier were tape afaict (at least when it was first created), just the observation that hard drives are much bigger than you can reasonably access in useful time.

In the early days when there were articles speculating on what Glacier was backed by, it was actually on crusty old S3 gear (and at the very beginning, it was just on S3 itself as a wrapper and a hand wavy price discount, eating the costs to get people to buy in to the idea!). Later on (2018 or so) they began moving to a home grown tape-based solution (at least for some tiers).

I'm not aware of AWS ever confirming tape for glacier. My own speculation is they likely use hdd for glacier - especially so for the smaller regions - and eat the cost.

Someone recently came across some planning documents filed in London for a small "datacenter" which wasn't attached to their usual London compute DCs and built to house tape libraries (this was explicitly called out as there was concern about power - tape libraries don't use much). So I would be fairly confident they wait until the glacier volumes grow enough on hdd before building out tape infra.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#84

Post by pjdesno »

Note that you can kind of infer that S3 is still using hard drives for their basic service by looking at pricing and calculating the IOPS rate that doubles the cost per GB per month.

S3 GET and PUT requests are sufficiently expensive that AWS can afford to let disk space sit idle to satisfy high-performance tenants, but not a lot more expensive than that.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#85

Post by pjdesno »

Const-me wrote: post #77

> full-platter seek time: ~8ms; half-platter seek time (avg): ~4ms Average distance between two points (first is current location, second is target location) when both are uniformly distributed in [ 0 .. +1 ] interval is not 0.5, it’s 1/3. If the full platter seek time is 8ms, average seek time should be 2.666ms.

Full seek on a modern drive is a lot closer to 25ms than 8ms. It's pretty easy to test yourself if you have a hard drive in your machine and root access - fire up fio with --readonly and feed it a handmade trace that alternates reading blocks at the beginning and end of disk. (--readonly does a hard disable of any code that could write to the drive)

Here's a good paper that explains why the 1/3 number isn't quite right on any drives manufactured in the last quarter century or so - https://www.msstconference.org/MSST-history/2024/Papers/msst...

I'd be happy to answer any other questions about disk drive mechanics and performance.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#86

Post by pjdesno »

nerdjon wrote: post #9

So is any of S3 powered by SSD's? I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.

Repeating a comment I made above - for standard tier, requests are expensive enough that it's cost-effective to let space on the disks go unused if someone wants an IOPS/TB ratio that's higher than what disk drives can provide. But not much more expensive than that.

The latest generation of drives store about 30TB - I don't know how much AWS pays for them, but a wild-ass guess would be $300-$500. That's a lot cheaper than 30TB of SSD.

Also important - you can put those disks in high-density systems (e.g. 100 drives in 4U) that only add maybe 25% to the total cost, at least if you're AWS, a bit more for the rest of us. The per-slot cost of boxes that hold lots of SSDs seems to be a lot higher.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#87

Post by pjdesno »

yanslookup wrote: post #40
ttfvjktesd wrote: post #34

> tens of millions of disks If we assume enterprise HDDs in the double digit TB range then one can estimate that the total S3 storage volume of AWS is in the triple digit Exabyte range. That's propably the biggest storage system on planet earth.

Production scale enterprise HDDs are in the 30TB range, 50TB on the horizon...

Google for Seagate Mozaic

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#88

Post by pjdesno »

crabique wrote: post #13
bayindirh wrote: post #8

Earlier quoted context omitted.

Lustre and ZFS can do similar speeds. However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.

Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…

Ceph's S3 protocol implementation is really good.

Getting Ceph erasure coding set up properly on a big hard disk pool is a pain - you can tell that EC was shoehorned into a system that was totally designed around triple replication.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#89

Post by pjdesno »

wg0 wrote: post #11

Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.

The only scholarly paper they've written about it is this one: https://www.amazon.science/publications/using-lightweight-fo...

(well, I think they may have submitted one or two others, but this is the only one that got published)

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#90

Post by pjdesno »

derefr wrote: post #53
hnexamazon wrote: post #22

Earlier quoted context omitted.

I was an SDE on the S3 Index team 10 years ago, but I doubt much of the core stack has changed. S3 is comprised primarily of layers of Java-based web services. The hot path (object get / put / list) are all served by synchronous API servers - no queues or workers. It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career. For a get call,…

> The hot path (... list) are all served by synchronous API servers Wait; how does that work, when a user is PUTting tons of objects concurrently into a bucket, and then LISTing the bucket during that? If the PUTs are all hitting different indexing-cluster nodes, then...? (Or do you mean that there are queues/workers, but only outside the hot path; with hot-path requests emitting events that then get chewed through a…

LIST is dog slow, and everyone expects it to be. (my research group did a prototype of an ultra-high-speed S3-compatible system, and it really helps not needing to list things quickly)
Post reply on HN