Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AWS S3 serves 1 petabyte per second on top of slow HDDs

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

31–40 of 172 posts

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#31

Post by taeric »

Twirrim wrote: post #15
wg0 wrote: post #11

Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.

Amazon biases towards Systems Oriented Architecture approach that is in the middle ground between monolith and microservices. Biasing away from lots of small services in favour of larger ones that handle more of the work so that as much as possible you avoid the costs and latency of preparing, transmitting, receiving and processing requests. I know S3 has changed since I was there nearly a decade ago, so this is outd…

Amusingly, I suspect that the "dozen main services" is still quite a few more than most smaller companies would consider on their stacks.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#32

Post by electroly »

nerdjon wrote: post #9

So is any of S3 powered by SSD's? I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.

It's assumed that the new S3 Express One Zone is backed by SSDs but I believe Amazon doesn't say so explicitly.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#33

Post by huntaub »

littlesnugblood wrote: post #25
EwanToo wrote: post #2

I think a more interesting article on S3 is "Building and operating a pretty big storage system called S3" https://www.allthingsdistributed.com/2023/07/building-and-op...

[flagged]

Woah buddy, I worked with Andy for years and this is not my experience. Moving a large product like S3 around is really, really difficult, and I've always thought highly of Andy's ability to: (a) predict where he thought the product should go, (b) come up with novel ways of getting there, and (c) trimming down the product to get something in the hands of customers.

Also, did you create this account for the express purpose of bashing Andy? That's not cool.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#35

Post by Twirrim »

taeric wrote: post #31
Twirrim wrote: post #15

Earlier quoted context omitted.

Amazon biases towards Systems Oriented Architecture approach that is in the middle ground between monolith and microservices. Biasing away from lots of small services in favour of larger ones that handle more of the work so that as much as possible you avoid the costs and latency of preparing, transmitting, receiving and processing requests. I know S3 has changed since I was there nearly a decade ago, so this is outd…

Amusingly, I suspect that the "dozen main services" is still quite a few more than most smaller companies would consider on their stacks.

Probably. Conway's law comes into effect, naturally.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#36

Post by g-mork »

MDGeist wrote: post #10
nerdjon wrote: post #9

So is any of S3 powered by SSD's? I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.

I always assumed the really slow tiers were tape.

There might be surprisingly little value in going tape due to all the specialization required. As the other comment suggest, many of the lower tiers likely represent basically IO bandwidth classes. a 16 TB disk with 100 IOPs can only offer 1 IOP/s over 1.6 TB for 100 customers, or 0.1 IOP/s over 160 GB for 1000, etc. Just scale up that thinking to a building full of disks, it still applies

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#37

Post by Powdering7082 »

huntaub wrote: post #33
littlesnugblood wrote: post #25

Earlier quoted context omitted.

[flagged]

Woah buddy, I worked with Andy for years and this is not my experience. Moving a large product like S3 around is really, really difficult, and I've always thought highly of Andy's ability to: (a) predict where he thought the product should go, (b) come up with novel ways of getting there, and (c) trimming down the product to get something in the hands of customers. Also, did you create this account for the express pu…

[flagged]

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#39

Post by pickle-wizard »

crabique wrote: post #3

Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments. Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).

At a past job we had an object store that used SwiftStack. We just used SSDs for the metadata storage but all the objects were stored on regular HDDs. It worked well enough.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#40

Post by yanslookup »

ttfvjktesd wrote: post #34

> tens of millions of disks If we assume enterprise HDDs in the double digit TB range then one can estimate that the total S3 storage volume of AWS is in the triple digit Exabyte range. That's propably the biggest storage system on planet earth.

Production scale enterprise HDDs are in the 30TB range, 50TB on the horizon...
Post reply on HN