Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AWS S3 serves 1 petabyte per second on top of slow HDDs

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

71–80 of 172 posts

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#71

Post by roncesvalles »

dekhn wrote: post #69
temp0826 wrote: post #45

Earlier quoted context omitted.

In the early days when there were articles speculating on what Glacier was backed by, it was actually on crusty old S3 gear (and at the very beginning, it was just on S3 itself as a wrapper and a hand wavy price discount, eating the costs to get people to buy in to the idea!). Later on (2018 or so) they began moving to a home grown tape-based solution (at least for some tiers).

That's surprising given how badly restoration worked (much more like tape than drives).

I'd be curious whether simulating a shitty restoration experience was part of the emulation when they first ran Glacier on plain S3 to test the market.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#72

Post by gregates »

A few factual inaccuracies in here that don't affect the general thrust. For example, the claim that S3 uses a 5:9 sharding scheme. In fact they use many different sharding schemes, and iirc 5:9 isn't one of them.

The main reason being that a ratio of 1.8 physical bytes to 1 logical byte is awful for HDD costs. You can get that down significantly, and you get wider parallelism and better availability guarantees to boot (consider: if a whole AZ goes down, how many shards can you lose before an object is unavailable for GET?).

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#73

Post by [deleted] »

crazygringo wrote: post #51
ffsm8 wrote: post #50

Earlier quoted context omitted.

That's like saying "how to get to the moon is obvious: traveling"

I dunno, the article's tl;dr is just parallelism. Data gets split into redundant copies, and is rebalanced in response to hot spots. Everything in this article is the obvious answer you'd expect.

[deleted]

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#74

Post by gregates »

dgllghr wrote: post #17

I enjoyed this article but I think the answer to the headline is obvious: parallelism

I think the article's title question is a bit misleading because it focuses on peak throughput for S3 as a whole. The interesting question is "How can the throughput for a GET exceed the throughput of an HDD?"

If you just replicated, you could still get big throughput for S3 as a whole by doing many reads that target different HDDs. But you'd still be limited to max HDD throughput * number of GETs. S3 is not so limited, and that's interesting and non-obvious!

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#75

Post by gregates »

hnexamazon wrote: post #22
wg0 wrote: post #11

Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.

I was an SDE on the S3 Index team 10 years ago, but I doubt much of the core stack has changed. S3 is comprised primarily of layers of Java-based web services. The hot path (object get / put / list) are all served by synchronous API servers - no queues or workers. It is the best example of how many transactions per second a pretty standard Java web service stack can handle that I’ve seen in my career. For a get call,…

It's not all java anymore. There's some rust now, too. ShardStore, at least (which the article mentions).

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#76

Post by toolslive »

crazygringo wrote: post #51
ffsm8 wrote: post #50

Earlier quoted context omitted.

That's like saying "how to get to the moon is obvious: traveling"

I dunno, the article's tl;dr is just parallelism. Data gets split into redundant copies, and is rebalanced in response to hot spots. Everything in this article is the obvious answer you'd expect.

It's not really "redundant copies". It's erasure coding (ie, your data is the solution of an overdetermined system of equations).

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#77

Post by Const-me »

> full-platter seek time: ~8ms; half-platter seek time (avg): ~4ms

Average distance between two points (first is current location, second is target location) when both are uniformly distributed in [ 0 .. +1 ] interval is not 0.5, it’s 1/3. If the full platter seek time is 8ms, average seek time should be 2.666ms.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#78

Post by yabones »

nerdjon wrote: post #9

So is any of S3 powered by SSD's? I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.

The storage itself is probably (mostly) on HDDs, but I'd imagine metadata, indices, etc are stored on much faster flash storage. At least, that's the common advice for small-ish Ceph cluster MDS servers. Obviously S3 is a few orders of magnitude bigger than that...

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#79

Post by edude03 »

crabique wrote: post #44
creiht wrote: post #24

Earlier quoted context omitted.

It is probably worth noting that most of the listed storage systems (including S3) are designed to scale not only in hard drives, but horizontally across many servers in a distributed system. They really are not optimized for a single storage node use case. There are also other things to consider that can limit performance, like what does the storage back plane look like for those 80 HDDs, and how much throughput can…

It's a very beefy server with 4 NVMe and 20 HDD bays + a 60-drive external enclosure, 2 enterprise grade HBA cards set to multipath round-robin mode, even with 80 drives it's nowhere near the data path saturation point. The link is a 10G 9K MTU connection, the server is only accessed via that local link. Essentially, the drives being HDD are the only real bottleneck (besides the obvious single-node scenario). At the…

> Essentially, the drives being HDD are the only real bottleneck

? on the low end a single HD can deliver 100MB/s, 80 can deliver 8,000MB/s, a single nvme can do 700MB/s and you have 4, 2,800MB/s - a 10Gb link can only do 1000MB/s, so isn't your bottle neck Network and then probably CPU?

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#80

Post by nick49488171 »

Const-me wrote: post #77

> full-platter seek time: ~8ms; half-platter seek time (avg): ~4ms Average distance between two points (first is current location, second is target location) when both are uniformly distributed in [ 0 .. +1 ] interval is not 0.5, it’s 1/3. If the full platter seek time is 8ms, average seek time should be 2.666ms.

There's acceleration of the read head to move it between the tracks. So it may well be 4ms because shorter distances are penalized by a lower peak speed of the read head as well as constant factors (settling at the end of motion)
Post reply on HN