Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AWS S3 serves 1 petabyte per second on top of slow HDDs

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

161–170 of 172 posts

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#161

Post by pjdesno »

Dylan16807 wrote: post #133
Dylan16807 wrote: post #128

Earlier quoted context omitted.

> mostly by the shift from SLC (1 bit/cell) to TLC (3 bits) and QLC (4 bits) and from planar to 300+ layer 3D flash That "and" is doing a lot of work. In 2012 most flash was MLC. In 2025 most flash is TLC. > During that time HDD technology was pretty stagnant, with a mere 2x increase due to higher platter count with the use of helium. They've advanced slower than SSDs but it wasn't that slow. Between 2012 and 2025, e…

This is one of those times a downvote confuses me. I corrected some numbers. Was I accidentally rude? If I made a mistake on the numbers please give the right numbers. If my first line was unclear: We might say the denser bits give us a 65% density improvement. And quick math shows that a 80-100x improvement is actually nine 65% improvements in a row. So the denser bits per cell aren't doing much, it's pretty much al…

It’s mostly 3D, not process.

3D flash is over 300 layers now. The size of a single 300-bit stack on the surface of the chip is bigger than an old planar cell, but that 300x does a lot more than make up for it.

3D NAND isn’t a “process improvement” - it’s a fundamental new architecture. It’s radically cheaper because it’s a set of really cheap steps to make all 300+ layers, not using any of the really expensive lithography systems in the fab, then a single (really complicated) set of steps to drill holes through the layers for the bit stacks and coat the insides of the holes. Chip cost basically = the depreciation of the fab investment during the time a chip spends in the fab, so 3D NAND is a huge win. (just stacking layers by running the chip through the process N times wouldn’t save any money, and would probably just decrease yields)

A total guess - 2x more expensive for extra steps, bit stacks take 4x more area than planar cells, 300 layer would have 300/8 = 37.5x cheaper bits. (That 4x is pulling a lot of weight - for all I know it might be more like 8x, but the point stands)

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#162

Post by Dylan16807 »

pjdesno wrote: post #161
Dylan16807 wrote: post #133

Earlier quoted context omitted.

This is one of those times a downvote confuses me. I corrected some numbers. Was I accidentally rude? If I made a mistake on the numbers please give the right numbers. If my first line was unclear: We might say the denser bits give us a 65% density improvement. And quick math shows that a 80-100x improvement is actually nine 65% improvements in a row. So the denser bits per cell aren't doing much, it's pretty much al…

It’s mostly 3D, not process. 3D flash is over 300 layers now. The size of a single 300-bit stack on the surface of the chip is bigger than an old planar cell, but that 300x does a lot more than make up for it. 3D NAND isn’t a “process improvement” - it’s a fundamental new architecture. It’s radically cheaper because it’s a set of really cheap steps to make all 300+ layers, not using any of the really expensive lithog…

I was counting all the 3D manufacturing innovations as "process improvement". I'm not sure why you don't.

Anyway the point stands that bits per cell is barely doing anything compared to making the cells cheaper.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#163

Post by zenmac »

crabique wrote: post #144
zenmac wrote: post #132

Earlier quoted context omitted.

>Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC). What you mean by no EC?

In their design document at https://garagehq.deuxfleurs.fr/documentation/design/goals/ they state: "erasure coding or any other coding technique both increase the difficulty of placing data and synchronizing; we limit ourselves to duplication"

Nice! Learned something new today. Seems like a way for error correction. One can store parts of data with some more metadata and if some parts of the data are lost, the original can be reconstructed via some use of computational power.

Seems like some kind of compression?

Is that how the error correction on DVD works? I

And is that how GridFS is can keep file store slow low compare to regular file system?

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#165

Post by pjdesno »

Dylan16807 wrote: post #162
pjdesno wrote: post #161

Earlier quoted context omitted.

It’s mostly 3D, not process. 3D flash is over 300 layers now. The size of a single 300-bit stack on the surface of the chip is bigger than an old planar cell, but that 300x does a lot more than make up for it. 3D NAND isn’t a “process improvement” - it’s a fundamental new architecture. It’s radically cheaper because it’s a set of really cheap steps to make all 300+ layers, not using any of the really expensive lithog…

I was counting all the 3D manufacturing innovations as "process improvement". I'm not sure why you don't. Anyway the point stands that bits per cell is barely doing anything compared to making the cells cheaper.

Because they made something different with the same process, instead of making the same thing with a different process. Feature size didn’t get any smaller. (or, rather, you get the order of magnitude improvement without it, and those gains were vastly more than the feature size improvements over that time period)

Also because “process improvement” usually refers to things where you get incremental improvements basically for free as each new generation of fab rolls out. Unless you can invent a 4D flash, this is a single (huge) improvement that’s mostly played out.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#166

Post by Dylan16807 »

pjdesno wrote: post #165
Dylan16807 wrote: post #162

Earlier quoted context omitted.

I was counting all the 3D manufacturing innovations as "process improvement". I'm not sure why you don't. Anyway the point stands that bits per cell is barely doing anything compared to making the cells cheaper.

Because they made something different with the same process, instead of making the same thing with a different process. Feature size didn’t get any smaller. (or, rather, you get the order of magnitude improvement without it, and those gains were vastly more than the feature size improvements over that time period) Also because “process improvement” usually refers to things where you get incremental improvements basic…

> with the same process

Same process node.

Node is part of process, but all the layering and etching techniques they figured out to make 3D cells are also process. At least that's how I see it.

Oh well, I don't want to argue definitions, I just want to clarify what I meant.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#167

Post by nh2 »

pjdesno wrote: post #124
nh2 wrote: post #121

Earlier quoted context omitted.

Coudl you eleborate what you mean by the last sentence?

Originally Ceph divided big objects into 4MB chunks, sending each chunk to an OSD server which replicated it to 2 more servers. 4MB was chosen because it was several drive rotations, so the seek+ rotational delay didn’t affect the throughput very much. Now the first OSD splits it into k data chunks plus d parity chunks, so the disk write size isn’t 4MB, it’s 4MB/k, while the efficient write size has gone up 2x? 4x? s…

I'm still not sure which exact Ceph concept you are referring to. Thre is the "minimum allocation size" [1], but that is currently 4 KB (not MB).

There is also striping [2], which is the equivalent of RAID-10 functionality to split a large file into independent segments that can be written in parallel. Perhaps you are referring to RGW's default stripe size of 4 MB [3]?

If yes, I can understand your point about one 4 MB RADOS object being erasure-coded to e.g. 6 = 4+2 "parity chunks", making it But would you not simply raise `rgw_obj_stripe_size` to address that, according to the k you choose? E.g. 24 MB? You mention it can be changed, but I don't understand the "but still the tuning is based on the size of the block to be coded" part, (why) is that a problem?

Also, how else would you do it when designing EC writes?

Thanks!

[1]: https://docs.ceph.com/en/squid/rados/configuration/bluestore...

[2]: https://docs.ceph.com/en/squid/architecture/#data-striping

[3]: https://docs.ceph.com/en/squid/radosgw/config-ref/#confval-r...

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#168

Post by fair_enough »

crazygringo wrote: post #51
ffsm8 wrote: post #50

Earlier quoted context omitted.

That's like saying "how to get to the moon is obvious: traveling"

I dunno, the article's tl;dr is just parallelism. Data gets split into redundant copies, and is rebalanced in response to hot spots. Everything in this article is the obvious answer you'd expect.

You're right if you're only looking at peak sequential throughput. However, and this is the part that the author could have emphasized more, the impressive part is their strategy for dealing with disk access latency to improve random read throughput.

They shard the data as you might expect of a RAID, 5, 6, etc array and the distributed parity solves the problem of failure tolerance as you would expect and also improves bandwidth via parallelism as you describe.

The interesting part is their best strategy for sharding the data: plain-old-simple random. The decision of which disks and at which sectors to shard the data is done at random, and this creates the best change that at least one of the two copies of data can be accessed with much lower latency (~1ms instead of ~8ms).

The most crude, simple approach turns out to give them the best mileage. There's something vaguely poetic about it, an aesthetic beauty reminiscent of Euler's Identity or the solution to the Basel Problem; a very simple statement with powerful implications.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#169

Post by MrDarcy »

glitchcrab wrote: post #91
MrDarcy wrote: post #81

Earlier quoted context omitted.

If you’re curious about this at home, try Ceph in Proxmox.

Unless you have a large cluster with many tens of nodes/OSDs (and who does in a homelab?) then using Ceph is a bad idea (I've run large Ceph clusters at previous jobs).

Disagree. I run Ceph in Proxmox and have for years on a small cluster of 3 used R620 servers without any SSDs.

It’s just worked. I’ve lost two of the machines due to memory failures at two different points in time and the k8s clusters sitting on top didn’t fail, even the Postgres databases running with cnpg remained ready and available during both hardware failures.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#170

Post by glitchcrab »

MrDarcy wrote: post #169
glitchcrab wrote: post #91

Earlier quoted context omitted.

Unless you have a large cluster with many tens of nodes/OSDs (and who does in a homelab?) then using Ceph is a bad idea (I've run large Ceph clusters at previous jobs).

Disagree. I run Ceph in Proxmox and have for years on a small cluster of 3 used R620 servers without any SSDs. It’s just worked. I’ve lost two of the machines due to memory failures at two different points in time and the k8s clusters sitting on top didn’t fail, even the Postgres databases running with cnpg remained ready and available during both hardware failures.

Oh sure it works, not denying that. My point is that performance isn't great and if you only have a small cluster then it doesn't take much to make everything fall over because your failure domains are huge (in your case, you only have 3).

But then to offset the above, it also depends on how important your environment is; homelabs don't usually require five nines.

I am a big Proxmox fan but I dislike how easy it makes Ceph to run (or rather, how it appears to be easy). Ceph can fail in so many ways (I've seen a lot of them) and most people who set a Ceph cluster up through the UI are going to have a hard time recovering their data when things go south.

Post reply on HN