Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How AWS S3 serves 1 petabyte per second on top of slow HDDs

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

111–120 of 172 posts

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#111

Post by ignoramous »

booi wrote: post #103
chippiewill wrote: post #82

Earlier quoted context omitted.

I think that's close to the truth. IIRC it's something like a massive cluster of machines that are effectively powered off 99% of the time with a careful sharding scheme where they're turned on and off in batches over a long period of time for periodic backup or restore of blobs.

it's amazing that Glacier is such a huge system with so many people working on it and it's still a public mystery how it works. I've not seen a single confirmation of how it works..

Glacier could be doing similar to what Azure does: https://www.microsoft.com/en-us/research/project/project-sil...

Also see this thread: https://news.ycombinator.com/item?id=13011396

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#112

Post by rbranson »

Pxtl wrote: post #109
rushi_agrawal wrote: post #105

It is interesting that even after falling prices of HDDs, S3 costs have remained the same for at least 8 years. There's just not enough competition to push them to reduce costs. But imagine money it brings in in AWS because of this.

How much have hdd prices really fallen? AFAIK the incredible improvements in price per byte in HDD had slowed so much that they'll be eclipsed by SSDs in a few years.

Magnetic hard drives are 100X cheaper per GB than when S3 launched, and are about 3X cheaper than in 2016 when the price last dropped. Magnetic prices have actually ticked up recently due to supply chain issues, but HAMR is expected to cause a significant drop (50-75%/GB) in magnetic storage prices as it rolls out in next few years. SSDs are ~$120/T and magnetic drives are ~$18/T. This hasn't changed much in the last 2 years.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#114

Post by pyuser583 »

ignoramous wrote: post #111
booi wrote: post #103

Earlier quoted context omitted.

it's amazing that Glacier is such a huge system with so many people working on it and it's still a public mystery how it works. I've not seen a single confirmation of how it works..

Glacier could be doing similar to what Azure does: https://www.microsoft.com/en-us/research/project/project-sil... Also see this thread: https://news.ycombinator.com/item?id=13011396

I doubt it’s using WORM drives.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#115

Post by UltraSane »

bombela wrote: post #100
UltraSane wrote: post #96

Earlier quoted context omitted.

VAST data uses 146+4 https://www.vastdata.com/whitepaper/#similarity-reduction-in...

page loads then quicky move up some video loads, and content is gone

Wow that is very annoying. Here is a better page

https://www.vastdata.com/blog/introducing-rack-scale-resilie...

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#116

Post by shric »

glitchcrab wrote: post #91
MrDarcy wrote: post #81

Earlier quoted context omitted.

If you’re curious about this at home, try Ceph in Proxmox.

Unless you have a large cluster with many tens of nodes/OSDs (and who does in a homelab?) then using Ceph is a bad idea (I've run large Ceph clusters at previous jobs).

Why is it a bad idea?

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#117

Post by pjdesno »

Pxtl wrote: post #109
rushi_agrawal wrote: post #105

It is interesting that even after falling prices of HDDs, S3 costs have remained the same for at least 8 years. There's just not enough competition to push them to reduce costs. But imagine money it brings in in AWS because of this.

How much have hdd prices really fallen? AFAIK the incredible improvements in price per byte in HDD had slowed so much that they'll be eclipsed by SSDs in a few years.

Flash went from within 2x the price of DRAM in 2012 or so to maybe 40-50x cheaper today, driven somewhat by shrinking feature sizes, but mostly by the shift from SLC (1 bit/cell) to TLC (3 bits) and QLC (4 bits) and from planar to 300+ layer 3D flash.

Flash is near the end of the “S-curve” of those technologies being rolled out.

During that time HDD technology was pretty stagnant, with a mere 2x increase due to higher platter count with the use of helium.

New HDD technologies (HAMR) are just starting their rollout, promising major improvements in $/GB over the next few years as they roll out.

You can’t just look at a price curve on a graph and predict where it’s going to go. The actual technologies responsible for that curve matter.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#118

Post by pjdesno »

pjdesno wrote: post #117
Pxtl wrote: post #109

Earlier quoted context omitted.

How much have hdd prices really fallen? AFAIK the incredible improvements in price per byte in HDD had slowed so much that they'll be eclipsed by SSDs in a few years.

Flash went from within 2x the price of DRAM in 2012 or so to maybe 40-50x cheaper today, driven somewhat by shrinking feature sizes, but mostly by the shift from SLC (1 bit/cell) to TLC (3 bits) and QLC (4 bits) and from planar to 300+ layer 3D flash. Flash is near the end of the “S-curve” of those technologies being rolled out. During that time HDD technology was pretty stagnant, with a mere 2x increase due to highe…

Oh, and no one has a solution to make HDDs faster. If anything, they may have gotten slower as they get optimized for capacity instead of speed.

(Well, peak data transfer rate keeps going up as bits get packed tighter, but capacity goes up linearly with areal bit density, while the speed the bits go under the head goes up with the square root.)

(Well, sort of. For a while a lot of the progress came from making the bits skinnier but not much shorter, so transfer rates didn’t go up that much)

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#119

Post by pjdesno »

rubiquity wrote: post #58
electroly wrote: post #32

Earlier quoted context omitted.

It's assumed that the new S3 Express One Zone is backed by SSDs but I believe Amazon doesn't say so explicitly.

I've always felt it's probably a wrapper around the Amazon EFS due to the similar pricing and that S3 One Zone has "Directory" buckets, a very file system-y idea.

Seems to indicate the storage underneath might be similar in cost and performance, and this might in fact really be similar. Not that the software on top is the same.

Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs

#120

Post by pjdesno »

hobs wrote: post #26
MDGeist wrote: post #10

Earlier quoted context omitted.

I always assumed the really slow tiers were tape.

Not even the higher tiers of Glacier were tape afaict (at least when it was first created), just the observation that hard drives are much bigger than you can reasonably access in useful time.

The “drain time” for a 30TB drive is probably between 36 and 48 hours. I don’t have one in my lab to test, or the patience to do so if I did.
Post reply on HN