Building and operating a pretty big storage system called S3 • Hacker News Board

Re: Building and operating a pretty big storage system called S3

#131

Post by mabbo » Fri, Jul 28, 2023, 10:38 AM UTC

Dylan16807 wrote: post #51

mabbo wrote: post #35

Earlier quoted context omitted.

Even at a billion requests per second, 128 bit UUIDs shouldn't collide for something like a billion years. And that's if you're going completely random and not taking care to try to reduce collisions.

Are you sure about that math? A billion seconds at a billion requests per second is already 2^60 items. You'd only need a few billion seconds to have a 50:50 collision chance with 128 random bits, and even less with a real UUID that only has 122 random bits. You'd hit 1% odds of collision after less than a decade. If you actually want to go for a billion years, you need to expand that UUID by 50%.

You know I think I converted powers of two and powers of ten interchangeably in my calculations. You're very likely correct.

Re: Building and operating a pretty big storage system called S3

#132

Post by bombcar » Fri, Jul 28, 2023, 12:35 PM UTC

aitchnyu wrote: post #128

bombcar wrote: post #65

Earlier quoted context omitted.

Glacier is just run on S3 with some sleep statements added.

The perceived value of results is higher if it takes a longer time to load, users feel the computer is hard at work. If its true for flight searches, its true for backup systems.

Reminds me of the automated phone systems that play random keystrokes while telling you they’re looking up your info - people don’t trust it if they come back instantly, I guess.

Re: Building and operating a pretty big storage system called S3

#133

Post by da39a3ee » Fri, Jul 28, 2023, 1:04 PM UTC

aborsy wrote: post #111

ignoramous wrote: post #4

Earlier quoted context omitted.

James Hamilton, AWS' chief architect, wrote about this phenomena in 2017: At scale, rare events aren't rare ; https://news.ycombinator.com/item?id=14038044

This phenomenon is just multiplication of the sample size (scale) times a probability (rare).

I agree with what I think is your sentiment -- that people seem to be treating this as if it's some sort of profound insight that you only get if you work at a very senior level in engineering for major US cloud providers, when it's in fact blindingly obvious!

Re: Building and operating a pretty big storage system called S3

#134

Post by dosman33 » Fri, Jul 28, 2023, 1:27 PM UTC

Not trying to be an arse, but the guy spent a lot more time talking about himself and other unrelated stuff than about how S3 works. And I don't mind a good article on RAMAC, but that seems... out of place in a discussion about peta-scale storage. I got the strong impression he doesn't really know the finer details of how S3 really works. And that's probably fine for what he's doing, there is plenty of room for application coding, firefighting, and problem management without having to get into the finer details of how it all works.

Re: Building and operating a pretty big storage system called S3

#135

Post by jeffbarr » Fri, Jul 28, 2023, 1:44 PM UTC

blindhippo wrote: post #38

jl6 wrote: post #18

Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).

Amazon engineer here - can confirm that Glacier transcodes all data on to the backs of the shells of the turtles that hold up the universe. Infinite storage medium, if a bit slow.

Shh....

Re: Building and operating a pretty big storage system called S3

#136

Post by dosman33 » Fri, Jul 28, 2023, 1:53 PM UTC

jl6 wrote: post #18

Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).

HSM is a neat technology, and lots of ways it has been implemented over the years. But it starts with a shim to insert some other technology into the middle a typical posix filesystem. It has to tolerate the time penalty for data recovery of your favored HSM'd medium, but that's kind of the point. You can do it with a lower tier disk, tape, wax cylinder, etc. There's no reason it wouldn't be tape though, tape capacity has kept up and HPSS continues to be developed. The traditional tape library vendors still pump out robotic tape libraries.

I remember installing 20+ fully configured IBM 3494 tape libraries for AT&T in the mid-2000's. These things were 20+ frames long with dual accessors (robots) in each. The robots were able to push a dead accessor out of the way into a "garage" and continue working in the event one of them died (and this actually worked). Someone will have to invent a cheaper medium of storage than tape before tape will ever die.

Re: Building and operating a pretty big storage system called S3

#137

Post by anderspitman » Fri, Jul 28, 2023, 1:53 PM UTC

afr0ck wrote: post #125

jamesblonde wrote: post #107

Earlier quoted context omitted.

Most apps, however, assume POSIX-like data access. I would love to see a client-side minimally dependent library that mounts a local directory that is actually the user's S3 bucket.

Linux has FUSE, which is a framework to develop user-level filesystems. Mounting S3 buckets is a very good use case. Sshfs and httpfs are more or less similar in this regard.

Yep, and WinFSP and dokany are two options for FUSE on Windows. I'd recommend using rclone or maybe check this list: https://winfsp.dev/doc/Known-File-Systems/

Re: Building and operating a pretty big storage system called S3

#138

Post by dosman33 » Fri, Jul 28, 2023, 2:06 PM UTC

supermatt wrote: post #113

How does S3 handle particularly hot objects? Is there some form of rebalancing to account for access rates?

I was disappointed too, this article was very light on details about the subject matter. I wasn't expecting a blue-print, but what was presented was all very hand-wavy.

In large systems (albeit smaller than S3) the way this works is that you slurp out some performance metrics from storage system to identify your hot spots and then feed that into a service that actively moves stuff around (below the namespace of the filesystem though, will be fs-dependant). You have some higher-performance disk pools at your disposal, and obviously that would be nvme storage today.

So in practice, it's likely proprietary vendor code chewing through performance data out of a proprietary storage controller and telling a worker job on a mounted filesystem client to move the hot data to the high performance disk pool. Always constantly rebalancing and moving data back out of the fast pool once it cools off. Obviously for S3 this is happening at an object level though using their own in-house code.

Re: Building and operating a pretty big storage system called S3

#139

Post by ruckfool » Fri, Jul 28, 2023, 2:36 PM UTC

wrboyce wrote: post #25

ldjkfkdsjnv wrote: post #20

Earlier quoted context omitted.

Also worked at Amazon, saw some issues with major well known open source libraries that broke in places nobody would ever expect.

Any examples you can share?

Redis Node failover

Re: Building and operating a pretty big storage system called S3

#140

Post by Twirrim » Fri, Jul 28, 2023, 2:43 PM UTC

jamesblonde wrote: post #104

Twirrim wrote: post #27

Earlier quoted context omitted.

Yes, I remember tcp checksumming coming up as not sufficient at one stage. Even saw S3 deal with a real head-scratcher of a non-impacting event that came down to a single NIC in a single machine corrupting the tcp checksum under very specific circumstances.

HDFS never relied on only network checksums. Blocks should be checksummed and validated at clients - a reliable end-to-end guarantee.

Well... yeah. S3 has checksums and all sorts of fixity checks right throughout. At no stage do they ever rely on a single mechanism. If there's one thing they're insanely paranoid about, it's data correctness and durability.

It has been several years, so I really don't remember much about the tcp checksum / corrupting NIC thing. Typically tcp checksum failures are handled entirely by the NIC, you wouldn't even notice it. My vague recollection was it coming up between two services not in the customer synchronous path (so e.g. not involved in getting data to or from the customer), and it caused something on the OS side.

I do remember that there was a contingent of engineers that were convinced it was a cosmic ray bit flip, which seems this whole thing certain types of engineers end up doing when presented with improbable seeming circumstances. It wasn't until it had happened a second or third time (weeks later) that they realised the origin machine was the same each time, and were able to dig in deeper to the point of reproduction.