Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building and operating a pretty big storage system called S3

allthingsdistributed.com

Post reply on HN
Print view View original

81–90 of 169 posts

Re: Building and operating a pretty big storage system called S3

#81

Post by jandrewrogers »

mabbo wrote: post #35
Waterluvian wrote: post #10

Earlier quoted context omitted.

Ever see a UUID collision?

Even at a billion requests per second, 128 bit UUIDs shouldn't collide for something like a billion years. And that's if you're going completely random and not taking care to try to reduce collisions.

There have been many cases of UUIDv4 collisions because an RNG wasn’t as random as expected, due to broken RNG or developer error. It is one of those cases where practice is not as reliable as theory, and it is banned in some places as a consequence.

It depends on how paranoid you need to be.

Re: Building and operating a pretty big storage system called S3

#82

Post by [deleted] »

Waterluvian wrote: post #10
Twirrim wrote: post #2

> That’s a bit error rate of 1 in 10^15 requests. In the real world, we see that blade of grass get missed pretty frequently – and it’s actually something we need to account for in S3. One of the things I remember from my time at AWS was conversations about how 1 in a billion events end up being a daily occurrence when you're operating at S3 scale. Things that you'd normally mark off as so wildly improbable it's not…

Ever see a UUID collision?

[deleted]

Re: Building and operating a pretty big storage system called S3

#83

Post by lazide »

mabbo wrote: post #35
Waterluvian wrote: post #10

Earlier quoted context omitted.

Ever see a UUID collision?

Even at a billion requests per second, 128 bit UUIDs shouldn't collide for something like a billion years. And that's if you're going completely random and not taking care to try to reduce collisions.

Shouldn’t != never happens. All sorts of weird implementation issues can cause problems.

Re: Building and operating a pretty big storage system called S3

#85

Post by mr_toad »

hobo_in_library wrote: post #17
baq wrote: post #13

> What’s interesting here, when you look at the highest-level block diagram of S3’s technical design, is the fact that AWS tends to ship its org chart. This is a phrase that’s often used in a pretty disparaging way, but in this case it’s absolutely fascinating. I’d go even further: at this scale, it is essential and required to develop these kind of projects with any sort of velocity. Large organizations ship their c…

This is also why reorgs tend to be pretty common at large tech orgs. They know they'll almost inevitably ship their org chart. And they'll encounter tons of process-based friction if they don't. The solution: Change your org chart to match what you want to ship

A more cynical take is that it makes it look like the new management is doing something.

An even more cynical take is that it makes it difficult to compare performance with past performance.

Re: Building and operating a pretty big storage system called S3

#86

Post by pkaye »

jl6 wrote: post #18

Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).

Glacier was originally using actual glaciers as a storage media since they have been around forever. Bu then climate change happened so they quickly shifted to tiered storage of tape and hard drives.

Re: Building and operating a pretty big storage system called S3

#87

Post by rubiquity »

Twirrim wrote: post #30
rubiquity wrote: post #8

Earlier quoted context omitted.

Daily? A component I worked on that supported S3’s Index could hit a 1 in a billion issue multiple times a minute. Thankfully we had good algorithms and hardware that is a lot more reliable these days!

This was 7-8 years ago now. Lot of scaling up since those days :)

I’m sure my numbers are out of date now too

Re: Building and operating a pretty big storage system called S3

#88

Post by ldjkfkdsjnv »

wrboyce wrote: post #25
ldjkfkdsjnv wrote: post #20

Earlier quoted context omitted.

Also worked at Amazon, saw some issues with major well known open source libraries that broke in places nobody would ever expect.

Any examples you can share?

Apache tomcat starts to break down

Re: Building and operating a pretty big storage system called S3

#89

Post by awithrow »

mannyv wrote: post #72

What most people don't realize is that the magic isn't in handling the system itself; the magic is making authorization appear to be zero-cost. In distributed systems authorization is incredibly difficult. At the scale of AWS it might as well be magic. AWS has a rich permissions model with changes to authorization bubbling through the infrastructure at sub-millisecond speed - while handling probably trillions of requ…

Keep in mind that S3 predates IAM by several years. So part of the reason that access to buckets/keys is special is because it was already in place by the time IAM came around.

Its likely persisted since than largely since removing the old model would be a difficult taks without potentially breaking a lot of customer's setup

Post reply on HN