Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building and operating a pretty big storage system called S3

allthingsdistributed.com

Post reply on HN
Print view View original

121–130 of 169 posts

Re: Building and operating a pretty big storage system called S3

#121

Post by benou »

ilyt wrote: post #12
Twirrim wrote: post #2

> That’s a bit error rate of 1 in 10^15 requests. In the real world, we see that blade of grass get missed pretty frequently – and it’s actually something we need to account for in S3. One of the things I remember from my time at AWS was conversations about how 1 in a billion events end up being a daily occurrence when you're operating at S3 scale. Things that you'd normally mark off as so wildly improbable it's not…

I think Ceph hit similar problems and they had to add more robust checksumming to the system, as relying on just tcp checksums for integrity for example was no longer enough

Not that surprising, given this was already extensively documented in the 2000's (so already widely known by then) with iSCSI and such, see https://www.rfc-editor.org/rfc/rfc3385 for example.

Re: Building and operating a pretty big storage system called S3

#122

Post by mannyv »

awithrow wrote: post #89
mannyv wrote: post #72

What most people don't realize is that the magic isn't in handling the system itself; the magic is making authorization appear to be zero-cost. In distributed systems authorization is incredibly difficult. At the scale of AWS it might as well be magic. AWS has a rich permissions model with changes to authorization bubbling through the infrastructure at sub-millisecond speed - while handling probably trillions of requ…

Keep in mind that S3 predates IAM by several years. So part of the reason that access to buckets/keys is special is because it was already in place by the time IAM came around. Its likely persisted since than largely since removing the old model would be a difficult taks without potentially breaking a lot of customer's setup

Exactly. This difference makes it easier to (1)understand how IAM works, and (2) how the s3 works...because IAM and S3 work together, but in a different way than the other services.

I heard that AA is done via asics, but resource-level permissions implies that authorization is done at the local level for s3. To me that implies that the system extracts S3 permissions from IAM and sends them downstream s3, which get merged with stuff that s3 manages.

I guess that occurs when permissions are saved up in IAM world. At some point those need to be joined against a principal somewhere, as roles can exist without assignment.

Again, it's be so interesting to see how this is done IRL.

Re: Building and operating a pretty big storage system called S3

#123

Post by fooker »

rkagerer wrote: post #36
Twirrim wrote: post #2

> That’s a bit error rate of 1 in 10^15 requests. In the real world, we see that blade of grass get missed pretty frequently – and it’s actually something we need to account for in S3. One of the things I remember from my time at AWS was conversations about how 1 in a billion events end up being a daily occurrence when you're operating at S3 scale. Things that you'd normally mark off as so wildly improbable it's not…

Personally I'd love working in that kind of environment. That one in a billion hole still itches at me. There's also a slightly-perverse little voice in my head ready with popcorn in case I'm lucky enough to watch the ensuing fallout from the first major crypto hash collision :-).

That probability is significantly lower than one in a billion.

One in a billion would be if keys were ~30 bits. Luckily it isn't.

Re: Building and operating a pretty big storage system called S3

#124

Post by abwizz »

[deleted] wrote: post #82
Waterluvian wrote: post #10

Earlier quoted context omitted.

Ever see a UUID collision?

[deleted]

> two bit flips in the same tcp packet cancel each other out and cause the checksum to pass

checksum != parity check

not sure if there even exists a chance for this to happen

Re: Building and operating a pretty big storage system called S3

#125

Post by afr0ck »

jamesblonde wrote: post #107
apitman wrote: post #11

The things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to access data on the user's behalf. Google Drive is the closest to this but it only has a single provider and other issues[0]. I'm sad remoteStorage never caught on. I really hope Solid does well but it feels too complex to me. My own take on the problem is https:…

Most apps, however, assume POSIX-like data access. I would love to see a client-side minimally dependent library that mounts a local directory that is actually the user's S3 bucket.

Linux has FUSE, which is a framework to develop user-level filesystems. Mounting S3 buckets is a very good use case. Sshfs and httpfs are more or less similar in this regard.

Re: Building and operating a pretty big storage system called S3

#126

Post by raverbashing »

[deleted] wrote: post #82
Waterluvian wrote: post #10

Earlier quoted context omitted.

Ever see a UUID collision?

[deleted]

Wow this is at the level of Homer Simpson "Cereal with Milk catching fire"

But yeah, mathematically possible (in AWS scale, but still) so of course it will happen once in a lifetime.

Re: Building and operating a pretty big storage system called S3

#127

Post by PaulRobinson »

Twirrim wrote: post #2

> That’s a bit error rate of 1 in 10^15 requests. In the real world, we see that blade of grass get missed pretty frequently – and it’s actually something we need to account for in S3. One of the things I remember from my time at AWS was conversations about how 1 in a billion events end up being a daily occurrence when you're operating at S3 scale. Things that you'd normally mark off as so wildly improbable it's not…

Was an SDM of a team of brand new SDEs standing up a new service. In a code review, pointed to an issue that could cause a Sev2, and the SDE pushed back "that's like one in a million chance, at most". Pointed out once we were dialled up to 500k TPS (which is where we needed to be at), that was 30 times a minute... "You want to be on call that week?". Insist on Highest Standards takes on a different meaning in that stack compared to most orgs.

Re: Building and operating a pretty big storage system called S3

#128

Post by aitchnyu »

bombcar wrote: post #65
anyoneamous wrote: post #37

Earlier quoted context omitted.

My impression is that the ambiguity gives them freedom to implement in different ways across different regions and over time. The original Glacier was very clearly tape, but given the instant retrieval capabilities the newer S3-Glacier tiers are most likely just low-margin HDDs, maybe with some dynamic powering on and off of drives/servers.

Glacier is just run on S3 with some sleep statements added.

The perceived value of results is higher if it takes a longer time to load, users feel the computer is hard at work. If its true for flight searches, its true for backup systems.

Re: Building and operating a pretty big storage system called S3

#129

Post by romantomjak »

Apologies if this comes off as blunt, but this is the type of content I come to read at hacker news rather than it being just a series of obituaries.

The author has made a lot of great points, but one that stuck with me was:

> I consciously spend a lot more time trying to develop problems, and to do a really good job of articulating them, rather than trying to pitch solutions.

I haven’t thought of it in this way, but this is an excellent way of motivating someone to “own” a problem.

Re: Building and operating a pretty big storage system called S3

#130

Post by CrQuYt6fWiUe2 »

jl6 wrote: post #18

Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).

It's just low powered hard drives that aren't turned on all the time. Nothing special.
Post reply on HN