The things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to access data on the user's behalf. Google Drive is the closest to this but it only has a single provider and other issues[0]. I'm sad remoteStorage never caught on. I really hope Solid does well but it feels too complex to me. My own take on the problem is https://gemdrive.io/, but it's mostly on hold while I'm focused on other parts of the self-hosting stack.
Building and operating a pretty big storage system called S3
11–20 of 169 posts
Re: Building and operating a pretty big storage system called S3
#12> That’s a bit error rate of 1 in 10^15 requests. In the real world, we see that blade of grass get missed pretty frequently – and it’s actually something we need to account for in S3. One of the things I remember from my time at AWS was conversations about how 1 in a billion events end up being a daily occurrence when you're operating at S3 scale. Things that you'd normally mark off as so wildly improbable it's not…
I think Ceph hit similar problems and they had to add more robust checksumming to the system, as relying on just tcp checksums for integrity for example was no longer enough
Re: Building and operating a pretty big storage system called S3
#13> What’s interesting here, when you look at the highest-level block diagram of S3’s technical design, is the fact that AWS tends to ship its org chart. This is a phrase that’s often used in a pretty disparaging way, but in this case it’s absolutely fascinating.
I’d go even further: at this scale, it is essential and required to develop these kind of projects with any sort of velocity.
Large organizations ship their communication structure by design. The alternative is engineering anarchy.
Re: Building and operating a pretty big storage system called S3
#14> Now, let’s go back to that first hard drive, the IBM RAMAC from 1956. Here are some specs on that thing: > Storage Capacity: 3.75 MB > Cost: ~$9,200/terabyte Those specs can't possibly be correct. If you multiply the cost by the storage, the cost of the drive works out to 3¢. This site[1] states, > It stored about 2,000 bits of data per square inch and had a purchase price of about $10,000 per megabyte So perhaps t…
https://en.m.wikipedia.org/wiki/IBM_305_RAMAC has the likely source of the error: 30M bits (using the 6 data bits but not parity), but it rented for $3k per month so you didn’t have a set cost the same as buying a physical drive outright - very close to S3’s model, though.
Re: Building and operating a pretty big storage system called S3
#15> Now, let’s go back to that first hard drive, the IBM RAMAC from 1956. Here are some specs on that thing: > Storage Capacity: 3.75 MB > Cost: ~$9,200/terabyte Those specs can't possibly be correct. If you multiply the cost by the storage, the cost of the drive works out to 3¢. This site[1] states, > It stored about 2,000 bits of data per square inch and had a purchase price of about $10,000 per megabyte So perhaps t…
oh shoot. good catch, thanks!
Re: Building and operating a pretty big storage system called S3
#16> Now, let’s go back to that first hard drive, the IBM RAMAC from 1956. Here are some specs on that thing: > Storage Capacity: 3.75 MB > Cost: ~$9,200/terabyte Those specs can't possibly be correct. If you multiply the cost by the storage, the cost of the drive works out to 3¢. This site[1] states, > It stored about 2,000 bits of data per square inch and had a purchase price of about $10,000 per megabyte So perhaps t…
Must've put a decimal point in the wrong place or something. I always do that. I always mess up some mundane detail.
Re: Building and operating a pretty big storage system called S3
#17> What’s interesting here, when you look at the highest-level block diagram of S3’s technical design, is the fact that AWS tends to ship its org chart. This is a phrase that’s often used in a pretty disparaging way, but in this case it’s absolutely fascinating. I’d go even further: at this scale, it is essential and required to develop these kind of projects with any sort of velocity. Large organizations ship their c…
This is also why reorgs tend to be pretty common at large tech orgs.
They know they'll almost inevitably ship their org chart. And they'll encounter tons of process-based friction if they don't.
The solution: Change your org chart to match what you want to ship
Re: Building and operating a pretty big storage system called S3
#18Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).
Re: Building and operating a pretty big storage system called S3
#19Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).
[deleted]
Re: Building and operating a pretty big storage system called S3
#20> That’s a bit error rate of 1 in 10^15 requests. In the real world, we see that blade of grass get missed pretty frequently – and it’s actually something we need to account for in S3. One of the things I remember from my time at AWS was conversations about how 1 in a billion events end up being a daily occurrence when you're operating at S3 scale. Things that you'd normally mark off as so wildly improbable it's not…
Also worked at Amazon, saw some issues with major well known open source libraries that broke in places nobody would ever expect.