51–60 of 79 posts
S3 may be my favorite AWS product. It's the epitome of "It just works" design.
https://www.allthingsdistributed.com/2010/05/amazon_s3_reduc...
> S3 is designed to provide “11 9’s” (99.999999999%) of durability Has anyone lost data on S3 or know anyone who has?
> tens of millions of requests per second How does something like that get handled? At some point isn't contacting this service really just contacting a handful of IP addresses? Are those just "regular" computers? Specialized hardware? Also with the amount of centralization here, presumably they are the target of bad actors. How do they defend against things like DDOS's to provide their claimed uptimes?
"Handful". S3 has 283 subnets allocated to it, biggest one being a /15, so there are some few hunderd thousand IP addresses reserved for S3. While impossible to say how many of them are actively in use, I imagine large fraction of them.
The AWS IP ranges are documented in https://ip-ranges.amazonaws.com/ip-ranges.json
Earlier quoted context omitted.
This recent interview with Werner Vogels[1] touches on S3 design priorities as well: https://cacm.acm.org/magazines/2021/3/250706-a-second-conver... https://news.ycombinator.com/item?id=26365873 [1] which you commented on, but others might not have seen.
>Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center, which gives them four 9s. If they're really good, they may have two data centers and actually know how to replicate over two data centers, and that gives them five 9s. But eleven 9's, in terms of durability, is just unparalleled I have a tough time taking him seriously reading this - I'm not sure i…
He said:
> Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center
You said:
> I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters
His customers don't equal your enterprise customers. You are both right most likely.
> That’s almost 13,000 objects for each person in the world, or 50 objects for every one of the roughly two trillion galaxies (according to this 2021 estimate) in the Universe. That’s mind boggling to think about. I wonder how much paper would’ve been needed to store 100T objects on paper. It’s like the new Library of Alexandria.
What’s way more mind boggling to me is that there are at least 260 galaxies per person in the world. I am not shocked at all that S3 is home to any number of cat pictures many of which are probably duplicates.
Earlier quoted context omitted.
This recent interview with Werner Vogels[1] touches on S3 design priorities as well: https://cacm.acm.org/magazines/2021/3/250706-a-second-conver... https://news.ycombinator.com/item?id=26365873 [1] which you commented on, but others might not have seen.
>Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center, which gives them four 9s. If they're really good, they may have two data centers and actually know how to replicate over two data centers, and that gives them five 9s. But eleven 9's, in terms of durability, is just unparalleled I have a tough time taking him seriously reading this - I'm not sure i…
Longtime consultant here with BCP/DR insight into 20+ large F500 companies... I think you would be seriously surprised at how common it is for even enterprise customers to not bother with multi-DC or even offsite backups. And even among the ones that do, many of the ones that are non-cloud-based are so immature at it that I would not put money on their backups being restorable if needed. And this is even more true for your “we’re a startup, we don’t have time to worry about backing up our data!” companies, which are in abundance.
For a small peak, go look at how many people were freaking out about losing their entire business due to the loss of a single OVH data center.
Of course, as a consultant I do naturally skew towards customers that need help with this stuff, so my perspective is probably biased towards the companies that are worse off in this regard. But they’re definitely out there.
One of the best things that made AWS what it is today. Sure it can be a little pricey compared to competitors but it was revolutionary when it came out and still is a great product. Also a lot of its competitors still lack certain key features that makes it hard to move away from s3. aws s3 sync ... And you are in business.
When people talk about avoiding vendor lock in with AWS, they tend to talk in terms of using k8s or another compute abstraction layer. But I think the real lock-in lies in services like S3, making so many workflows simple and intuitive (while controlling your data, which is where the real lock-in lies)