Earlier quoted context omitted.
There is no way
prove me wrong.
You have the burden of proof for your claim.
Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
41–50 of 79 posts
Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
#42Do they de-dupe at all?
no
Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
#43One of the best things that made AWS what it is today. Sure it can be a little pricey compared to competitors but it was revolutionary when it came out and still is a great product. Also a lot of its competitors still lack certain key features that makes it hard to move away from s3. aws s3 sync ... And you are in business.
When people talk about avoiding vendor lock in with AWS, they tend to talk in terms of using k8s or another compute abstraction layer.
But I think the real lock-in lies in services like S3, making so many workflows simple and intuitive (while controlling your data, which is where the real lock-in lies)
Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
#44> That’s almost 13,000 objects for each person in the world, or 50 objects for every one of the roughly two trillion galaxies (according to this 2021 estimate) in the Universe. That’s mind boggling to think about. I wonder how much paper would’ve been needed to store 100T objects on paper. It’s like the new Library of Alexandria.
What’s way more mind boggling to me is that there are at least 260 galaxies per person in the world. I am not shocked at all that S3 is home to any number of cat pictures many of which are probably duplicates.
Found the IPFS user.
Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
#45Do they de-dupe at all?
Yes. The contents of zero-byte objects are deduplicated. (The metadata, of course, are not.)
Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
#46One of the best things that made AWS what it is today. Sure it can be a little pricey compared to competitors but it was revolutionary when it came out and still is a great product. Also a lot of its competitors still lack certain key features that makes it hard to move away from s3. aws s3 sync ... And you are in business.
When people talk about avoiding vendor lock in with AWS, they tend to talk in terms of using k8s or another compute abstraction layer. But I think the real lock-in lies in services like S3, making so many workflows simple and intuitive (while controlling your data, which is where the real lock-in lies)
Minio and other S3-compatibles make moving off of AWS that much simpler, though obviously it might be difficult if you're using Amazon-only S3 features extensively.
Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
#47If you're curious about some "S3 internals", this is a talk I gave in 2009 [0], based on a ton of interesting stuff that S3's then-GM Alyssa Henry [1] shared with me. [0]: https://vimeo.com/7330740 [1]: http://itc.conversationsnetwork.org/shows/detail5273.html
This recent interview with Werner Vogels[1] touches on S3 design priorities as well: https://cacm.acm.org/magazines/2021/3/250706-a-second-conver... https://news.ycombinator.com/item?id=26365873 [1] which you commented on, but others might not have seen.
>Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center, which gives them four 9s. If they're really good, they may have two data centers and actually know how to replicate over two data centers, and that gives them five 9s. But eleven 9's, in terms of durability, is just unparalleled
I have a tough time taking him seriously reading this - I'm not sure if he's being intentionally ignorant or intentionally misleading.
I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters, and then backed up to some other medium (tape or disk-based backup appliance) for anything business critical.
Their goal is 5-9's of UPTIME - the expectation is 100% data durability. You would get fired if you architected a solution keeping two copies of data in one datacenter.
Object storage absolutely has a place and I'm happy amazon was able to push a quasi-standard for the industry. Object storage prior to S3 was a hodge-podge of proprietary plays (EMC Centera, Bytecast, etc) and sort-of standards that nobody really used (CDMI). I just wish they did a better job of fairly representing it vs. turning every opportunity into a sales pitch spreading FUD about the alternatives.
Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
#48Earlier quoted context omitted.
When people talk about avoiding vendor lock in with AWS, they tend to talk in terms of using k8s or another compute abstraction layer. But I think the real lock-in lies in services like S3, making so many workflows simple and intuitive (while controlling your data, which is where the real lock-in lies)
Minio and other S3-compatibles make moving off of AWS that much simpler, though obviously it might be difficult if you're using Amazon-only S3 features extensively.
Especially if you layer on additional functionality, like triggering a lambda function to resize an image when it hits the S3 bucket.
Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
#49Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects
#50> tens of millions of requests per second How does something like that get handled? At some point isn't contacting this service really just contacting a handful of IP addresses? Are those just "regular" computers? Specialized hardware? Also with the amount of centralization here, presumably they are the target of bad actors. How do they defend against things like DDOS's to provide their claimed uptimes?
I’m wildly guessing here, but I assume that there are many tiers of load balancing to turn your initial request into the message that reaches the thing that gives back your file data. They also likely do have specialized hardware in various parts of the AWS infrastructure.
A service that can handle tens of millions of requests per second can probably outscale most DDoS attacks without further engineering.