Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

aws.amazon.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

#31

Post by jodrellblank »

simonebrunozzi wrote: post #28

If you're curious about some "S3 internals", this is a talk I gave in 2009 [0], based on a ton of interesting stuff that S3's then-GM Alyssa Henry [1] shared with me. [0]: https://vimeo.com/7330740 [1]: http://itc.conversationsnetwork.org/shows/detail5273.html

This recent interview with Werner Vogels[1] touches on S3 design priorities as well:

https://cacm.acm.org/magazines/2021/3/250706-a-second-conver...

https://news.ycombinator.com/item?id=26365873

[1] which you commented on, but others might not have seen.

Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

#32

Post by herodoturtle »

PaulHoule wrote: post #25

I am worried. We will start having trouble with random UUID collisions when we have made 2^64 of them, which is a huge number. It's about as many iron atoms as are in an iron filing. It is also about 200,000 times 100 trillion, so AWS is just 18 doubling times away from the UUID system breaking down.

At this rate, won't AWS run out of storage space before they run out of UUIDs?

Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

#33

Post by PaulHoule »

herodoturtle wrote: post #32
PaulHoule wrote: post #25

I am worried. We will start having trouble with random UUID collisions when we have made 2^64 of them, which is a huge number. It's about as many iron atoms as are in an iron filing. It is also about 200,000 times 100 trillion, so AWS is just 18 doubling times away from the UUID system breaking down.

At this rate, won't AWS run out of storage space before they run out of UUIDs?

or the waste heat boils the oceans... or something

Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

#34

Post by codegeek »

One of the best things that made AWS what it is today. Sure it can be a little pricey compared to competitors but it was revolutionary when it came out and still is a great product. Also a lot of its competitors still lack certain key features that makes it hard to move away from s3.

   aws s3 sync ...
And you are in business.

Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

#35

Post by pradn »

PaulHoule wrote: post #25

I am worried. We will start having trouble with random UUID collisions when we have made 2^64 of them, which is a huge number. It's about as many iron atoms as are in an iron filing. It is also about 200,000 times 100 trillion, so AWS is just 18 doubling times away from the UUID system breaking down.

Is this a real problem? It should be possible for Amazon to update their code to generate IDs a different way, and for them to update their client libraries well ahead of a time when this becomes a problem. Yes, code that relies on the ID being a particular format will break. But users have to deal with deprecated client library features all the time. It's reasonable to give a few year grace period before old APIs are shut down.

UUIDs are usually not simply random numbers. So it's possible to tell when a particular standard of UUID is used. Amazon's code can branch based on this detection, or even something simpler like "UUID-type", a new field that would be initialized by default to the old UUID type.

I simply don't see this as a problem for anyone.

Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

#37

Post by donavanm »

ZiiS wrote: post #21
TedDoesntTalk wrote: post #4

> S3 is designed to provide “11 9’s” (99.999999999%) of durability Has anyone lost data on S3 or know anyone who has?

If it works as designed I guess 1000 have; but how would you know. Did your perfect code running on an perfect CPU and RAM connected though a perfect network upload the file with 11 9's certainty?

This is a real problem, but it is partially accounted for:

> To ensure that data is not corrupted traversing the network, use the Content-MD5 header. When you use this header, Amazon S3 checks the object against the provided MD5 value and, if they do not match, returns an error. Additionally, you can calculate the MD5 while putting an object to Amazon S3 and compare the returned ETag to the calculated MD5 value.

It’s not as obvious with multipart uploads where each chunk is (iirc) individually checksumd and stored, but only a single md5 is returned in the “etag”

Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

#38

Post by gonzo41 »

PaulHoule wrote: post #25

I am worried. We will start having trouble with random UUID collisions when we have made 2^64 of them, which is a huge number. It's about as many iron atoms as are in an iron filing. It is also about 200,000 times 100 trillion, so AWS is just 18 doubling times away from the UUID system breaking down.

All AWS need to do is add something like a namespace. Shazzam! problem solved for your lifetime.

Re: Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects

#40

Post by throwawaysea »

> tens of millions of requests per second

How does something like that get handled? At some point isn't contacting this service really just contacting a handful of IP addresses? Are those just "regular" computers? Specialized hardware?

Also with the amount of centralization here, presumably they are the target of bad actors. How do they defend against things like DDOS's to provide their claimed uptimes?

Post reply on HN