What is an 'object' in this sense?
Anything stored in S3
Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
11–20 of 63 posts
Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
#12what do u mean by objects?
Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
#13What is an 'object' in this sense?
A file
Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
#14what do u mean by objects?
An object is a single blob of data. S3 objects vary in size from 1 byte up to 5 Terabytes. They can be uploaded with a single PUT, or with multiple PUTs in series or in parallel (which we call multipart upload). They can be downloaded as a unit, in full (GET) or in part (range GET).
Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
#15So how many objects should have been corrupted/lost according to their SLA and how many actually did get corrupted?
>how many objects should have been corrupted/lost according to their SLA Anywhere from 20 to 200M per year depending on how many people use RRS...
It's staggering that out of 2 trillion only 20 might be corrupted, assuming they lived up to their SLA.
2000000000000 - (.99999999999 * 2000000000000) = 20
Makes me feel pretty OK about having backups there.
Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
#16what do u mean by objects?
An object is a single blob of data. S3 objects vary in size from 1 byte up to 5 Terabytes. They can be uploaded with a single PUT, or with multiple PUTs in series or in parallel (which we call multipart upload). They can be downloaded as a unit, in full (GET) or in part (range GET).
Thanks jeffbarr for valuable information...
Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
#17It would be really interesting to know the average size of an object and visualize the amount of harddisks it takes to store all this data.
Average size is irrelevant. A handful of 1G objects dwarf hundreds of 1 byte objects when computing the average. The overal distribution is interesting. There are actually three: GET sizes, PUT sizes, and stored sizes. They are not identical distributions, especially since the as the PUT size distribution has changed it becomes out of synch with the stored size distribution. Wish I could tell you more, there are some fascinating data points in there but, you know, NDA. Source: form S3 employee.
Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
#18Anyone else find this surprisingly low? I'd imagine your typical web service holds a few thousand objects in S3 for images etc, then backups and anything else. Then you have your big players like netflix, dropbox etc that use the service. Who store data for tens of millions of customers...
Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
#19Anyone else find this surprisingly low? I'd imagine your typical web service holds a few thousand objects in S3 for images etc, then backups and anything else. Then you have your big players like netflix, dropbox etc that use the service. Who store data for tens of millions of customers...
Netflix is unlikely to store customer data on S3. They use it for movie files IIRC, but customer data would go in a proper database of some sort.
Dropbox does use S3, but I think you're probably doing the fairly standard human mistake of not realizing just how big a trillion is. Dropbox hit 100M users in November of 2012, so for Dropbox to use up two trillion objects each user would need to have 20,000 of them. Dropbox does deduplication, has lots of inactive/minimal users, etc., so they're probably a percent or two of S3 objects.
Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second
#20Anyone else find this surprisingly low? I'd imagine your typical web service holds a few thousand objects in S3 for images etc, then backups and anything else. Then you have your big players like netflix, dropbox etc that use the service. Who store data for tens of millions of customers...
I think you are desensitized to what a trillion means.
http://demonocracy.info/infographics/usa/us_debt/us_debt.htm...