41–50 of 63 posts
Earlier quoted context omitted.
I'm open to more creative analogies. I'll happily send some AWS stickers to the first 5 truly great ones that show up under this reply...
My math is probably wrong... but I believe if you ate a twinkie for every request at the end of a year it would take 1,350 Blue Marlin heavy lift ships to move you across the ocean.
Anyone know what technology S3 uses to meet this kind of demand?
Jk... but seriously, good question! it's gotta just be a massive feet of engineering
"If you added one S3 object every 60 hours starting at the Big Bang, you'd have accumulated almost two trillion of them by now." That actually sounds underwhelming! IMHO our brains have an easier time thinking "hey, 1 every 60 hours that's not much" compared to figuring out the universe is really incredibly old ;-)
I'm open to more creative analogies. I'll happily send some AWS stickers to the first 5 truly great ones that show up under this reply...
"If you added one S3 object every 60 hours starting at the Big Bang, you'd have accumulated almost two trillion of them by now." That actually sounds underwhelming! IMHO our brains have an easier time thinking "hey, 1 every 60 hours that's not much" compared to figuring out the universe is really incredibly old ;-)
I'm open to more creative analogies. I'll happily send some AWS stickers to the first 5 truly great ones that show up under this reply...
In fact, even if you and all your friends spent your lives reading the lists of object names, you wouldn't have time to get to the end.
In fact, even if you and all your friends and all of their friends devoted your lives to reading lists of S3 object names, you wouldn't even make a dent because new objects are arriving faster than you could collectively read their names.
Earlier quoted context omitted.
To put this into perspective it's 2.7B new objects per day (assuming 1 Trillion objects averaged over 365 days). Assuming each object is 100KB (generous estimate, after compression) that would be 270GB per day -- or assuming ten levels of redundancy and striped across three RAID storage devices (per level of redundancy) then 8.1TB per day. I'm not familiar with their hard disk procurement policies but it wouldn't be…
I'm not convinced that 100KB is a great estimate on file size, but either way you're off by a few zeroes. It's not 270GB per day, it's 270TB. Even if each object were just one byte, that would be 2.7GB. 100KB is one hundred thousand bytes. So it's quite a bit more than eleven drives per day!
After applying the same shoddy math with each object being 100KB -- 270TB with 10 levels of redundancy across 3 RAID drives resulting in 8,100TB per day. This would be 8,100 drives (at 1TB per drive), or 8,910 drives after 10% being dead-on-arrival.
The math is sketchy, so let's cut it down by 10x (10KB per object): 891 drives per day. Keep in mind this is just for S3 and it doesn't account for existing drives failing, growth, or what other services require (eg: EC2, RDS, Cloudwatch, Cloudfront, etc).
Anyone else find this surprisingly low? I'd imagine your typical web service holds a few thousand objects in S3 for images etc, then backups and anything else. Then you have your big players like netflix, dropbox etc that use the service. Who store data for tens of millions of customers...
Netflix is unlikely to store customer data on S3. They use it for movie files IIRC, but customer data would go in a proper database of some sort. Dropbox does use S3, but I think you're probably doing the fairly standard human mistake of not realizing just how big a trillion is. Dropbox hit 100M users in November of 2012, so for Dropbox to use up two trillion objects each user would need to have 20,000 of them. Dropb…
(At one point, the stuff I was storing/logging for Cydia actually represented over a percent, maybe it was even over two percent, of all objects in S3; now I'm between 0.1% and 0.5%.)
"If you added one S3 object every 60 hours starting at the Big Bang, you'd have accumulated almost two trillion of them by now." That actually sounds underwhelming! IMHO our brains have an easier time thinking "hey, 1 every 60 hours that's not much" compared to figuring out the universe is really incredibly old ;-)
I'm open to more creative analogies. I'll happily send some AWS stickers to the first 5 truly great ones that show up under this reply...
Earlier quoted context omitted.
I'm open to more creative analogies. I'll happily send some AWS stickers to the first 5 truly great ones that show up under this reply...
I have a feeling that having 1/8 as many objects as there are dollars in the US National Debt might hit home right about now. ;) Not least people will have seen the many, many visualisations of that figure. :P
If you had $8 for every object you could wipe out US National Debt.
Earlier quoted context omitted.
An object is a single blob of data. S3 objects vary in size from 1 byte up to 5 Terabytes. They can be uploaded with a single PUT, or with multiple PUTs in series or in parallel (which we call multipart upload). They can be downloaded as a unit, in full (GET) or in part (range GET).
Thanks jeffbarr for valuable information...
"If you added one S3 object every 60 hours starting at the Big Bang, you'd have accumulated almost two trillion of them by now." That actually sounds underwhelming! IMHO our brains have an easier time thinking "hey, 1 every 60 hours that's not much" compared to figuring out the universe is really incredibly old ;-)
I'm open to more creative analogies. I'll happily send some AWS stickers to the first 5 truly great ones that show up under this reply...
(1 card is 0.11cm x 0.15cm x 0.01cm, 32GB. 32k objects, 31.2M cards needed. 165 cm x 165 cm x 165cm = 110 * 150 * 1650 cards = 27.2M. Instead of 165cm which is 5 and a half feet, say 6 feet, so try 166 * 122 * 1829 = 37.04M, enough to use some error-correcting codes just in case.)