Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

aws.typepad.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

#51

Post by amock »

fooyc wrote: post #21
tomschlick wrote: post #18

Anyone else find this surprisingly low? I'd imagine your typical web service holds a few thousand objects in S3 for images etc, then backups and anything else. Then you have your big players like netflix, dropbox etc that use the service. Who store data for tens of millions of customers...

1.1 Million Requests/Second seems especially low for Amazon. The average server can serve more than 1,000 static objects / second easily.

There is more to S3 than just serving static objects off of the local drive. Maintaining the integrity of the data and ensuring that the data sent out is consistent is not a trivial task. A constantly changing map with 2 trillion keys is a hard problem on its own. Also, serving 1000 1MiB objects per second is not the same as serving 1000 1KiB objects per second so it's hard to say how many resources just the serving portion consumes.

Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

#52

Post by johnbm »

lsb wrote: post #50
jeffbarr wrote: post #26

Earlier quoted context omitted.

I'm open to more creative analogies. I'll happily send some AWS stickers to the first 5 truly great ones that show up under this reply...

If each object is a megabyte, you could stick them on microSD cards packed in a cube as tall as an adult. (1 card is 0.11cm x 0.15cm x 0.01cm, 32GB. 32k objects, 31.2M cards needed. 165 cm x 165 cm x 165cm = 110 * 150 * 1650 cards = 27.2M. Instead of 165cm which is 5 and a half feet, say 6 feet, so try 166 * 122 * 1829 = 37.04M, enough to use some error-correcting codes just in case.)

I like to use dice to think about storage. If a byte is the size of a die (let's say 1cm^3), then a kilobyte is a 10x10x10 cm^3 cube. A megabyte is a 1m^3 crate, a gigabyte is a 10m^3 house, a terabyte is a 100m^3 tall building, a petabyte is a 1km wide borg cube etc.

Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

#53

Post by NewAccnt2 »

charlesju wrote: post #20
tomschlick wrote: post #18

Anyone else find this surprisingly low? I'd imagine your typical web service holds a few thousand objects in S3 for images etc, then backups and anything else. Then you have your big players like netflix, dropbox etc that use the service. Who store data for tens of millions of customers...

I think you are desensitized to what a trillion means. http://demonocracy.info/infographics/usa/us_debt/us_debt.htm...

Or ~1 year of money at the current rate of money being printed by the fed ($85 Billion / month)

Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

#54

Post by kondro »

fooyc wrote: post #21
tomschlick wrote: post #18

Anyone else find this surprisingly low? I'd imagine your typical web service holds a few thousand objects in S3 for images etc, then backups and anything else. Then you have your big players like netflix, dropbox etc that use the service. Who store data for tens of millions of customers...

1.1 Million Requests/Second seems especially low for Amazon. The average server can serve more than 1,000 static objects / second easily.

If you ignore the fact that these aren't actually static objects and require a lot more computation to work out where they are and where they need to go.

1.1M RPS is the amount they actually serve, not how much they can serve. Just because your single server can serve more than 1,000 static objects/second (in fact, that number should be much higher), it doesn't mean you need to.

Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

#55

Post by angersock »

HeyImAlex wrote: post #4
MartinMond wrote: post #2

So how many objects should have been corrupted/lost according to their SLA and how many actually did get corrupted?

>how many objects should have been corrupted/lost according to their SLA Anywhere from 20 to 200M per year depending on how many people use RRS...

Do objects tend to get lost in clusters, or what?

Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

#56

Post by upthedale »

Wow, Windows Azure seems to be beating it by a long way. Genuine surprise... I assumed it would be much closer.

9 months ago, they announced they stored twice this amount - 4 trillion objects. A year before that, 1 Trillion. Given that previous rate of growth, we can expect they have a lot more than this now.

They also announced peaks of 880,000 requests a second. Whilst Amazon wins here, I'd say its fair to assume this number has increased in those 9 months.

http://blogs.msdn.com/b/windowsazurestorage/archive/2012/07/...

Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

#57

Post by theatraine »

upthedale wrote: post #56

Wow, Windows Azure seems to be beating it by a long way. Genuine surprise... I assumed it would be much closer. 9 months ago, they announced they stored twice this amount - 4 trillion objects. A year before that, 1 Trillion. Given that previous rate of growth, we can expect they have a lot more than this now. They also announced peaks of 880,000 requests a second. Whilst Amazon wins here, I'd say its fair to assume t…

Bear in mind that Azure storage includes "Blobs, Disks/Drives, Tables, and Queues" however S3 is only blobs (other services like Amazon's DynamoDB would be analogous to table storage). Hence, it's not an apples-to-apples comparison.

Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

#58

Post by upthedale »

theatraine wrote: post #57
upthedale wrote: post #56

Wow, Windows Azure seems to be beating it by a long way. Genuine surprise... I assumed it would be much closer. 9 months ago, they announced they stored twice this amount - 4 trillion objects. A year before that, 1 Trillion. Given that previous rate of growth, we can expect they have a lot more than this now. They also announced peaks of 880,000 requests a second. Whilst Amazon wins here, I'd say its fair to assume t…

Bear in mind that Azure storage includes "Blobs, Disks/Drives, Tables, and Queues" however S3 is only blobs (other services like Amazon's DynamoDB would be analogous to table storage). Hence, it's not an apples-to-apples comparison.

Fair point. Do we have any other numbers that might make it a more apples-to-apples comparison?

Re: Amazon S3 – 2 Trillion Objects, 1.1 Million Requests/Second

#59

Post by jacques_chester »

jl6 wrote: post #40
Smrchy wrote: post #3

It would be really interesting to know the average size of an object and visualize the amount of harddisks it takes to store all this data.

You could then multiply by 2 trillion to work out their total data volume under management, which I believe they consider commercially sensitive information, but I can't quite put my finger on why.

It would help competitors form a view of their cost structure, which would allow them to optimise a price which would put Amazon into the red.
Post reply on HN