61–63 of 63 posts
Earlier quoted context omitted.
I have a feeling that having 1/8 as many objects as there are dollars in the US National Debt might hit home right about now. ;) Not least people will have seen the many, many visualisations of that figure. :P
Let's put a positive slant on it ... If you had $8 for every object you could wipe out US National Debt.
Earlier quoted context omitted.
1.1 Million Requests/Second seems especially low for Amazon. The average server can serve more than 1,000 static objects / second easily.
There is more to S3 than just serving static objects off of the local drive. Maintaining the integrity of the data and ensuring that the data sent out is consistent is not a trivial task. A constantly changing map with 2 trillion keys is a hard problem on its own. Also, serving 1000 1MiB objects per second is not the same as serving 1000 1KiB objects per second so it's hard to say how many resources just the serving…
I was assuming that the number of requests referred to read requests, and I guess that the system is designed in a way that makes read requests very cheap, maybe even cheaper than reading a file on a usual filesystem, at least for hot data.
Just because it's huge and complex doesn't mean it's slow and that requests are expensive.
Earlier quoted context omitted.
1.1 Million Requests/Second seems especially low for Amazon. The average server can serve more than 1,000 static objects / second easily.
If you ignore the fact that these aren't actually static objects and require a lot more computation to work out where they are and where they need to go. 1.1M RPS is the amount they actually serve, not how much they can serve. Just because your single server can serve more than 1,000 static objects/second (in fact, that number should be much higher), it doesn't mean you need to.
It depends what you call a static object.
If the content of your object is stored somewhere and you can just send it without transformations, it's some kind of static object.
Now, looking at a single S3 bucket as a key-value store, with some kind of routing mapping an object's URL to a set of shards each containing the object, one could argue it's serving static objects.
> require a lot more computation to work out where they are and where they need to go
I hope not. I would bet it's not very far from serving a file from the filesystem. There may be a lot of i/o contension though.
> in fact, that number should be much higher
Yes, very probably, and that only makes the 1.1M RPS number seem even less impressive - for amazon.
> 1.1M RPS is the amount they actually serve, not how much they can serve
My whole point was this number of requests seem low for amazon, not that they couldn't handle more.