Any word on when boto will support this?
Sooner than you think
https://github.com/boto/boto/commit/8ef550ec028b379bcd85e550...
Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
121–130 of 135 posts
Re: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
#122I wrote an Erlang API for DDB while it was still private. I'm hoping my client will let me opensource it in the near future.
Re: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
#123Earlier quoted context omitted.
Lock-in? You'll need to get all your data out and transform it into whatever alternative you pick. I'd say as far as AWS lock-in, this is on the low side of things.
The lock-in comes not from the data format, but rather from the quirks of the particular system du jour: API lock-in.
Getting your data out will also cost you a boat load of money. Same as putting it in.
Re: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
#124Earlier quoted context omitted.
Remember that the throughput is per item, not per query. For instance we have an indexed query that returns ~1500 rows each time. Just doing that query a couple of times per second would create that kind of throughput requirement.
The amount of consumed read units by a query is not necessarily proportional to the # of items. It is equal to the cumulative size of processed items, rounded up to the next kilobyte increment. For example if you have a query returning 1,500 items of 64 bytes each, then you’ll consume 94 read units, not 1,500.
The official documentation seems to clearly contradict you. The pricing calculator doesn't let you specify a value of less than 1KB. Who's right? Or maybe I'm just not understanding what either you or the official pricing doc is saying :)
Re: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
#125Question about Composite Hash Keys that someone might have the answer to (or be able to relate to other known implementations):
The composite key has two attributes, a “hash attribute” and a “range attribute.” You can do a range query within records that have the same hash attribute.
It would obviously be untenable if they spread records with the same hash attribute across many servers. You'd have a scatter-gather issue. Range queries would need to query all servers and pop the min from each until it's done, and that significantly taxes network IO.
This implies that they try to keep the number of servers that host records for the same hash attribute to a minimum. Consequently, if you store too many documents with the same hash attribute, wouldn't you overburden that subset of servers, and thus see performance degradation?
Azure has similar functionality for their table service, requiring a partition key, and they explicitly say that there is a throughput limit for records with the same partition key. I haven't seen similar language from Amazon.
Whether you scatter-gather or try to cluster values to a small set of servers, you'll eventually degrade in performance. Does anyone have insight into Amazon's implementation?
Re: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
#126Earlier quoted context omitted.
The amount of consumed read units by a query is not necessarily proportional to the # of items. It is equal to the cumulative size of processed items, rounded up to the next kilobyte increment. For example if you have a query returning 1,500 items of 64 bytes each, then you’ll consume 94 read units, not 1,500.
The official documentation seems to clearly contradict you. The pricing calculator doesn't let you specify a value of less than 1KB. Who's right? Or maybe I'm just not understanding what either you or the official pricing doc is saying :)
From the pricing page (http://aws.amazon.com/dynamodb/pricing):
---
If your items are less than 1KB in size, then each unit of Read Capacity will give you 1 read/second of capacity and each unit of Write Capacity will give you 1 write/second of capacity. For example, if your items are 512 bytes and you need to read 100 items per second from your table, then you need to provision 100 units of Read Capacity.
---
Looks like 1KB is the minimum for calculations.
Re: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
#127https://img.skitch.com/20120118-etwh2gwu52isnn47fw2jpcir5p.p... You could buy a lot of Riak or Cassie for that.
5000 reads per sec of 64Kb items, would make you stream 2.5 Gbits/sec using consistent reads and 1Gbits/sec writes, moving close to 1.5TB each hour. At the end of the month you have read well over 800 TB and updates 160 TB... That is a substantial application you have in mind... :-)
That may be true for an application with a constant load, but applications with a less balanced load have to provision for their peaks. My company (Malwarebytes) has very irregular traffic (at the hour mark we get very big spikes, but only for a couple of minutes) and it seems like we would have to provision (for this specific app) that peak for the entire hour. I might be misunderstanding the billing for this service though- if we ask for more units for 15 minutes, would the billing be prorated?
This actually hits on my only real issue with AWS in general, which is the hourly billing. We've used the mapreduce service a bit, and having a cluster roll up and fail on the first job is heartbreaking when it has a hundred machine hours associated with it. Obviously that is far, far cheaper than us building out our own cluster (especially with spot instances, which I can't even describe how much I love), but for some of the services smaller billing times would be useful.
Re: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
#128Earlier quoted context omitted.
Remember that the throughput is per item, not per query. For instance we have an indexed query that returns ~1500 rows each time. Just doing that query a couple of times per second would create that kind of throughput requirement.
The amount of consumed read units by a query is not necessarily proportional to the # of items. It is equal to the cumulative size of processed items, rounded up to the next kilobyte increment. For example if you have a query returning 1,500 items of 64 bytes each, then you’ll consume 94 read units, not 1,500.
If that's the case then it's a completely different ball-game. I was about to abandon the whole idea of using DynamoDB due to the pricing of throughput. This makes it a whole lot more interesting!
Re: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
#129Earlier quoted context omitted.
For one, it responds quicker than Riak: Riak has (cold) response times of about 300ms, while this service claims single-digit ms response times. Also setting up Riak is not exactly trivial, and using this service outsources that hassle.
You can use Riak Smartmachines on Joyent's cloud that would get similar performance for an order of magnitude cheaper than what amazon is charging. If you are seeing 300ms response times, you are not using SSD's and you are not using a similar number of nodes that Amazon is charging you for. I'm not hating on Amazon, it is a good move for them and they are doing some things that Riak cannot do, but cost and response…
I both wouldn't bet on Joyent's Smartmachines being a magnitude cheaper and being as fast:
On the speed: We run a Riak cluster, and for us the actual response times we get from Riak are, as described, slower than what Amazon promises in its docs.
On the price: If you would get 3x16 GB machines with Joyent, that would cost you 1400$/month. You can get a lot of resources for that with this new AWS service.
I don't have any experience with either Joyent or this new DynamoDB, but I do have some experience with Riak, and from the docs this new service would be a very viable competitor.
Re: Amazon DynamoDB – a Fast and Scalable NoSQL Database Service from AWS
#130Earlier quoted context omitted.
The cost per transaction, performance & ease. Reads per $0.01 = (50.60).60 = 180000 Writes per $0.01 = (10.60).60 = 36000 Assuming that you hit your usage is at 100% capacity then from a read prospective DynamoDB is half the price. Writes are much more expensive but many applications are heavily read oriented. DynamoDB claims single digit millisecond reads, azure tables does not (from my experience.) Azure tables hav…
Can you please explain your math? AFAIK Azure txns are not paid by the hour - they are a flat cost of $.01 per 10000 storage txns. If you do batched GETs and PUTs you make only 550 txns (55000/100 entites/batch). http://www.windowsazure.com/en-us/home/tour/storage/ I agree that Dynamo's provisioned throughput capacity is a very useful feature though. Azure does not provide any such performance guarantee; the throughp…
I should have explained that my costs were calculated on a "per day" assumption. Thus the costs are for:
5000 x 60 x 60 x 24 = 432000000 Writes
50000 x 60 x 60 x 24 = 4320000000 Reads
(432000000/10000) x 0.01 = $432
(4320000000/10000) x 0.01 = $4320
Azure Total Cost For One Days Use: $4752
((5000/10) x 0.01) x 24 = $120
((50000/50) x 0.01) x 24 = $240
AWS Total Cost For One Days Use: $360
You are right that I don't take into account the bulk feature of azure reads & writes but this is down to bulk requests only being possible on a single partition at a time which in my personal experience (not exhaustive) is non-trivial to take advantage of.