Google App Engine's Datastore Admin is Terribly Inefficient
marram.posterous.com
Google App Engine's Datastore Admin is Terribly Inefficient
1–10 of 18 posts
Re: Google App Engine's Datastore Admin is Terribly Inefficient
#2I'm pretty sure this uses the map reduce API which has a lot of overhead in the datastore. In principle map reduce is nice because it could make very large jobs fast. But since Google engineers don't pay for anything, they optimized for time, not cost.
And with regards to your script, you can't just delete 3k keys in one request. If you want I'll send you the script I've adapted for jobs that make large changes to the datastore.
Re: Google App Engine's Datastore Admin is Terribly Inefficient
#3I'm pretty sure this uses the map reduce API which has a lot of overhead in the datastore. In principle map reduce is nice because it could make very large jobs fast. But since Google engineers don't pay for anything, they optimized for time, not cost. And with regards to your script, you can't just delete 3k keys in one request. If you want I'll send you the script I've adapted for jobs that make large changes to th…
I meant that I needed 3k requests to finish the job, deleting 1k entities in each request :).
Re: Google App Engine's Datastore Admin is Terribly Inefficient
#4number of writes also depends on the number of indexes you have on the data
Re: Google App Engine's Datastore Admin is Terribly Inefficient
#5I ran into the same issue. If you want to purge all data from an app, it's much cheaper (and sometimes even faster) to start over and create a complete new app with an empty data store than to use the data store admin and delete the data from there.
Re: Google App Engine's Datastore Admin is Terribly Inefficient
#6I'm pretty sure this uses the map reduce API which has a lot of overhead in the datastore. In principle map reduce is nice because it could make very large jobs fast. But since Google engineers don't pay for anything, they optimized for time, not cost. And with regards to your script, you can't just delete 3k keys in one request. If you want I'll send you the script I've adapted for jobs that make large changes to th…
From my experience purging data via MapReduce API use a lot less write quota than admin interface (but with a bit of instance hour overhead which doesn't seems like a problem)
I can't remember the exact number but it was about 10 times less than deleting via admin interface and finish in 5 minutes rather than 3 hours.
Re: Google App Engine's Datastore Admin is Terribly Inefficient
#7A GAE datastore delete takes multiple operations because it also updates indexes:
1 entity delete = 2 Writes + 2 Writes per indexed property value + 1 Write per composite index value
All from this page: http://code.google.com/appengine/docs/billing.html#Billable_... And more about why it is so: http://code.google.com/appengine/articles/life_of_write.html
Well, the OP is just another coder who can't read docs, but can write a blog.
Re: Google App Engine's Datastore Admin is Terribly Inefficient
#8This is something that a lot of GAE developers misunderstand: put()ing a datastore entity is not a single write operation. There are indexes to update - in your case, lots of them - and updating these indexes can require several write operations. A simple delete is one write per index but changing a value can be two operations; one to delete the old index value and one to write the new one. And since each property has two indexes (ascending and descending), these numbers are X2.
If you create your own bulk delete method, you will find that it takes exactly as many write ops as the admin console tool.
You probably have defined more indexes on your entities than you need to - you will likely be able to make your app cheaper by removing unnecessary indexes. Managing indexes carefully is a critical part of making apps affordable on GAE.
Re: Google App Engine's Datastore Admin is Terribly Inefficient
#9A GAE datastore delete takes multiple operations because it also updates indexes: 1 entity delete = 2 Writes + 2 Writes per indexed property value + 1 Write per composite index value All from this page: http://code.google.com/appengine/docs/billing.html#Billable_... And more about why it is so: http://code.google.com/appengine/articles/life_of_write.html Well, the OP is just another coder who can't read docs, but can…
In any case, good that something is pointed out that can and will be easily overlooked :)
Re: Google App Engine's Datastore Admin is Terribly Inefficient
#10Can somebody explain the article in laymen terms ? For those not too familiar with GAE...