Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google App Engine's Datastore Admin is Terribly Inefficient

marram.posterous.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Google App Engine's Datastore Admin is Terribly Inefficient

#11

Post by marram »

easy_rider wrote: post #9
peterknego wrote: post #7

A GAE datastore delete takes multiple operations because it also updates indexes: 1 entity delete = 2 Writes + 2 Writes per indexed property value + 1 Write per composite index value All from this page: http://code.google.com/appengine/docs/billing.html#Billable_... And more about why it is so: http://code.google.com/appengine/articles/life_of_write.html Well, the OP is just another coder who can't read docs, but can…

In any case, good that something is pointed out that can and will be easily overlooked :)

I deleted all indices referencing those entities before starting the entity deletions.

Re: Google App Engine's Datastore Admin is Terribly Inefficient

#12

Post by marram »

stickfigure wrote: post #8

This is something that a lot of GAE developers misunderstand: put()ing a datastore entity is not a single write operation. There are indexes to update - in your case, lots of them - and updating these indexes can require several write operations. A simple delete is one write per index but changing a value can be two operations; one to delete the old index value and one to write the new one. And since each property ha…

I had "vaccumed" all indices referencing those entities before issuing a delete. Albeit, there was only once index per purged entity type. So this would not explain the 20x write operations.

Also, note that the deletions were through the "Datasore Admin" app, which was recently added. It is different from the classic Datastore Viewer.

Re: Google App Engine's Datastore Admin is Terribly Inefficient

#13

Post by stickfigure »

marram wrote: post #12
stickfigure wrote: post #8

This is something that a lot of GAE developers misunderstand: put()ing a datastore entity is not a single write operation. There are indexes to update - in your case, lots of them - and updating these indexes can require several write operations. A simple delete is one write per index but changing a value can be two operations; one to delete the old index value and one to write the new one. And since each property ha…

I had "vaccumed" all indices referencing those entities before issuing a delete. Albeit, there was only once index per purged entity type. So this would not explain the 20x write operations. Also, note that the deletions were through the "Datasore Admin" app, which was recently added. It is different from the classic Datastore Viewer.

You misunderstand how GAE indexes work.

There are two kinds of indexes:

* multi-property indexes which you configure via datastore-indexes.xml (or yaml). You can remove these by removing them from the xml/yaml and vacuuming.

* single-property indexes, which you decide when you define your data model. You can't vacuum these, and they are defined on a per-entity basis. The only way to make them go away is to re-save the relevant entities without the index defined. Note: multiproperty indexes require single-property indexes on all the properties covered.

These single-property indexes are almost certainly causing your high write op counts. You really should examine your data model with this new understanding; by removing unnecessary single-property indexes, you may be able to dramatically reduce your bill.

Re: Google App Engine's Datastore Admin is Terribly Inefficient

#15

Post by tnuc »

There are plenty of things that are wrong with Google App Engine. And there are plenty of bugs that exist that have cost me money.

Why don't you try filing a bug report/suggesting a warning and send an email requesting something of a refund. They tend to be a friendly bunch who give refunds to obvious problems.

Moving to AWS will of course save you lots of money in the longer term, depending on what your hosting requirements are.

Re: Google App Engine's Datastore Admin is Terribly Inefficient

#16

Post by latchkey »

Should Google refund developers when they make an uninformed decision that costs them money?

One could argue it is a bug in GAE that allows developers to make an expensive mistake when they don't fully understand how something (fairly complicated) works.

Someone else could argue that we are all developers and we should know the costs associated with the systems we are building. There is a real cost associated with PaaS systems like GAE.

What do you think?

Re: Google App Engine's Datastore Admin is Terribly Inefficient

#17

Post by theli0nheart »

peterknego wrote: post #7

A GAE datastore delete takes multiple operations because it also updates indexes: 1 entity delete = 2 Writes + 2 Writes per indexed property value + 1 Write per composite index value All from this page: http://code.google.com/appengine/docs/billing.html#Billable_... And more about why it is so: http://code.google.com/appengine/articles/life_of_write.html Well, the OP is just another coder who can't read docs, but can…

Regardless of how decent Google's AppEngine documentation is, this is indeed a bug.

The correct behavior would be to recalculate the indices just once, instead of reindexing after every single delete operation.

It then becomes

    2*entities + 2*indexed property values + composite index values
operations to delete all entities in the datastore, instead of

    2*entities + 2*entities*indexed property values + entities*composite index values
operations.

Re: Google App Engine's Datastore Admin is Terribly Inefficient

#18

Post by tantalor »

theli0nheart wrote: post #17
peterknego wrote: post #7

A GAE datastore delete takes multiple operations because it also updates indexes: 1 entity delete = 2 Writes + 2 Writes per indexed property value + 1 Write per composite index value All from this page: http://code.google.com/appengine/docs/billing.html#Billable_... And more about why it is so: http://code.google.com/appengine/articles/life_of_write.html Well, the OP is just another coder who can't read docs, but can…

Regardless of how decent Google's AppEngine documentation is, this is indeed a bug. The correct behavior would be to recalculate the indices just once, instead of reindexing after every single delete operation. It then becomes 2*entities + 2*indexed property values + composite index values operations to delete all entities in the datastore, instead of 2*entities + 2*entities*indexed property values + entities*composi…

To delete all entities should be free. Who cares about indexes? `rm -rf`, done.
Post reply on HN