Google App Engine is now much faster, after latest maintenance • Hacker News Board

Re: Google App Engine is now much faster, after latest maintenance

#41

Post by mmastrac » Sun, Nov 07, 2010, 4:38 PM UTC

bad_user wrote: post #25

mmastrac wrote: post #14

Our app ( http://gri.pe ) feels a lot snappier after this latest bit of datastore maintenance. We've been using as much memcache as possible to avoid datastore access because prior to this maintenance, datastore access was spiky and even the simplest of queries had a small chance of timing out. I can say for sure that some of our datastore-heavy cron jobs were taking > 200ms, but are around 70ms now. It's a shame tha…

Unrelated question: how are you doing those cron jobs. I tried using the task queue, but it kept timing out on me, even when I broke my tasks into smaller ones (couldn't figure out a threshold as the behavior of the datastore was totally unpredictable).

We have to break our cron jobs down into single datastore queries for them to succeed near 100% of the time. If you want more complex logic, use a cron job to trigger tasks that process smaller bits of your operation.

Even our cron job to compute the top users on our site (a simple query + ordering) uses one second of CPU time between the application and the datastore. It's just something you get used to in AppEngine.

Re: Google App Engine is now much faster, after latest maintenance

#42

Post by mmastrac » Sun, Nov 07, 2010, 4:39 PM UTC

brown9-2 wrote: post #40

mmastrac wrote: post #14

Our app ( http://gri.pe ) feels a lot snappier after this latest bit of datastore maintenance. We've been using as much memcache as possible to avoid datastore access because prior to this maintenance, datastore access was spiky and even the simplest of queries had a small chance of timing out. I can say for sure that some of our datastore-heavy cron jobs were taking > 200ms, but are around 70ms now. It's a shame tha…

It's a shame that the stats for > 12hrs ago seem to have been wiped. I bet we'd be seeing a marked improvement on our ms/request stat. Just click the left-facing arrow next to the date. Here are the stats for 11/5: http://code.google.com/status/appengine/detail/datastore/201... and here are the stats for 11/4: http://code.google.com/status/appengine/detail/datastore/201...

Sorry, I meant the internal application statistics. You can visit your dashboard to view "milliseconds per request", but the data doesn't exist from before this maintenance period.

One interesting aside: our nightly datastore backup (using the remote API) no longer causes 500 errors! Hooray!

Re: Google App Engine is now much faster, after latest maintenance

#43

Post by mmastrac » Sun, Nov 07, 2010, 4:41 PM UTC

mshafrir wrote: post #38

bad_user wrote: post #37

Earlier quoted context omitted.

Quote from that article: A URL invoked by cron is subject to the same limits and quotas as a normal HTTP request, including the request time limit. And I'm looking for some tips and tricks to do heavy processing (which is really needed when working with non-relational datastores).

The key is to combine cron jobs with Tasks. Tasks are am easy to use and powerful feature of Google Appengine that let you deal with the standard limitations. For example, rather than doing some operation on 1000 entities in one request, you spin off 1000 tasks, each handling the operation on one entity. Read up about them at http://code.google.com/appengine/docs/python/taskqueue/ You can also explore the new Mapper…

Agreed, the Mapper API is the best way to deal with processing large numbers of entities from the datastore.

AppEngine requires careful thought of application design at this point. Everything is an HTTP request and subject to the same restrictions.

I've heard rumours that they'll be launching longer-lived tasks at some point in the future, but I have no idea when that will happen.

Re: Google App Engine is now much faster, after latest maintenance

#44

Post by wkornewald » Sun, Nov 07, 2010, 5:08 PM UTC

benjaminfox wrote: post #32

wkornewald wrote: post #28

It's still too early to tell. This would not be the first time that speed improves after a maintenance and then it slowly gets worse every week until it looks almost like before the maintenance.

Is it possible that the degradation in performance is due to more developers building more apps and driving more traffic to App Engine as time goes on?

The performance degrades too quickly for this. In two or three months we might be on our old performance level, again. Well, I seriously hope that we'll stay fast this time. Only time will tell.

Re: Google App Engine is now much faster, after latest maintenance

#45

Post by pjscott » Sun, Nov 07, 2010, 11:50 PM UTC

gfodor wrote: post #23

As any ops guy will tell you, this is the finest form of graph porn there is.

It would have been sexier on a semi-log-y scale. That said, as someone who's always been annoyed at the high latency of App Engine's datastore, this is huge if they can deliver this level of latency with low variability.