The Unofficial Guide to Migrating Off of Google App Engine
www-cs-students.stanford.edu
The Unofficial Guide to Migrating Off of Google App Engine
1–10 of 63 posts
Re: The Unofficial Guide to Migrating Off of Google App Engine
#2I found that the AppEngine 1.4 SDK addressed many of these concerns. Personally, I've managed 50 requests per second on my blog without trouble, and with minimal CPU overhead. That's probably because everything is in memcache, so the database almost never gets hit. The pricing structure seems to actively encourage you to memcache as much as possible, too. Things are probably pretty different in a more write-intensive app, though.
Re: The Unofficial Guide to Migrating Off of Google App Engine
#3I've been a heavy App Engine user since the beta days -- even gave a talk at I|O 2009 about scaling with App Engine. I have clients that run complex GAE sites that handle millions of daily requests.
I somewhat disagree with the author about scalability. There is a very narrow sweet spot of apps for which App Engine is a quite natural solution. If you're in the sweet spot, you'll probably scale well without too much up-front engineering investment.
The sweet spot _is_ narrow, though. For example, as the OP states: geo data doesn't really belong on GAE -- you can do a limited set of bounding box/proximity queries with the third party geomodel library, but wow, it's expensive and dog slow!
I do agree with the author about both the status dashboard (it only sometimes reflects my current experience with the system) and the surprising variation it data store latency. Latency has been much improved of late with 1.4 and beyond.
Re: The Unofficial Guide to Migrating Off of Google App Engine
#4I'm starting to notice that App Engine's memcache is incredibly slow. If serving up a page requires one memcache hit (for, say, caching entire response objects), it works well. If it starts to require, say, a dozen, the response time slows to ~700ms at best, and ~4s at worst.
Re: The Unofficial Guide to Migrating Off of Google App Engine
#5A few points (disclaimer: I work at Google on projects including App Engine)
1) It's not fair at all to say that Google "continue to ignore the two critical issues of uptime and data store latency". The HR datastore is specifically designed to address the concerns about variable latency of datastore operations, and to prevent both planned and unplanned downtime.
2) In my experience, high CPU cost for datastore operations is generally tied to things like having large numbers of indexes, or doing queries that don't have the right indexes. Each index written involves a bigtable write, if you are doing hundreds of these per entity it can become very expensive. That said, in general it isn't easy enough to know when you are doing something that will be expensive.
3) The bulkloader is definitely painful to use if you aren't familiar with it. In particular, casting schemaless data to a format such as CSV is hard because of the problem that rows can have unexpected keys. An improved "Bulk Datastore Import and Export tool" is on the 6-month roadmap for the product.
Re: The Unofficial Guide to Migrating Off of Google App Engine
#6I'm starting to notice that App Engine's memcache is incredibly slow. If serving up a page requires one memcache hit (for, say, caching entire response objects), it works well. If it starts to require, say, a dozen, the response time slows to ~700ms at best, and ~4s at worst.
[deleted]
Re: The Unofficial Guide to Migrating Off of Google App Engine
#7I'm starting to notice that App Engine's memcache is incredibly slow. If serving up a page requires one memcache hit (for, say, caching entire response objects), it works well. If it starts to require, say, a dozen, the response time slows to ~700ms at best, and ~4s at worst.
Can you do the memcache calls in parallel with get_multi?
Unless you're doing something like retrieving 1MB objects, it should take no more than 10ms per memcache call even if they are done in serial.
Re: The Unofficial Guide to Migrating Off of Google App Engine
#8I don't quite understand with some of the commenters: why do you guys talk about caching immediately as if your app needs to scale from the ground?
Isn't the point of GAE is that it scales (as long as you don't do stupid queries)?
If we have to put everything in memcached, what's the point of using GAE?
I also don't quite understand the push of using memcached for almost everything (especially for young startups). How do you handle data integrity? I'm guessing most data models of young startups are fairly simple and only contain at most 10 models with almost no relationship? Otherwise data integrity is painful.
Re: The Unofficial Guide to Migrating Off of Google App Engine
#9Can someone provide some enlightenment about why one would migrate away from GAE? I only ask because I'm currently building an a web application on GAE and am starting to look down the road, wondering if building on GAE will allow us to grow as a business in the ways we want to, or if there are some arbitrary limitations that will come to bite us later.
(knowing of course that we're more or less tied to Google's infrastructure and the GAE way of doing things)
Re: The Unofficial Guide to Migrating Off of Google App Engine
#10I don't quite understand with some of the commenters: why do you guys talk about caching immediately as if your app needs to scale from the ground? Isn't the point of GAE is that it scales (as long as you don't do stupid queries)? If we have to put everything in memcached, what's the point of using GAE? I also don't quite understand the push of using memcached for almost everything (especially for young startups). Ho…
I think in this reference they are not talking about storing live data in memcache, just copies of objects for faster access. So when you are getting the information about a user you check memcache first, if its not there look it up in datastore and add it to memcache. Then when the user loads a second page you can get the information directly from memcache instead of having to look it up again.