Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Unofficial Guide to Migrating Off of Google App Engine

www-cs-students.stanford.edu

Post reply on HN
Print view View original

31–40 of 63 posts

Re: The Unofficial Guide to Migrating Off of Google App Engine

#31

Post by jasonkester »

mhusby wrote: post #10
hello_moto wrote: post #8

I don't quite understand with some of the commenters: why do you guys talk about caching immediately as if your app needs to scale from the ground? Isn't the point of GAE is that it scales (as long as you don't do stupid queries)? If we have to put everything in memcached, what's the point of using GAE? I also don't quite understand the push of using memcached for almost everything (especially for young startups). Ho…

I think in this reference they are not talking about storing live data in memcache, just copies of objects for faster access. So when you are getting the information about a user you check memcache first, if its not there look it up in datastore and add it to memcache. Then when the user loads a second page you can get the information directly from memcache instead of having to look it up again.

I think that's what he's talking about.

The point in question is why are you doing this on a new application? The sort of caching you describe is something you put in place after your site starts seeing a lot of traffic, and running queries for every request start to slow things down to the point where optimizing them doesn't help.

If you're designing your app well and running it on a solid stack, that should be something you need to worry about in year 3, after you're being TechCrunched on a regular basis. In the Rails/Django/AppEngine world, it seems to be the case that you need to resort to that level of caching just to see regular, day to day, 100 request/second traffic.

It raises red flags to see that.

Re: The Unofficial Guide to Migrating Off of Google App Engine

#32

Post by realmojo »

bartman wrote: post #13

Typhoon App Engine allows you to set up your own GAE compatible environment: http://code.google.com/p/typhoonae/ I never used it, but it sure looks interesting if your application doesn't fit GAE anymore or you want to make specific infrastructural adjustments. They support multiple different database/http/.. servers too.

Another project is http://code.google.com/p/appscale/.

Re: The Unofficial Guide to Migrating Off of Google App Engine

#34

Post by sorenbs »

jasonkester wrote: post #31
mhusby wrote: post #10

Earlier quoted context omitted.

I think in this reference they are not talking about storing live data in memcache, just copies of objects for faster access. So when you are getting the information about a user you check memcache first, if its not there look it up in datastore and add it to memcache. Then when the user loads a second page you can get the information directly from memcache instead of having to look it up again.

I think that's what he's talking about. The point in question is why are you doing this on a new application? The sort of caching you describe is something you put in place after your site starts seeing a lot of traffic, and running queries for every request start to slow things down to the point where optimizing them doesn't help. If you're designing your app well and running it on a solid stack, that should be some…

On what setup do you serve 100 pages a second that is talking directly to the database?

Re: The Unofficial Guide to Migrating Off of Google App Engine

#35

Post by thurn »

StavrosK wrote: post #25
thurn wrote: post #2

I found that the AppEngine 1.4 SDK addressed many of these concerns. Personally, I've managed 50 requests per second on my blog without trouble, and with minimal CPU overhead. That's probably because everything is in memcache, so the database almost never gets hit. The pricing structure seems to actively encourage you to memcache as much as possible, too. Things are probably pretty different in a more write-intensive…

Which blogging engine do you use and how much do you pay? I'm looking to migrate my blog to GAE too. Also, how's the latency?

I wrote my own. http://github.com/thurn/ackbar. If an article gets popular, I've paid up to $5, but below 6000 hits, it's all free. Latency could be better, but there's a lot of factors there (Clojure might be one of them?).

Re: The Unofficial Guide to Migrating Off of Google App Engine

#36

Post by jasonkester »

sorenbs wrote: post #34
jasonkester wrote: post #31

Earlier quoted context omitted.

I think that's what he's talking about. The point in question is why are you doing this on a new application? The sort of caching you describe is something you put in place after your site starts seeing a lot of traffic, and running queries for every request start to slow things down to the point where optimizing them doesn't help. If you're designing your app well and running it on a solid stack, that should be some…

On what setup do you serve 100 pages a second that is talking directly to the database?

C#/ASP.NET/SQL Server, all out of the box, running database and web server on a single machine that would have been a middle of the road dev box 4 years ago.

That box is running about 20 sites, some big, some small, usually no more than one of them getting TechCrunched/Redditted at a time.

Here's a writeup of one particularly heavy day in the life of that box:

http://expatsoftware.com/articles/2008/03/6-million-hits-day...

Re: The Unofficial Guide to Migrating Off of Google App Engine

#37

Post by StavrosK »

thurn wrote: post #35
StavrosK wrote: post #25

Earlier quoted context omitted.

Which blogging engine do you use and how much do you pay? I'm looking to migrate my blog to GAE too. Also, how's the latency?

I wrote my own. http://github.com/thurn/ackbar . If an article gets popular, I've paid up to $5, but below 6000 hits, it's all free. Latency could be better, but there's a lot of factors there (Clojure might be one of them?).

I see, thanks. If you don't mind, how come you pay? The free tier seems very large for a simple blog, which resource did you run out of?

Re: The Unofficial Guide to Migrating Off of Google App Engine

#38

Post by kamens »

powera wrote: post #5

A few points (disclaimer: I work at Google on projects including App Engine) 1) It's not fair at all to say that Google "continue to ignore the two critical issues of uptime and data store latency". The HR datastore is specifically designed to address the concerns about variable latency of datastore operations, and to prevent both planned and unplanned downtime. 2) In my experience, high CPU cost for datastore operat…

Can you elaborate on queries not having "the right indexes"? Since the development server automatically adds indexes for all queries run on local dev machines and the production machines cannot run a query for which the necessary indexes don't exist, it's easy to assume that your indexes must be right.

I'd love to hear more about tweaks to existing indexes that can alleviate these sorts of problems.

Re: The Unofficial Guide to Migrating Off of Google App Engine

#39

Post by joelburget »

On the other hand, GAE looks like a great fit for a static site built with Jekyll, like a personal blog. In fact I'm planning to migrate my site over today. Almost any personal site will be free to host. Even though I would only be paying a few dollars with S3, I see no reason not to give app engine a shot.

One thing I worry about is the reported downtime. I'm not sure whether or not that will affect a static site.

Re: The Unofficial Guide to Migrating Off of Google App Engine

#40

Post by bane »

vsiva68 wrote: post #23
shimon wrote: post #15

Earlier quoted context omitted.

The original article does point out a few reasons, but the way I see it is this: AE is great if you want to scale and your app can scale in the AE way. There is a sweet spot and the tools work pretty well if you're in that sweet spot, but if you stray outside that spot you don't have a lot of options. Even getting your data out can be difficult, making the "export data and start over" nuclear option difficult. If you…

Another way of saying the same thing is that GAE is perfectly fine if you know all the requirements of your app before you build it. In a perfect world, you know exactly what you want to build, everything you want is available with GAE, then you go build it and it scales nicely. In the real world however, requirements change: * You come up with a new idea that requires a certain library. Chances are, the library won'…

If I understand correctly then, it would appear that GAE may be poor for applications that require heavy computation on the server side (say, a facebook style graph, crawling and computing various metrics on it) but great for serving tons of dynamic pages?
Post reply on HN