Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, Google App Engine

carlosble.com

Post reply on HN
Print view View original

31–40 of 102 posts

Re: Goodbye, Google App Engine

#31

Post by nl »

Does no one read the documentation before deciding to use a platform anymore?

App Engine supports Python 2.5. The Python interpreter runs in a secured "sandbox" environment to isolate your application for service and security. The interpreter can run any Python code, including Python modules you include with your application, as well as the Python standard library. The interpreter cannot load Python modules with C code; it is a "pure" Python environment.

At the top of the FIRST page of documentation: http://code.google.com/appengine/docs/python/overview.html

Google Apps domains do not currently support HTTPS. HTTPS support is limited to apps accessed via .appspot.com domains. Accessing an HTTPS URL on a Google Apps domain will return a "host not found" error, and accessing a URL whose handler only accepts HTTPS (see below) using HTTP will return an HTTP 403 "Forbidden" error. You can link to an HTTPS URL with the .appspot.com domain for secure features, and use the Apps domain and HTTP for the rest of the site.

HIGHLIGHTED on http://code.google.com/appengine/docs/python/config/appconfi...

A request handler has a limited amount of time to generate and return a response to a request, typically around 30 seconds. Once the deadline has been reached, the request handler is interrupted.

and

While a request can take as long as 30 seconds to respond, App Engine is optimized for applications with short-lived requests, typically those that take a few hundred milliseconds. An efficient app responds quickly for the majority of requests. An app that doesn't will not scale well with App Engine's infrastructure.

http://code.google.com/appengine/docs/python/runtime.html#Th...

I could go on and on.. reading this I see "I wasted 15000€ by not reading the documentation"

I usually think the title technical architect is a bit stupid (my business card says I'm one, so I can say that) but this guy needs a good technical architect to make platform decisions prior to wasting that much money

Re: Goodbye, Google App Engine

#32

Post by jlees »

st3fan wrote: post #17

"developing on GAE introduced such a design complexity that working around it pushes us 5 months behind schedule" This is the core of all their problems. It is a mindset incompatibility between these app designers and GAE. The GAE APIs and rules are actually pretty simple and well defined. It works really well, but only if you work WITH those rules. You have to adopt the GAE application design philosophy. If you don'…

Learning to do things the App Engine way frustrated me at first, too, until I realised in a very roundabout way it was - as you say - teaching me how to engineer for scale.

Re: Goodbye, Google App Engine

#33

Post by mmastrac »

nl wrote: post #31

Does no one read the documentation before deciding to use a platform anymore? App Engine supports Python 2.5. The Python interpreter runs in a secured "sandbox" environment to isolate your application for service and security. The interpreter can run any Python code, including Python modules you include with your application, as well as the Python standard library. The interpreter cannot load Python modules with C co…

Agreed, absolutely. Before we settled on GAE, I spent time playing with it and wrote a few nontrivial test applications to get a good feel for it. We haven't been surprised by anything on the platform, except possibly the spurious 500 errors that were happening way too often in September (and have been fixed by the latest rounds of maintenance)

We made the trade of zero platform maintenance and near-free scaling and accepted the limits placed on what we can do. Many of those limits are being lifted each month as well. I can say for sure that there are a number of items mentioned in this post that have fixes in the works right now.

I will say that choosing GAE let us ship a project much faster than we could have without it and with much less manpower. The time we spend working around limits is easily made up by not having to do IT work.

Re: Goodbye, Google App Engine

#34

Post by krosaen »

This hasn't been my experience.

Yes, existing techniques for full text search works or things like geolocation queries won't work but there are other[1] techniques[2] that work just as well; it's just not the sql way. Basically, support for multiple set membership queries against a list of tags stored with entities is extremely powerful and if you index properly, you can do a lot of cool things [3]. Plus, you can do datastore queries in parallel [4], which means you don't have to denormalize as much as you think; just parallelize and memcache results; e.g for a complicated front page, you can fetch different types of content in parallel.

The local server behaves remarkably the same as the deployed server, it's quite rare I find a situation where something behaves differently in production. the entire datastore can be tested locally, including complicated schemas / indices / queries in fast running unit tests. This means when I do need to do something fancy with the datastore, I can fully test it with unit tests and be confident it will work when deployed.

Long running tasks can always be broken up using the task queue. the limit will soon be 10 minutes for individual tasks and cron jobs [5]

I agree that cold start is a huge issue, but looks like it is being addressed in the 1.4 release [5] where you can pay for 3 reserve instances at all ties. Lack of support for https on your domain definitely sucks too, but I don't see how he wouldn't have been aware of that before going with GAE.

Finally, there are a number of things that are a huge time / money savers: - really easy deployment process including support for multiple versions. This let's you have staging instances and quickly roll back to a previous version if there are any problems - a nice admin console with a number of tools, including comprehensive access to logs that are coherent across all instances - some really nice libraries for examining performance of datastore queries and other api calls [6] and getting daily email reports of any exceptions [7]. these are built using hooks available to you in case you want to build something similar (for instance I used hooks to have regression tests on the number of datastore queries each page requires). - the services and apis made available are really nice. for instance, the image hosting infrastructure that provides fast access to different sizes for a stored picture based on a url is pretty slick; they basically opened up the same infrastructure that is used by picasaweb to app engine users - virtually no hosting costs until you get a lot of traffic. thousands of daily visitors is still in the free range

That said, my biggest outstanding gripes:

- cold start problem (until 1.4 is out)

- datastore latency spikes sometimes. this has gotten a lot better in the past few weeks, but I'll still have this gripe until I see it more consistent for a couple months

- no support for incoming emails with attachments > 1mb (makes incoming photos from smart phones impossible since they are usually > 3Mp these days)

- no support for long polling (upcoming channel API seems to be more for chat rooms than for general purpose server push) [8]

[1] http://www.billkatz.com/2009/6/Simple-Full-Text-Search-for-A...

[2] http://fluffybunnysoftware.com/node/8

[3] http://code.google.com/events/io/sessions/BuildingScalableCo...

[4] http://code.google.com/p/asynctools

[5] http://groups.google.com/group/google-appengine/browse_threa...

[6] http://googleappengine.blogspot.com/2010/03/easy-performance...

[7]http://code.google.com/appengine/articles/python/recording_e...

[8] http://bitshaq.com/2010/09/01/sneak-peak-gae-channel-api

Re: Goodbye, Google App Engine

#35

Post by powera »

A few of these points are just ridiculous:

1) If you want "SQL and Joins", use SQL. This is like complaining that you can't play Halo on Linux.

1A) There isn't full text search. If you need full text search, use a system with full text search as a feature.

2) Some of the points are out of date (or will be out of date soon). The 30 second limit for cron jobs will be 10 minutes after the next release. As noted, the 1000 results per query limit is gone already.

3) Anything can fail. If you assume your own system won't fail, you're going to be in worse shape later.

4) What objects would you cache that are >1MB anyhow? In almost any case, you'd be better off caching it as multiple, smaller objects.

Re: Goodbye, Google App Engine

#36

Post by dantkz »

nl wrote: post #31

Does no one read the documentation before deciding to use a platform anymore? App Engine supports Python 2.5. The Python interpreter runs in a secured "sandbox" environment to isolate your application for service and security. The interpreter can run any Python code, including Python modules you include with your application, as well as the Python standard library. The interpreter cannot load Python modules with C co…

In the comments he says that the restrictions were not THE problem, THE problem was the instability. List of restrictions is kind of reminder of what he had to cope with, and wasted money on, until it turned out that GAE doesn't work properly.

Re: Goodbye, Google App Engine

#38

Post by wrath »

We love AppEngine for many of the reasons which he gives. Because of its limitations, the platform makes you think and write code with proper design patterns in mind. If you just start to write code without thinking about it beforehand, yes you won't like AppEngine and it's not for you. It'll cost you LOTS of money and won't perform very well. On the other hand, by writing code designed for AppEngine we've been able to reduce our costs by several thousand $$ a month. Also a side benefit is that we migrated one of our IT roles to a development role, which means that we're able to iterate faster.

Granted we don't have a need for SSL and not being able to use C libraries in python has caused us many hours "pain", but compared to the alternative for a small company like we have, it's well worth it.

My biggest issue with AppEngine is that there's no full-text index functionality, and there's no way to create your own. We've tried everything, and nothing works if you have millions of documents like we have. Our search is still external to AppEngine but we're hoping that Google will do something about it sooner or later.

Re: Goodbye, Google App Engine

#39

Post by krosaen »

powera wrote: post #35

A few of these points are just ridiculous: 1) If you want "SQL and Joins", use SQL. This is like complaining that you can't play Halo on Linux. 1A) There isn't full text search. If you need full text search, use a system with full text search as a feature. 2) Some of the points are out of date (or will be out of date soon). The 30 second limit for cron jobs will be 10 minutes after the next release. As noted, the 100…

I've gotten full text search working using techniques similar to those described here:

http://www.billkatz.com/2009/6/Simple-Full-Text-Search-for-A...

Re: Goodbye, Google App Engine

#40

Post by nl »

dantkz wrote: post #36
nl wrote: post #31

Does no one read the documentation before deciding to use a platform anymore? App Engine supports Python 2.5. The Python interpreter runs in a secured "sandbox" environment to isolate your application for service and security. The interpreter can run any Python code, including Python modules you include with your application, as well as the Python standard library. The interpreter cannot load Python modules with C co…

In the comments he says that the restrictions were not THE problem, THE problem was the instability. List of restrictions is kind of reminder of what he had to cope with, and wasted money on, until it turned out that GAE doesn't work properly.

In my experience "instability" with GAE means you are pushing it beyond where it wants to go. If you a have code that just completes within the timeout, then guess what? When the system is under load it isn't going to complete.

You can use all kinds of techniques to protect your application from things like variable datastore performance. For example, nowdays I usually use a pattern where user-facing servlets only ever read from the datastore - all updates go via a taskqueue, which means that even if the datastore is in maintenance mode the updates will eventually be applied.

GAE isn't for everyone, but the first step to getting the most out of it is accepting it isn't anything like a typical frontend plus database architecture.

Post reply on HN