Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

code.google.com

Post reply on HN
Print view View original

61–70 of 129 posts

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#61

Post by efdee »

timme wrote: post #38

The headline blows this out of proportion. Few people (who act obnoxious as hell) report a problem that can be solved by moving away from a deprecated system, yet they fail to even read the note because they're busy smashing exclamation marks into the issue tracker.

The problem apparently also occurs on the non-deprecated system. I can understand their frustration after not getting a reply for X days on what seems to be a critical issue for them. That's not "obnoxious as hell", that's customers panicking. You really don't want your customers panicking about your service.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#62

Post by okku »

ThingTwo wrote: post #57
okku wrote: post #44

I am also a GAE-user, I have had no problems like the OP. But I start to miss a fundamental feature, sockets. I have worked around it by using other services and polling. Maybe wrong forum, but is there any infrastructure templates for setting up a scalable web/db/loadbalancer/memcached for a simple tradional webservice, in my case a game? I want to be able to sleep at night, and easily scale up by adding some more m…

Channels.

Channels are for communicating with javascript clients. I want to communicate with other servers and applications.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#63

Post by kaolinite »

pyalot2 wrote: post #25

Ok, so here's the deal. If your app runs exclusively on GAE you've essentially tied yourself to one cloud vendor. Now disregarding the respective benefits and drawbacks of google as a hosting company for your app (I would never do that), being dependent on one cloud provider is a very bad idea. No matter if you run on EC2, Azure or GAE, if you can't seamlessly switch to another provider, you're screwed. These all go…

Agree with this to an extent however the company I work for deploys on AWS and is far too cautious about vendor lock-in, to the point where we use AWS basically as a VPS, not a cloud service, and get none of the advantages (and all of the disadvantages, e.g. worse performance, higher price).

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#64

Post by nailer »

afhof wrote: post #3

Comment from the WontFix mark: "M/S is deprecated and there is a clear and straightforward path to migrating to HRD." M/S was deprecated April 4, 2012, so it has been some time since the notice has been out there. High replication data store has been available for over 2 years now. Whether or not less than a year is too short a deprecation period is another issue.

Read the following message - other customers using Python on HRD are reporting the same issue as the parent, who is Java on M/S.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#65

Post by codeka »

raverbashing wrote: post #40
dylanvee wrote: post #34

Earlier quoted context omitted.

You're correct--because we have a Premier account, which anyone else can obtain too: https://developers.google.com/appengine/docs/premier/

Well, Caveat emptor Also, Khan academy receives funding from Google ( http://en.wikipedia.org/wiki/Khan_academy ), so I don't think it boils down to only having a Premier account.

"You just have to pay for support and you get support? I don't believe it, there must be more to it than that!"

Whether you believe it or not, a Premier account is what you need if you want support. You could argue that $500/mo is too expensive, but it is what it is.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#66

Post by jfoster »

meaty wrote: post #30
bodegajed wrote: post #10

Earlier quoted context omitted.

I consider him panicking more than shouting and threatening. I couldn't imagine having that kind of treatment as a vps customer else I'll be moving out asap.

If you were a VPS customer, you'd be less locked in and can just walk to another vendor so they actually are shit hot with support usually. It was obvious when I first tried it that GAE has crappy support.

What are we actually talking about when discussing "good support" and "bad support"? Is it just someone nice to talk to whilst someone else fixes a problem for you? There was an interesting article along these lines by the former President of Enterprise at Google written recently: http://gigaom.com/2013/01/26/the-delusions-that-companies-ha...

In this case, the GAE feature that underlies this issue is the Master/Slave (MS) datastore. It's been deprecated for ages in favour of the High-Replication Datastore (HRD).

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#67

Post by jfoster »

moobirubi wrote: post #36
ronyeh wrote: post #31

Earlier quoted context omitted.

M/S is Master/Slave: https://developers.google.com/appengine/docs/python/datastor... HRD is High Replication Datastore: https://developers.google.com/appengine/docs/adminconsole/mi... M/S is deprecated, and HRD is the new hotness (and it conveniently costs more).

I think it cost the same, when it was first out it cost more, but now it cost the same.

The rates are the same ($1 per million writes and $0.70 per million reads beyond the daily free threshold), but the daily free threshold is 0.05 million of each for master/slave, and 0.01 million of each for high-replication.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#68

Post by stingraycharles »

rachelbythebay wrote: post #26

Remember the old "thundering herd" problem with Apache children and things of that nature? You'd basically have a whole bunch of processes which had a listening fd from an earlier call to listen(). When a new connection would come in, the kernel would wake all of them, even though only one of them would actually have something to get. The others would go through the process for nothing. It caused a big performance hi…

On what basis do you think these issues are related? The bug report provides very little insight in what's going on, only that there's a severe performance degradation at 9AM.

The thundering herd problem applied to waking up child processes is one possible explanation, but there are dozens of other explanations that are just as likely, based on the information we're provided with.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#69

Post by StavrosK »

pestaa wrote: post #20
neya wrote: post #16

I'm sorry, but this is the price you pay for running your business that is dependent TOTALLY on a 3rd party service. Forget Google, everyone out there is most likely the same, that's why it's important for you to run your 'apps' on something you have control over - Like Linode, AWS, Rackspace, Openshift, etc. and also have back-up nodes from other providers for redundancy, for emergency situations, incase of storms,…

Interesting comment. When AWS went down in the US, devops rightly said "we told you to run your apps on something you have control over."

What's that? A server I own on a rack I own in a datacenter I own on a power factory I own and a telco I own?

"Having control over" something is a scale, it's not binary.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#70

Post by jfoster »

Ironlink wrote: post #43
timme wrote: post #38

The headline blows this out of proportion. Few people (who act obnoxious as hell) report a problem that can be solved by moving away from a deprecated system, yet they fail to even read the note because they're busy smashing exclamation marks into the issue tracker.

I couldn't agree more. In fact, it was deprecated just about eleven months ago: http://googleappengine.blogspot.com/2012/04/masterslave-data... When your datastore gets deprecated, you act sooner rather than later.

The two datastores even have the same API. As long as your app doesn't depend on the exact performance characteristics of the old one, the migration is very straightforward. I did it for one of my apps in a morning and was done well before lunch.
Post reply on HN