I've heard a lot of people saying this is why Google can't get a lot of businesses to sign on. There's no one for the CEO to call and complain to directly when their stuff is down.
Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance
121–129 of 129 posts
Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance
#122A GAE user sees a problem of his service being slow, writes a frantic bug report with caps and exclamation marks and threatens to leave GAE. As a GAE user myself, two questions come to mind: 1. Is GAE outside of their .9995 SLA* uptime? If they aren't, then it probably isn't important enough spend time looking into it. Customers cannot expect better than the agreed upon uptime percent, and hosting companies are oblig…
Not trying to apologize for G here, but just making sure the facts are straight:
There is no SLA for M/S applications (which run on completely different infrastructure). GAE has only ever offered a SLA for HRD applications.
Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance
#123> I'm going to assume everyone experiencing this issue is using M/S. Upgrading to HRD will solve your issue. This is the reason I abandoned AE and part of why adopting a platform that isn't standardized is incredibly dangerous. The problem is technical debt constantly accrues even when you aren't making changes . Even though the API was unchanged, HRD differs subtly enough that breakage can occur on any non-trivial p…
> This means a project written and signed off circa 2011 requires mandatory engineering costs just to continue running in a functioning and supported fashion. Funny (?) thing is, as an engineer at Google, stuff like that happened to me ALL THE TIME. I don't even want to think about how much of my time was spent simply migrating to the "latest greatest" replacement for some critical service that was being deprecated.
I think thats the modus operandi of any big internet company with significant internal infrastructure. The next big thing is always coming.
Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance
#124It happens that 9 AM Brussels time is midnight pacific time. I'm sure Google is running some maintentance cron at midnight thinking "This is a low demand time," and it is, across the US, but not in Brussels. These are old instances, and Google probably doesn't want to re-time or rewrite the cron job to be more efficient.
Do people go to bed that early in the US?
Looked at my project's charts, the low demand time is definitely 4AM and midnight is 40% of peak.
Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance
#125Remember the old "thundering herd" problem with Apache children and things of that nature? You'd basically have a whole bunch of processes which had a listening fd from an earlier call to listen(). When a new connection would come in, the kernel would wake all of them, even though only one of them would actually have something to get. The others would go through the process for nothing. It caused a big performance hi…
On what basis do you think these issues are related? The bug report provides very little insight in what's going on, only that there's a severe performance degradation at 9AM. The thundering herd problem applied to waking up child processes is one possible explanation, but there are dozens of other explanations that are just as likely, based on the information we're provided with.
9 AM in Brussels is midnight here on the west coast if I've done my time zone math properly. It's the perfect time to push something. Unfortunately, if that means having everything snap-to and then freeze for a couple of seconds, that's not good.
Again, I don't know if this is what happened here. I've just seen this sort of thing before.
Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance
#126Earlier quoted context omitted.
What are we actually talking about when discussing "good support" and "bad support"? Is it just someone nice to talk to whilst someone else fixes a problem for you? There was an interesting article along these lines by the former President of Enterprise at Google written recently: http://gigaom.com/2013/01/26/the-delusions-that-companies-ha... In this case, the GAE feature that underlies this issue is the Master/Slav…
He falls in to a trap of knowing machine behavior, but not dealing with people behavior. Insanity #2: I need somebody to talk to when a service interruption occurs You hear about an earthquake in California, you call your aunt to make sure she is ok. You are getting bad weather in the area you live, your mom calls and checks on you. The server you use disappears off the internet and your providers status page hasn't…
That's true. I think that his broader point still stands, though. Once you get beyond variants of "are you working on it or do I need to convince you to?" the role of support is basically catering to irrational desires.
Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance
#127Hi folks. We are fully aware of this issue. We've added it to external issue tracker (https://code.google.com/p/googleappengine/issues/detail?id=8...), please follow up there.
Response from us was initially muted because it looked like it only affected M/S apps, but it turns out (a) it can impact HRD as well, and (b) we're pretty unhappy about the level of impact for many M/S apps so we're looking at ways to resolve. It's a high priority and we're looking at a number of ways to address it. It's also a pretty interesting issue, because indirectly it's caused by (a) the large scale that App Engine is running, and (b) the large extent with which GAE is running free applications.
Regardless, apologies to those who felt support was unresponsive. We are working very hard to improve support. For the sophisticated audience that comes to these pages, please link to me on Google+ to get my attention if we are failing you (https://plus.sandbox.google.com/110401818717224273095).
Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance
#128Earlier quoted context omitted.
Salesforce have quite a nice solution for a similar problem. If you write custom Apex code, the platform will not let you deploy it to live unless it has sufficient unit test coverage. It runs the tests and calculates code coverage when you try and deploy, and if your tests aren't covering enough, no deployment happens. So, you have tests. Then, upcoming platform changes are released to sandbox environments six month…
I take it they then don't have any tests of their own to assure backwards compatibility? Maybe I've been spoiled by using Windows for 20 years but I feel that we should be able to expect better from vendors than this. That goes double if you're paying them, though it sounds in Salesforce's case as if they pay you, because I can't see any other way in which this arrangement would make sense.
> I take it they then don't have any tests of their own to assure backwards compatibility?
I think they do. They take that pretty seriously, breaking changes to the API are rare and usually obscure edge cases.