Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

code.google.com

Post reply on HN
Print view View original

81–90 of 129 posts

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#81

Post by brown9-2 »

A bug tracker seems like a horrible way to report production (or non-production) support issues. This is the same bug tracker OSS projects on Google Code use.

Is it really helpful for the public to comment on my support request? Seems like the signal to noise ratio would be quite low, and then you get inane comments like:

I got here from HackerNews, but after seeing the original poster spam the forums in multiple places and have a bad attitude, I can't blame Google for not fixing what looks to me like a non-issue.

Fuck 'em.

You have to believe that the choice of tools has some bearing on the quality of the response from Google. Seems like there is very little incentive for any "Project members" to trawl through open bug reports when no one is ever responsible.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#82

Post by brown9-2 »

stingraycharles wrote: post #68
rachelbythebay wrote: post #26

Remember the old "thundering herd" problem with Apache children and things of that nature? You'd basically have a whole bunch of processes which had a listening fd from an earlier call to listen(). When a new connection would come in, the kernel would wake all of them, even though only one of them would actually have something to get. The others would go through the process for nothing. It caused a big performance hi…

On what basis do you think these issues are related? The bug report provides very little insight in what's going on, only that there's a severe performance degradation at 9AM. The thundering herd problem applied to waking up child processes is one possible explanation, but there are dozens of other explanations that are just as likely, based on the information we're provided with.

The commenter you are replying to is a former Google employee; the description of the lock service sounds like Chubby (http://research.google.com/archive/chubby.html), App Engine likely uses some sort of distributed directory service for keeping track of things like quotas.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#83

Post by samuel »

codeka wrote: post #65
raverbashing wrote: post #40

Earlier quoted context omitted.

Well, Caveat emptor Also, Khan academy receives funding from Google ( http://en.wikipedia.org/wiki/Khan_academy ), so I don't think it boils down to only having a Premier account.

"You just have to pay for support and you get support? I don't believe it, there must be more to it than that!" Whether you believe it or not, a Premier account is what you need if you want support. You could argue that $500/mo is too expensive, but it is what it is.

The potential bad press that Google would get if Khan Academy had daily issues due bad GAE performance I'm sure that would cost them much more than the 500$/month that you're paying. Being big and nice(or cool) I think it gives and edge here.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#84

Post by logn »

Ironically, the guy who closed this issue owns this project:

https://code.google.com/p/sentimentally/

"sentimentally is a tool that determines sentiment of your emails. Once determined, it helps you gauge your relationships with co-workers, customers, friends, or other individuals based on the tone of your conversations with these people."

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#85

Post by spqr »

Confusion wrote: post #9

Well, the bug report doesn't really invite quick attention. Simply reporting your observations is not enough: you should position yourself as a competent customer, by explaining what you have done to ensure the problem isn't on your side. Mention the code hasn't changed, that you have no database cleanup cronjobs or similar running that could be interfering, etc. My first instinct when I see a report like this is: he…

Keep blaming that victim. As for me, I moved off GAE over a year ago and now I wear as many short skirts as I want without the fear of someone telling me I deserve to be raped by Google.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#86

Post by spqr »

Ironlink wrote: post #45
benjaminwootton wrote: post #42

Google support is an absolute disgrace. I had a Nexus 7 go AWOL at Christmas and I've never had such a shambolic customer service experience. They have absolutely no respect or customer service ethos when it comes to people who are actually paying them real cash money. Not in a million years would I sign off on hosting a production project on App Engine.

This customer is complaining about a service component which has been deprecated since almost 11 months ago. There is a tool which migrates application data from the old datastore to the new one. When you don't move off of deprecated infrastructure, I'd say you've set yourself up for problems.

Almost 11 months you say? Well any self respecting hobbyist hacker at home should be able to whip up an upgrade in a week. The rest of us are trying to run a business and don't appreciate quickly deprecated products with no support less than a year after it is deprecated.

Windiws XP was released 12 years ago, has been deprecated for years and only recently stopped support by Microsoft.

If Google isn't willing to provide the same level of support they shoe just admit they're for hobbyists.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#87

Post by hosay123 »

> I'm going to assume everyone experiencing this issue is using M/S. Upgrading to HRD will solve your issue.

This is the reason I abandoned AE and part of why adopting a platform that isn't standardized is incredibly dangerous. The problem is technical debt constantly accrues even when you aren't making changes.

Even though the API was unchanged, HRD differs subtly enough that breakage can occur on any non-trivial project. Edge cases (how indices behave within transactions comes to mind, but there are plenty more examples) will see new semantics compared to M/S, and so this "upgrade" involves not only thorough testing and auditing, but likely also code changes and potentially significant engineering hours.

http://goo.gl/HVuaC: These techniques are not needed with the (now deprecated) Master/Slave Datastore, which always returns strongly consistent results for all queries.

This means a project written and signed off circa 2011 requires mandatory engineering costs just to continue running in a functioning and supported fashion. An AE app will never quite resemble that ancient perl5 behemoth running uninterrupted since 1997, as the underlying implementation and recommended APIs are constantly modified and replaced (Datastore, NDB, Python major version).

"A strong test suite will save your soul!" I hear you say, tests that a small project might have survived without if targeting any other platform, and testing on AppEngine is also yet another moving target (for example, testing nested subrequests was all but impossible using the SDK until relatively recently).

The promise was a carefree life for a project willing to code against their proprietary APIs; the reality is a constantly moving target, "not quite free" autoscaling and the threat that while you're asleep an unannounced change will take down your app (I could name a few, but as many will attest this has happened regularly since launch).

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#88

Post by nightpool »

jfoster wrote: post #66
meaty wrote: post #30

Earlier quoted context omitted.

If you were a VPS customer, you'd be less locked in and can just walk to another vendor so they actually are shit hot with support usually. It was obvious when I first tried it that GAE has crappy support.

What are we actually talking about when discussing "good support" and "bad support"? Is it just someone nice to talk to whilst someone else fixes a problem for you? There was an interesting article along these lines by the former President of Enterprise at Google written recently: http://gigaom.com/2013/01/26/the-delusions-that-companies-ha... In this case, the GAE feature that underlies this issue is the Master/Slav…

There are at least 2 people on HRD that are experiencing this issue too, per TFA

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#89

Post by tbatterii »

davedx wrote: post #4

Why would anyone put anything production critical on Google these days, knowing that they provide 0 support across most of their business?

there is support if you pay for it. in general whenever I file a ticket I get a response within hours. that's at least as good(if not better) as working with any other vendor in my experience. So... where's this 0 support you are talking about?
Post reply on HN