Live data from Hacker News

Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

code.google.com

111–120 of 129 posts

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#111

Earlier quoted context omitted.

This customer is complaining about a service component which has been deprecated since almost 11 months ago. There is a tool which migrates application data from the old datastore to the new one. When you don't move off of deprecated infrastructure, I'd say you've set yourself up for problems.

A comment from Google explaining the issue wouldn't be much to ask. "Move off of deprecated infrastructure" is much better than radio silence.

[deleted]

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#112
post #105

Earlier quoted context omitted.

> The promise was a carefree life for a project willing to code against their proprietary APIs; the reality is a constantly moving target, "not quite free" autoscaling and the threat that while you're asleep an unannounced change will take down your app (I could name a few, but as many will attest this has happened regularly since launch). Yeah, I got sucked in with the same promise and I had the exact same sour expe…

[deleted]

What? Their message is at the bottom of the linked page, with their email above it.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#113

> I'm going to assume everyone experiencing this issue is using M/S. Upgrading to HRD will solve your issue. This is the reason I abandoned AE and part of why adopting a platform that isn't standardized is incredibly dangerous. The problem is technical debt constantly accrues even when you aren't making changes . Even though the API was unchanged, HRD differs subtly enough that breakage can occur on any non-trivial p…

> This means a project written and signed off circa 2011 requires mandatory engineering costs just to continue running in a functioning and supported fashion.

Funny (?) thing is, as an engineer at Google, stuff like that happened to me ALL THE TIME. I don't even want to think about how much of my time was spent simply migrating to the "latest greatest" replacement for some critical service that was being deprecated.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#114
post #65

Earlier quoted context omitted.

"You just have to pay for support and you get support? I don't believe it, there must be more to it than that!" Whether you believe it or not, a Premier account is what you need if you want support. You could argue that $500/mo is too expensive, but it is what it is.

Oh, I believe you can pay and get support What I don't believe is that the support provided by Google is good or sufficient. Based on experiences with paying Google Apps, I'd say it's not.

My question is whether or not you personally have paid google for said support and then not received what was promised?

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#115
post #60
post #32

A GAE user sees a problem of his service being slow, writes a frantic bug report with caps and exclamation marks and threatens to leave GAE. As a GAE user myself, two questions come to mind: 1. Is GAE outside of their .9995 SLA* uptime? If they aren't, then it probably isn't important enough spend time looking into it. Customers cannot expect better than the agreed upon uptime percent, and hosting companies are oblig…

1. 0.9995 SLA means about 6 minutes of downtime a month. Since it's a daily event, I'm guessing that yes, the SLA is violated. 2. It's a problem that is occurring daily, with a test case that has pretty much no code at all. That in itself does not prove anything, but it really makes me wonder how it could be a problem on the user's side.

My math for downtime per month works out a little differently: (1-0.9995)x30x24x60 = 21.6 minutes. Still I mostly agree with you.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#116

Well, the bug report doesn't really invite quick attention. Simply reporting your observations is not enough: you should position yourself as a competent customer, by explaining what you have done to ensure the problem isn't on your side. Mention the code hasn't changed, that you have no database cleanup cronjobs or similar running that could be interfering, etc. My first instinct when I see a report like this is: he…

You pretty much only have bug reports and google groups.

As other commenters note: they DID provide a minimal test case that showed the problem wasn't their side.

Unfortunately with GAE support screaming and yelling is pretty much the only recourse.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#117
post #16

I'm sorry, but this is the price you pay for running your business that is dependent TOTALLY on a 3rd party service. Forget Google, everyone out there is most likely the same, that's why it's important for you to run your 'apps' on something you have control over - Like Linode, AWS, Rackspace, Openshift, etc. and also have back-up nodes from other providers for redundancy, for emergency situations, incase of storms,…

God DAMN it, I'm sick of seeing this comment anytime a service goes down or basically anything anywhere has a problem of any sort. We depend on third parties, all of us, to some extent. Stop it!

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#118

> I'm going to assume everyone experiencing this issue is using M/S. Upgrading to HRD will solve your issue. This is the reason I abandoned AE and part of why adopting a platform that isn't standardized is incredibly dangerous. The problem is technical debt constantly accrues even when you aren't making changes . Even though the API was unchanged, HRD differs subtly enough that breakage can occur on any non-trivial p…

The Master/Slave datastore has been deprecated for almost a year. [0] Beyond the latency issues, it simply wasn't reliable; both reads and writes were failing way too often. I'm glad GAE is focusing its resources on the HRD. [0] http://googleappengine.blogspot.com/2012/04/masterslave-data...

The problem is occurring during the same timeframe for someone else who is using the HRD: https://code.google.com/p/googleappengine/issues/detail?id=8...

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#119
post #20

Earlier quoted context omitted.

Interesting comment. When AWS went down in the US, devops rightly said "we told you to run your apps on something you have control over."

What's that? A server I own on a rack I own in a datacenter I own on a power factory I own and a telco I own? "Having control over" something is a scale, it's not binary.

Indeed, I was trying to say it is all relative.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#120
post #119

Earlier quoted context omitted.

What's that? A server I own on a rack I own in a datacenter I own on a power factory I own and a telco I own? "Having control over" something is a scale, it's not binary.

Indeed, I was trying to say it is all relative.

Sure, but it's a tradeoff. No need for devops vs no control over outages.
Post reply on HN