Live data from Hacker News

Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

code.google.com

21–30 of 129 posts

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#21

Earlier quoted context omitted.

because we're paying customers? my bill is peanuts, but i know there are many big customers, e.g. Khan Academy. and also they have Premier Support which is $500/mo.

One thing I've learned with Google is that they don't give a rat's ass if you're a paying customer or not. Khan Academy may have it easier, I'm sure Google won't let them down Really, go somewhere else, spend less money and have better support. (At the expense of, if you're lucky, Google will give you almost zero headaches)

There's nothing about Khan Academy's application (or any other customer's) that would somehow immunize it from platform-wide serving issues.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#23
post #19

Having never used GAE, it would nice if someone could expand M/S and HRD for me. It looks like OP of the bug-report is using a depreciated feature/program which according to the Project Member is causing latency issues at a specific time daily. But that could not be the real issue since another commentator who is using the new HRD is also having the same problem. It is even frustrating for people who are reading this…

[deleted]

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#24
post #16

I'm sorry, but this is the price you pay for running your business that is dependent TOTALLY on a 3rd party service. Forget Google, everyone out there is most likely the same, that's why it's important for you to run your 'apps' on something you have control over - Like Linode, AWS, Rackspace, Openshift, etc. and also have back-up nodes from other providers for redundancy, for emergency situations, incase of storms,…

The Openshift looks quite promising as a PaaS. If anything goes wrong with RH you can just move your app to other host or even your own hardware.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#25
Ok, so here's the deal. If your app runs exclusively on GAE you've essentially tied yourself to one cloud vendor. Now disregarding the respective benefits and drawbacks of google as a hosting company for your app (I would never do that), being dependent on one cloud provider is a very bad idea. No matter if you run on EC2, Azure or GAE, if you can't seamlessly switch to another provider, you're screwed. These all go down regularly and have issues. They're big companies, you're a small company, you have no such thing as "recourse". The court of public opinion will not save your company.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#26
Remember the old "thundering herd" problem with Apache children and things of that nature? You'd basically have a whole bunch of processes which had a listening fd from an earlier call to listen(). When a new connection would come in, the kernel would wake all of them, even though only one of them would actually have something to get. The others would go through the process for nothing. It caused a big performance hit back in the day.

Well, imagine now that you have a directory or lock service where you can store things and perform atomic updates. When you do a write to something in it, it fans out to all of its clients, and they all wake up (nearly) simultaneously and receive the update. They then have to do whatever processing you do with new data of that type.

If they all do this at the same time, then you have no processes left to service incoming requests. They're all identically busy with whatever mutexes held in order to apply those config changes safely, so no other work happens on those clients while they load in the new data.

It's not so much that it's taking a mutex and is getting stuck for a little bit, since that's going to happen no matter what. It's that all of the children do it at the same time, so there's nobody to service your hit, and you're guaranteed to get stuck. If it was spread out, then only some percentage of incoming requests would get stuck behind this. The others would get lucky and would hit another instance which either had already run it or hadn't yet run it.

I'm not saying this is what's going on here, but it sure sounds familiar.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#28

Earlier quoted context omitted.

One thing I've learned with Google is that they don't give a rat's ass if you're a paying customer or not. Khan Academy may have it easier, I'm sure Google won't let them down Really, go somewhere else, spend less money and have better support. (At the expense of, if you're lucky, Google will give you almost zero headaches)

There's nothing about Khan Academy's application (or any other customer's) that would somehow immunize it from platform-wide serving issues.

Of course not, like when AWS fails Netflix stops working like every customer.

But you can bet that if Khan Academy has a problem it will be looked into with extra attention.

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#29
post #20
post #16

I'm sorry, but this is the price you pay for running your business that is dependent TOTALLY on a 3rd party service. Forget Google, everyone out there is most likely the same, that's why it's important for you to run your 'apps' on something you have control over - Like Linode, AWS, Rackspace, Openshift, etc. and also have back-up nodes from other providers for redundancy, for emergency situations, incase of storms,…

Interesting comment. When AWS went down in the US, devops rightly said "we told you to run your apps on something you have control over."

More relevant would be "where's your DR site?"

Re: Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

#30

Well, the bug report doesn't really invite quick attention. Simply reporting your observations is not enough: you should position yourself as a competent customer, by explaining what you have done to ensure the problem isn't on your side. Mention the code hasn't changed, that you have no database cleanup cronjobs or similar running that could be interfering, etc. My first instinct when I see a report like this is: he…

I consider him panicking more than shouting and threatening. I couldn't imagine having that kind of treatment as a vps customer else I'll be moving out asap.

If you were a VPS customer, you'd be less locked in and can just walk to another vendor so they actually are shit hot with support usually.

It was obvious when I first tried it that GAE has crappy support.

Post reply on HN