Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

App Engine down

code.google.com

Post reply on HN
Print view View original

131–139 of 139 posts

Re: App Engine down

#131

Post by sktrdie »

acdha wrote: post #114
stickfigure wrote: post #108

Earlier quoted context omitted.

GAE applications are distributed across multiple data centers[1], so in theory you get "scale wide" automatically. Unfortunately it looks like there was some sort of flaw in the architecture. I believe this is the first systemwide failure of the HRD. The real question is: Can you and your ops team build a "scale wide" system better than Google?[2] How much effort are you willing to put into it, when those development…

I completely agree that Google is likely to do better than many teams, modulo your second point (which I completely agree with - generic is much harder than specific). For me it really just comes down to the lock-in aspect: with GAE if you decide that Google isn't taking the platform in the right direction for your business you're looking at something close to rewriting your application. This is far from the most lik…

With the Java API you can write code that is standard (JDO/JPA for database stuff), that works across any Java environment. So it doesn't lock you in if you write your code in a smart way.

Re: App Engine down

#132

Post by Tobu »

abhijitr wrote: post #44

Meanwhile... Gmail etc are working quite fine. So the claim that if you build on GAE you "take advantage of the same infrastructure used for Google services!!" starts to ring a bit hollow.

I know the claim of running Amazon.com on AWS was initially bogus, but did anyone notice trouble on that site with the recent AWS outages? It would be instructive to know how much they have integrated it since.

Re: App Engine down

#133

Post by prodigal_erik »

lurker14 wrote: post #65
fidotron wrote: post #55

It's time we remembered the whole strength of the internet was that it was distributed and we avoided introducing single points of failure. We have ended up using vast amounts of infrastructure for no reason other than developer convenience (often with respect to security), when having local direct connections is often more suitable than shooting everything into the cloud.

Which is better? Having a day of downtime each year, or not launching at all?

That depends on how much long-term reputation damage you take from having availability that low before a large audience. Which in turn depends on how important your service really is—"can't play my game" is drastically different than "my landlord didn't receive my payment".

Re: App Engine down

#134

Post by rbanffy »

acdha wrote: post #84
davidjgraph wrote: post #6

Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening. We left AWS about 18 months ago after one of the outages and switched to GAE. I've counted 3-4 big downtimes for AWS compared to this one on GAE. That's still a good decision (for now)....

One thing to remember: this took down all of app engine for at least an hour. AWS has had only 17 minutes of downtime affecting all of us-east this year (that network glitch a couple days after PyCon) - the rest of it has been a subset of the service amplified by people rediscovering that they weren't as redundant as they thought. The correct less to draw is that any one point of infrastructure is a risk, so you need…

> you're committed to a single-vendor API as well as their infrastructure.

Not quite. You can roll out your own App Engine platform with AppScale:

http://appscale.cs.ucsb.edu/

Sadly, TyphoonAE (http://code.google.com/p/typhoonae/) seems abandoned.

Re: App Engine down

#135

Post by philwelch »

magicalist wrote: post #122
fidotron wrote: post #55

It's time we remembered the whole strength of the internet was that it was distributed and we avoided introducing single points of failure. We have ended up using vast amounts of infrastructure for no reason other than developer convenience (often with respect to security), when having local direct connections is often more suitable than shooting everything into the cloud.

Huh? Historically, almost all content on the internet has had a single point of failure. Moving "into the cloud" is moving to a distributed model, and generally app engine will protect you from single points of failure. As others have noted, if you distribute your own self-managed servers across the globe, you end up with a very similar system, but now you have to manage it. What appears to be happening here is that…

It's distributed for you, but with an oligopoly of cloud providers it's considerably less distributed overall.

Re: App Engine down

#136

Post by codeka »

peterwwillis wrote: post #79
tomgallard wrote: post #64

Earlier quoted context omitted.

But a DNS based failover is still going to take an hour or so to propagate right (given that a lot of browsers/proxies/DNS servers don't respect TTL very well at all)? And then you end up with a system with stale data, and the mess of trying to reconcile it when your other system comes back up. I'd take an hour long Appengine outage once a year over that anytime!

Your name server or stub resolver is what respects DNS TTL, not your browser or proxy. Everyone - including people hosting on AWS - needs to be able to fail over DNS, if the AWS IP you're using is in a zone that just went down, for example. Any time you have an outage you need to contact your service provider to get an estimate of downtime. If they can't give you one, assume it'll take forever and cut the DNS over. T…

I've seen plenty of crappy ISP DNS servers ignore TTL values and cache DNS entries for many hours longer than they're supposed to. Unfortunately, it's all too common.

Re: App Engine down

#137

Post by stickfigure »

riffraff wrote: post #130
latchkey wrote: post #117

Earlier quoted context omitted.

I keep hearing the lock-in argument over and over and I'm not quite sure that there is a solid basis for it beyond a level of paranoia. It's fine to be paranoid, I just don't want it to hold me back unnecessarily. Looking at things more closely, the only thing that you're truly locked into with GAE is the esoteric nature of the datastore. This isn't any worse than picking say MySQL vs. Oracle or Riak vs. Mongo. Most…

> The only reason to migrate away from GAE would be if you find out that your application doesn't work well on it (pricing, scalability, etc) or if Google decides to kill GAE entirely. Hopefully you do the analysis of your application before you decide to use GAE (ie: you can't blame GAE for you deciding to use it) This seem too simplistic. Off the top of my head, you may leave because: * you want to do something new…

FWIW, I frequently want to do things that are not possible/effective on GAE. And so I do it - several of my GAE apps communicate with services I set up in rackspace cloud (30-40 ms of latency away). It would probably be even lower running in Google's cloud service, but I haven't gotten an invite yet.

GAE is not "all or nothing". You can still run exotic services in other hosts. Or, for that matter, use GAE for specific services in your "other cloud" app. You get two bills. Not much of a downside.

Re: App Engine down

#138

Post by peterwwillis »

gtaylor wrote: post #126
peterwwillis wrote: post #115

Earlier quoted context omitted.

So to summarize you both: You want a rapid development platform that doubles as a production system and costs nothing to maintain. That does sound useful!

What? Did you read any of what was written? Point-by-point breakdown for you: > You want a rapid development platform No, we want low-maintenance infrastructure. > that doubles as a production system It is a production system. It does successfully serve many thousands of users for us every day. We've yet to have an outage that wasn't our own fault. > and costs nothing to maintain What? I specifically said we're willi…

I wasn't dismissing your point of view. Heroku is what I described. AWS, to a certain extent, is what I described. And I said it was a production system, too; you didn't have to over-emphasize what I had already said, as if I didn't just say it. "Costs nothing to maintain" is in comparison to paying to maintain it yourself. But thanks for knee-jerking.

Re: App Engine down

#139

Post by Nitramp »

sktrdie wrote: post #131
acdha wrote: post #114

Earlier quoted context omitted.

I completely agree that Google is likely to do better than many teams, modulo your second point (which I completely agree with - generic is much harder than specific). For me it really just comes down to the lock-in aspect: with GAE if you decide that Google isn't taking the platform in the right direction for your business you're looking at something close to rewriting your application. This is far from the most lik…

With the Java API you can write code that is standard (JDO/JPA for database stuff), that works across any Java environment. So it doesn't lock you in if you write your code in a smart way.

Except that writing code using JDO/JPA is a nightmare, at least on Datastore. So that's almost certainly not the smart way. I'd rather rewrite my app three times - go with Objectify (simple, easy, maps well to Datastore concepts) and just deal with the rewrite if it happens.
Post reply on HN