Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

App Engine down

code.google.com

Post reply on HN
Print view View original

121–130 of 139 posts

Re: App Engine down

#121

Post by stickfigure »

debacle wrote: post #116
stickfigure wrote: post #113

Earlier quoted context omitted.

In that case, what is the ratio of "time spent doing ops-related tasks" vs "time spent developing new features" in your company? Please offer an honest evaluation. Everything has a cost; I'm genuinely curious about data points other than my own.

I probably spend no more than an hour a week on ops, and most of that is reading emails from our service providers.

Today, maybe, assuming a calm ocean and no scaling issues. But I don't believe you spent an hour a week setting up your three data centers, backups, failover procedure, etc.

Re: App Engine down

#122

Post by magicalist »

fidotron wrote: post #55

It's time we remembered the whole strength of the internet was that it was distributed and we avoided introducing single points of failure. We have ended up using vast amounts of infrastructure for no reason other than developer convenience (often with respect to security), when having local direct connections is often more suitable than shooting everything into the cloud.

Huh? Historically, almost all content on the internet has had a single point of failure. Moving "into the cloud" is moving to a distributed model, and generally app engine will protect you from single points of failure. As others have noted, if you distribute your own self-managed servers across the globe, you end up with a very similar system, but now you have to manage it.

What appears to be happening here is that you're still vulnerable because the C&C infrastructure ultimately has a single command source, and so can be vulnerable if some code is pushed that affects the whole system. Your homegrown cloud will suffer from the same vulnerability, it may just be more or less easy to manage depending on how specialized your needs are compared to the more general requirements of GAE.

Edit: actually, maybe I missed what you're advocating with "local direct connections". This might make more sense from a user's perspective: if everyone ran their own little cloud, a failure may bring down reddit, but not reddit, heroku, pinterest, etc simultaneously. That's actually an interesting point, but I'm not sure if it really matters if they sync up their downtimes (since they would still have downtime, and maybe more or less of it depending on how much they could afford to invest into managing their distributed solution). I'm also not sure if that really solves the problem, since there are other concentrations in the network, just less visible ones (there are a fairly small number of major datacenters around the world, for instance, and managing your own colocated server doesn't matter if the whole building goes dark).

I do agree that at the very least we need to maintain an ecosystem of "cloud" providers, however.

Re: App Engine down

#123

Post by latchkey »

MatthewPhillips wrote: post #120
latchkey wrote: post #117

Earlier quoted context omitted.

I keep hearing the lock-in argument over and over and I'm not quite sure that there is a solid basis for it beyond a level of paranoia. It's fine to be paranoid, I just don't want it to hold me back unnecessarily. Looking at things more closely, the only thing that you're truly locked into with GAE is the esoteric nature of the datastore. This isn't any worse than picking say MySQL vs. Oracle or Riak vs. Mongo. Most…

> Looking at things more closely, the only thing that you're truly locked into with GAE is the esoteric nature of the datastore. This isn't any worse than picking say MySQL vs. Oracle or Riak vs. Mongo. Sure it is. If you pick the wrong DB your only locked into that DB. You pick GAE and you're locked into GAE's DB.... and GAE. I can move my MySql db to another cloud provider.

Your analogy of moving to another cloud provider is incorrect. Unless you've been 100% database agnostic, you can't just migrate your schema and data from say MySQL to Postgres.

With the datastore on GAE, you can get your data out of it and move it into something like MongoDB. I'd argue that it would probably be less code to move to MongoDB because the datastore has all sorts of esoteric issues that you have to code around (like the way that entity groups and transactions are handled).

In terms of the rest of your application, it is just a standard webapp in whatever platform you choose. The .war file I have for GAE will run just fine in Tomcat. The only real lockin is the way you store your data.

Re: App Engine down

#124

Post by mhurron »

fidotron wrote: post #55

It's time we remembered the whole strength of the internet was that it was distributed and we avoided introducing single points of failure. We have ended up using vast amounts of infrastructure for no reason other than developer convenience (often with respect to security), when having local direct connections is often more suitable than shooting everything into the cloud.

The resiliency of the internet has nothing to do with the uptime of any service running at endpoints. The resiliency is about the resiliency of the network, a router goes down, there are paths around it, not necessarily paths to every endpoint hanging off it or the services they offer.

Re: App Engine down

#125

Post by daave »

And they've sent the all-clear:

At this point, we have stabilized service to App Engine applications. App Engine is now successfully serving at our normal daily traffic level, and we are closely monitoring the situation and working to prevent recurrence of this incident.

This morning around 7:30AM US/Pacific time, a large percentage of App Engine’s load balancing infrastructure began failing. As the system recovered, individual jobs became overloaded with backed-up traffic, resulting in cascading failures. Affected applications experienced increased latencies and error rates. Once we confirmed this cycle, we temporarily shut down all traffic and then slowly ramped it back up to avoid overloading the load balancing infrastructure as it recovered. This restored normal serving behavior for all applications.

We’ll be posting a more detailed analysis of this incident once we have fully investigated and analyzed the root cause.

Regards,

Christina Ilvento on behalf of the Google App Engine Team

https://groups.google.com/forum/#!topic/google-appengine-dow...

Re: App Engine down

#126

Post by gtaylor »

peterwwillis wrote: post #115
gtaylor wrote: post #100

Earlier quoted context omitted.

This is a much more succinct explanation than my own. This is the point I was trying to make. Even given that I have a good bit of sysadmin skills, I am needed more as a software developer right now in the early goings. I expect, as you've pointed out, that priorities will change with time and growth. We may even move to bare metal eventually, if we find ourselves needing and able to do so. Excellent analogy.

So to summarize you both: You want a rapid development platform that doubles as a production system and costs nothing to maintain. That does sound useful!

What? Did you read any of what was written? Point-by-point breakdown for you:

> You want a rapid development platform

No, we want low-maintenance infrastructure.

> that doubles as a production system

It is a production system. It does successfully serve many thousands of users for us every day. We've yet to have an outage that wasn't our own fault.

> and costs nothing to maintain

What? I specifically said we're willing to pay more not to have to spend as much time on infrastructure.

It's OK if you're too set in your ways to even attempt to level with alternative points of view, but at least try to read a little more thoroughly. And maybe admit that you're not willing to budge, so nobody wastes time trying to explain an alternative point of view.

Re: App Engine down

#128

Post by marekmroz »

MatthewPhillips wrote: post #120
latchkey wrote: post #117

Earlier quoted context omitted.

I keep hearing the lock-in argument over and over and I'm not quite sure that there is a solid basis for it beyond a level of paranoia. It's fine to be paranoid, I just don't want it to hold me back unnecessarily. Looking at things more closely, the only thing that you're truly locked into with GAE is the esoteric nature of the datastore. This isn't any worse than picking say MySQL vs. Oracle or Riak vs. Mongo. Most…

> Looking at things more closely, the only thing that you're truly locked into with GAE is the esoteric nature of the datastore. This isn't any worse than picking say MySQL vs. Oracle or Riak vs. Mongo. Sure it is. If you pick the wrong DB your only locked into that DB. You pick GAE and you're locked into GAE's DB.... and GAE. I can move my MySql db to another cloud provider.

@latchkey I think you are missing the point the MatthewPhillips is making. With GAE's DB there is nowhere to move, so if you want to move you would have to migrate your data to a different DB engine. If your hosting/cloud provider uses something standard like MySQL you can find another provider or roll your own if you decide to migrate out.

Re: App Engine down

#129

Post by debacle »

stickfigure wrote: post #121
debacle wrote: post #116

Earlier quoted context omitted.

I probably spend no more than an hour a week on ops, and most of that is reading emails from our service providers.

Today, maybe, assuming a calm ocean and no scaling issues. But I don't believe you spent an hour a week setting up your three data centers, backups, failover procedure, etc.

The backup script was written in a night, and the most complicated part about failover is remembering to sync data when the outage is over.

Re: App Engine down

#130

Post by riffraff »

latchkey wrote: post #117
acdha wrote: post #114

Earlier quoted context omitted.

I completely agree that Google is likely to do better than many teams, modulo your second point (which I completely agree with - generic is much harder than specific). For me it really just comes down to the lock-in aspect: with GAE if you decide that Google isn't taking the platform in the right direction for your business you're looking at something close to rewriting your application. This is far from the most lik…

I keep hearing the lock-in argument over and over and I'm not quite sure that there is a solid basis for it beyond a level of paranoia. It's fine to be paranoid, I just don't want it to hold me back unnecessarily. Looking at things more closely, the only thing that you're truly locked into with GAE is the esoteric nature of the datastore. This isn't any worse than picking say MySQL vs. Oracle or Riak vs. Mongo. Most…

> The only reason to migrate away from GAE would be if you find out that your application doesn't work well on it (pricing, scalability, etc) or if Google decides to kill GAE entirely. Hopefully you do the analysis of your application before you decide to use GAE (ie: you can't blame GAE for you deciding to use it)

This seem too simplistic. Off the top of my head, you may leave because:

* you want to do something new in your app that is not possible/effective to do within GAE * you find another option that is cheaper * you find that some of the assumptions and architectural decisions you took on day 0 no longer hold * you did the initial analysis, and it was wrong

I do agree with GAE being awesome and lock-in not being so bad, but I doubt the "you have it all figured out before building it so it's going to work forever" idea.

Post reply on HN