Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

331–340 of 646 posts

Re: Tell HN: AWS appears to be down again

#331

Post by uvdn7 »

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

You could have just showed them historical data of both companies being unavailable for extended amount of time. What happened in the past few months is not new.

Re: Tell HN: AWS appears to be down again

#332

Post by dkryptr »

JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

100% agree. I'm actually surprised AWS hasn't built in a Chaos Monkey into their APIs/console so people can test their resiliency regularly if an AZ goes down.

edit: of course, AWS does have this: AWS Fault Injection Simulator

Re: Tell HN: AWS appears to be down again

#333

Post by Nextgrid »

laumars wrote: post #330
Nextgrid wrote: post #285

Earlier quoted context omitted.

> Can you make your on prem infrastructure go down less than Amazon's? Obviously depends on what you need, but for a small to medium web app that needs a load-balancer, a few app servers, a database and a cache, yes absolutely - all of these have been solved problems for over a decade and aren't rocket science to install & maintain. > Is it worth it? I'd argue that the "worth" would be less about immunity to occasion…

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…

That's true for on-prem infrastructure, but is all already handled for you if you rent servers from hosting providers such as OVH/Hetzner or even rent colocation space in an existing DC, and is still cheaper than the cloud equivalent (and as we saw recently, actually more reliable as well).

Re: Tell HN: AWS appears to be down again

#334

Post by kaashif »

jerf wrote: post #289
lmilcin wrote: post #275

Earlier quoted context omitted.

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

I think if you put a bit of effort into classifying importance, you can likely justify backing up certain critical systems in more than one way. Let "the cloud" handle everyone's desktop backups and all the ancillary systems you don't really need immediately to do business, but certain important systems should perhaps be backed up both to the cloud and locally, like Windows Domain Controllers and other things you can…

> In this case, the game isn't "going down less than Amazon", it's about going down uncorrelated to Amazon.

It's possible to go down in a mostly uncorrelated way to Amazon by just being down all the time.

Obviously this is implicit in your comment, but I'll say it anyway: your backups need to actually work when you need them. You need to test them (really test them) to make sure they're not secretly non-functional in some subtle way when Amazon is really down.

Re: Tell HN: AWS appears to be down again

#335

Post by Godel_unicode »

Bluecobra wrote: post #238
ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

> What am I missing? My guess is that they cheaped out in having redundant PSUs to get you to use multiple availability zones. (More zones = more revenue) Even a single PSU shouldn’t be an issue if they plugged in an ATS switch though.

Unless the ATS breaks, which happens.

Re: Tell HN: AWS appears to be down again

#336

Post by robjan »

SCdF wrote: post #305
aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between…

That's the theory but in practice very few companies bother because it's expensive, complicated and most workloads or customers can tolerate less than 100% uptime.

Re: Tell HN: AWS appears to be down again

#337

Post by gpm »

SCdF wrote: post #305
aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

So dumb question from someone who hasn't maintained large public infrastructure: Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails? IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between…

> or are all these apps that are failing built wrong

Deploying to multiple places is more expensive, it's not wrong to choose not to, it's trading off reliability for cost.

It's also unclear to me how often things fail in a way that actually only affect one AZ, but I haven't seen any good statistics either way on that one.

Re: Tell HN: AWS appears to be down again

#338

Post by matharmin »

JCM9 wrote: post #303

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…

AWS is under-reporting the severity of the issue though. The primary outage may be in a single AZ, but there are parts of the AWS stack that affected all AZs in us-east-1, and potentially other regions as well. For example, even now I'm unable to create a new ElastiCache cluster in different AZs of us-east-1.

Re: Tell HN: AWS appears to be down again

#339

Post by autosharp »

dijit wrote: post #321
autosharp wrote: post #312

Earlier quoted context omitted.

Of course that depends on what services you use and yes, even then there is some remaining correlation just because it is the same host. > are not truly independent of each other Indeed. They are even on the same planet! > please stop blaming the victim Excuse me?

>> are not truly independent of each other > Indeed. They are even on the same planet! Clever bastard, aren't you. >> please stop blaming the victim > Excuse me? "If you're affected by us-east-1 outages then you're not hosting in other regions and you're doing it wrong". Except: You can be affected by this outage if you did everything right. You're putting blame on people being down for not being hosted in different…

> "If you're affected by us-east-1 outages then you're not hosting in other regions and you're doing it wrong".

You are attributing a quote to me which I never expressed, nor was that expressed elsewhere in this thread. You are even using quotation marks....

I certainly didn't mean to blame anyone. You appear to see this AWS issue as one of victims and victimizers. I was just trying to point out an agency that people may have in some situations.

Re: Tell HN: AWS appears to be down again

#340

Post by jsmith99 »

NovemberWhiskey wrote: post #314
2-718-281-828 wrote: post #203

Earlier quoted context omitted.

is that just playing with words?

I think it's predicated on a misunderstanding of what "fail-safe" actually means. For example, in railway signaling, drivers are trained to interpret a signal with no light as the most restrictive aspect (e.g. "danger"). That way, any failure of a bulb in a colored light signal, or a failure of the signal as a whole, results in a safe outcome (albeit that the train might be delayed while the driver calls up the signa…

Or nuclear reactors that fail safe by dropping all the control rods into the core to stop all activity. The reactor may be permanently ruined after that (with a cost of hundreds of millions or billions to revert) but there will be no risk of meltdown.
Post reply on HN