Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

301–310 of 646 posts

Re: Tell HN: AWS appears to be down again

#301

Post by patentatt »

lmilcin wrote: post #275
quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

On the other hand, perhaps the large cloud providers bring a level of complexity that outweighs their skills at keeping everything up. What I mean is, a basic redundancy and failover setup with two data centers is kind of straightforward. Sure you need a person on call 24/7 to oversee it, but it's conceptually not that complicated. And if you're running bare metal, you get a surprising level of performance per dollar and rack unit. On the other hand, the big clouds are immensely complex with multiple layers of software defined networking, millions of tenants, thousands of employees, acres of floor space, org charts, etc. If you're running your own infra as one competent sysadmin, you know nobody else in another department will push a networking code change that will break your shit in the middle of the night. Maybe it's not right for everyone, but it's not unreasonable to go on prem in 2021 despite the popular opinions otherwise. Source: my company runs on prem and routinely has 100% uptime years. Most unplanned downtime occurs early on a Sunday morning following a planned action during a maintenance window.

Re: Tell HN: AWS appears to be down again

#302

Post by dijit »

growse wrote: post #288
ItsBob wrote: post #224

I've built out many 42U racks in DC's in my time and there were a couple of rules that we never skipped: 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens 2. Dual network (at minimum) - For the same reasons as above since the switches didn't always have dual power…

> 1. Dual power in each server/device - One PSU was powered by one outlet, the other PSU by a different one with a different source meaning that we can lose a single power supply/circuit and nothing happens Nothing happens if you remember that your new capacity limit per DC supply is 50% of the actual limit, and you're 100% confident that either of your supplies can seamlessly handle their load suddenly increasing by…

EDIT: I misunderstood you were talking about power feeds, the normal case is the run "48% as if it's 100%" (because of power spikes, but also most types of transformers run more efficiently under specific levels of load (40-60).

Normally this is factored into the Rack you buy from a hardware provider, they will tell you that you have 10A or 16A on each feed, if you exceed that: it will work, but you are overloading their feed and they might complain about it.

Re: Tell HN: AWS appears to be down again

#303

Post by JCM9 »

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs.

Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZs, typically by just dropping nodes in the down AZ.

People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down.

Re: Tell HN: AWS appears to be down again

#304

Post by notyourday »

bombcar wrote: post #250
notyourday wrote: post #241

Earlier quoted context omitted.

> I've only had a DC fail once when the engineer was performing work on the power circuitry for the DC and thought he was taking down one, but was in fact the wrong one and took both power circuits down at the same time. This is all local scale. Your setup would not survive a data center scale power outage. At scale power outages are datacenter scale. Data centers lose supply lines. They lose transformers. Sometimes…

The datacenter we were in had dual-sourced grid power (two separate grid connections on opposite sides of the block, coming from different substations) along with a room of batteries (good for iirc 1hr total runtime for the whole datacenter, setup in quad banks, two on each "rail"), _and_ multiple independent massive diesel generators, which they ran and switched power to every month for at least an hour . And to top…

We were in a triple sourced data center. Fed by three different substations. Everything worked like a charm. Until Sandy hit. It did not affect us at all. But it affected the power company. And everything still worked fine, until one of the transfer switches transferred into UPS position and stopped working in that position.

Re: Tell HN: AWS appears to be down again

#305

Post by SCdF »

aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

So dumb question from someone who hasn't maintained large public infrastructure:

Isn't the whole point of availability zones is that you deploy to more than one and support failing over if one fails?

IE why are we (consumers) hearing about this or being obviously impacted (eg Epic Games Store is very broken right now)? Is my assessment wrong, or are all these apps that are failing built wrong? Or something in between?

Re: Tell HN: AWS appears to be down again

#306

Post by Godel_unicode »

londons_explore wrote: post #276
nightpool wrote: post #253

Earlier quoted context omitted.

Amazon is claiming the failure is limited to a single AZ. Are you seeing failures for instances outside of that AZ? If not, how has this rendered "the entire region almost unusable"?

A lot of people will automatically fail over jobs to other AZ's. That often involves spinning up lots more EC2 instances and moving PB's of data. The end result is all capacity on other AZ's gets used up, and networks get full to capacity, and even if those other zones are technically working, practically they aren't really usable.

That doesn't appear to have happened though, I haven't seen issues outside az4

Re: Tell HN: AWS appears to be down again

#307

Post by Nextgrid »

wallacoloo wrote: post #220
schnebbau wrote: post #25

So, how many execs are going to push to move to self-managed hosting in the new year? Packaging a way to migrate off AWS could be a unicorn idea.

AWS has its Outpost product for on-prem hosting. not 100% self-managed, but maybe enough to satisfy the execs and make your market a bit smaller.

Does it come with its own locally-hosted console or does it still rely on the main AWS control plane? If the latter then it could be affected too.

Re: Tell HN: AWS appears to be down again

#308

Post by xyst »

aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

This region in general is a clusterfuck. If companies by now do not have a disaster recovery and resiliency strategy in place, you are just shooting yourself in the foot.

Re: Tell HN: AWS appears to be down again

#309

Post by jerf »

marcosdumay wrote: post #257
2-718-281-828 wrote: post #203

Earlier quoted context omitted.

is that just playing with words?

You mean to ask if it's a joke? Yes, it's a joke. Or you ask if it's a lesson about how real systems operate? Because yes, it's a very serious lesson about how real systems operate. Anyway, you seem out of grasp on system engineering. Your reply downthread isn't applicable (of course fail-safes can fail, anything can fail). If you want to learn more on this area (not everybody wants, and its ok), following that link…

"Notice that there is a huge amount of handwaving in system engineering. I don't think this is good, but I don't think it's avoidable either."

In my experience, you can be specific, but then you get the problem that people think that if they just 'what if' a narrow solution to the particular problem you're presenting they've invalidated the example, when the point was 1. that this is a representative problem, not this specific problem and 2. in real life you don't get a big arrow pointing at the exact problem 3. in real life you don't have one of these problems, your entire system is made out of these problems, because you can't help but have them, and 4. availability bias: the fact that I'm pointing an arrow at this problem for demonstration purposes makes it very easy to see, but in real life, you wouldn't have a guarantee that the problem you see is the most important one.

There's a certain mindset that can only be acquired through experience. Then you can talk systems engineering to other systems engineers and it makes sense. But prior to that it just sounds like people making excuses or telling silly stories or something.

"(of course fail-safes can fail, anything can fail)"

Another way to think of it is the correlation between failure. In principle, you want all your failures to be uncorrelated, so you can do analysis assuming they're all independent events, which means you can use high school statistics on them. Unfortunately, in real life there's a long tail (but a completely real tail) of correlation you can't get rid of. If nothing else, things are physically correlated by virtue of existing in the same physical location... if a server catches fire, you're going to experience all sorts of highly correlated failures in that location. And "just don't let things catch fire" isn't terribly practical, unfortunately.

Which reiterates the theme that in real life, you generally have very incomplete data to be operating on. I don't have a machine that I can take into my data center and point at my servers and get a "fire will start in this server in 89 hours" readout. I don't get a heads up that the world's largest DDOS is about to be fired at my system in ten minutes. I don't get a heads up that a catastrophic security vulnerability is about to come out in the largest logging library for the largest language and I'm going to have a never-before-seen random rolling restart on half the services in my company with who knows what consequences. All the little sample problems I can give in order to demonstrate systems engineering problems imply a degree of visibility you don't get in real life.

Re: Tell HN: AWS appears to be down again

#310

Post by jmartrican »

quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Seems like multi-cloud solution might be the way to go.
Post reply on HN