Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

291–300 of 646 posts

Re: Tell HN: AWS appears to be down again

#291

Post by Hippocrates »

mongrelion wrote: post #205
Hippocrates wrote: post #198

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to . But does anyone really have an objective measure of how clouds stack up reliability-wise? I doubt it, since outages and their effects are nuanced. The other move is that they want to go multi-cloud... But I’ve been involved in enough multi-cloud initiatives to know how much time and effort those soak up, not…

I agree with you. I think that having multi-AZ is the first thing to figure out before wanting to do multi-cloud, which is just another buzzword taken out of management's bullshit bucket :)

Agree, and multi AZ is usually easy. IME with AWS and GCP the control plane is the same, the scaling works across AZ, bandwidth is free and latency is near zero. The level of effort to do that is simply ticking the right boxes at setup time IME.

Re: Tell HN: AWS appears to be down again

#292

Post by woodruffw »

lmilcin wrote: post #275
quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

Not my company, but I work with another company that does (nearly?) all of their infrastructure on premise. They have pretty great uptime, in a large part because they're not dependent on the 3-4 global state mechanisms that consistently cause outages with cloud providers (DNS, BGP, AWS's role management/control plane, &c.).

I think you're right about what we over- & under-estimate, but that we also under-estimate the inflection point for when it makes sense to begin relying on major cloud services. Put another way: we over-estimate our requirements, causing us to pessimistically reach for services that have problems that we'd otherwise never have.

Re: Tell HN: AWS appears to be down again

#293

Post by TacticalCoder »

kasey_junk wrote: post #274
TacticalCoder wrote: post #164

Earlier quoted context omitted.

I should blog about this one day but... I have a server at OVH (not affiliated to them) which, at this point, I keep only for fun. It has 3162 days of uptime as I type this. 3 162 days. That's 8 years+ of uptime. Does it have the traffic of Amazon? No. Is it secure? Very likely not: it's running an old Debian version (Debian 7, which came out in, well, 2013). It only has one port opened though, SSH. And with quite a…

I read this as a cautionary tale. Here we have a server that only through the grace of god is still up, and is likely owned up. If it isn't, it's because of how little is going on with it. At its current use, it's likely not a major issue but imagine if someone saw this uptime and thought to take it as a statement of reliability and built a service on it. I for one, would want that disclosed because this is a disaste…

It works both ways: it is also a cautionary tale for those who are prone to believe it's all unreliable cattle that needs constant restart because nothing is stable nor reliable...

Re: Tell HN: AWS appears to be down again

#294

Post by pm90 »

spmurrayzzz wrote: post #256
omh2 wrote: post #226

Earlier quoted context omitted.

AWS doesn't follow their own advice about hosting multi-regional so every time us-east-1 has significant issues pretty much every AZ and region is affected. Specifically large parts of the management API, and IAM service are seemingly centrally hosted in us-east-1. So called Global endpoints are also dependent on us-east-1 and parts of AWS' internal event queues (eg. event bridge triggers) If your infrastructure is s…

Your last comment is really important, I think. I have always petitioned for "passive over active" design in distributed cloud systems. The recent outages, and also ones from the past, demonstrate why. The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for simil…

> The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better.

Agreed; however, this is somewhat difficult to do correctly. There are all sorts of systems that might have hidden dependencies on managed services. e.g. AWS IAM roles will almost always be checked at some point if your services need to interact with AWS managed services.

I think cloud providers could meet developers half way here, by providing ways to reduce API usage; but I'm not sure if it aligns with their incentives.

Re: Tell HN: AWS appears to be down again

#295

Post by jandrewrogers »

mbesto wrote: post #228
iso1631 wrote: post #211

Earlier quoted context omitted.

Not at this rate. I remember we had a power outage in 2006, it actually took one of my services off air. Since then of course that has been rectified, and the loss of a building wouldn't impact on any of the critical, essential or important services I provide.

> Not at this rate. Source? Has there ever been an industry wide survey that compares availability from "insert average colo/data center operations" with the cloud ones? And I'm not talking about "we have 12 SREs who are based in Cupertino and are all paid top dollar to support a colo"...I'm talking average .

I've worked at a few small companies over the years that had their own significant colos and/or data centers built on the cheap, and only a sysadmin or two to run them. Anecdotally, if the infrastructure is setup right, outages are very rare. Some of these were serving up massive loads at the time. I've done these build-outs a few times in my career and it isn't that hard to do, reliable software is more likely to be an issue. The only significant outage I remember is when the redundant power systems in one DC both failed at the same time for different reasons, which can happen.

It is as if the software industry has collectively forgotten how to run basic data center operations. Something that used to be a blue collar skill is now treated like arcane magic.

Re: Tell HN: AWS appears to be down again

#296

Post by dijit »

autosharp wrote: post #284
lmilcin wrote: post #275

Earlier quoted context omitted.

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

Also, you can just take two different amazon regions and hope they don't both go down at the same time. For extra safety, and extra work, you could even take Azure as a backup if you're not locked in with AWS.

forgive me repeating myself: AWS Zones are not truly independent of each other.

Global services such as route53, Cognito, the default cloud console and Cloudfront are managed out of US-East-1.

If us-east-1 is unavailable, as is commonly the case, and you depend on those systems, you are also down.

it does not matter if you're in timbuktu-1, you are dead in the water.

it is a myth that amazon availability zones are truly independent.

please stop blaming the victim, because you can do everything right and still fail if you are not aware of this; and you are perpetuating that unawareness.

Re: Tell HN: AWS appears to be down again

#297

Post by jtc331 »

mongrelion wrote: post #205
Hippocrates wrote: post #198

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to . But does anyone really have an objective measure of how clouds stack up reliability-wise? I doubt it, since outages and their effects are nuanced. The other move is that they want to go multi-cloud... But I’ve been involved in enough multi-cloud initiatives to know how much time and effort those soak up, not…

I agree with you. I think that having multi-AZ is the first thing to figure out before wanting to do multi-cloud, which is just another buzzword taken out of management's bullshit bucket :)

I’ve seen at least half a dozen full region AWS issues in the past 8 months.

You really need multi-region and also not be relying on any AWS service that’s located only in us-east-1 (including everything from creating new S3 buckets to IAM’s STS).

Re: Tell HN: AWS appears to be down again

#298

Post by dgudkov »

lmilcin wrote: post #275
quantumfissure wrote: post #261

Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

1) Can you make your on prem infrastructure go down less than Amazon's?

It's now hard to say how frequently Amazon's infrastructure goes down. The incident rate seems to have accelerated.

Re: Tell HN: AWS appears to be down again

#299

Post by badams2527 »

Nextgrid wrote: post #285
lmilcin wrote: post #275

Earlier quoted context omitted.

Think about it this way: 1) Can you make your on prem infrastructure go down less than Amazon's? 2) Is it worth it? In my experience most people grossly underestimate how expensive it is to create reliable infrastructure and at the same time overestimate how important it is for their services to run uninterrupted. -- EDIT: I am not arguing you shouldn't build your more reliable infrastructure. AWS is just a point on…

> Can you make your on prem infrastructure go down less than Amazon's? Obviously depends on what you need, but for a small to medium web app that needs a load-balancer, a few app servers, a database and a cache, yes absolutely - all of these have been solved problems for over a decade and aren't rocket science to install & maintain. > Is it worth it? I'd argue that the "worth" would be less about immunity to occasion…

Human capital side would disagree with that I think. You're assuming the organization which owns this small/medium web app has the personnel already on staff to handle such a thing.

If you're outsourcing that, you'd likely have to pay a boatload just for someone to be available for help, let alone the actual tasks themselves. Like you said, if you're on-prem and something goes down, you can do something. But you've gotta have the personnel to actually do something.

That said, I think you're spot-on as long as you have the skillset already.

Re: Tell HN: AWS appears to be down again

#300

Post by macintux »

chousuke wrote: post #279
GrumpyNl wrote: post #166

Earlier quoted context omitted.

How come they dont have power backups?

Sometimes, you have a component which fails in such a way that your redundancies can't really help. I once had to prepare for a total blackout scenario in a datacenter because there was a fault in the power supply system that required bypassing major systems to fix. Had some mistake or fault happened during those critical moments, all power would've been lost. Well-designed redundancy makes high-impact incidents less…

To my mind, among the more frustrating aspects to implementing protection against failure is that the mechanisms to be added can themselves cause failure.

It's turtles all the way down.

Post reply on HN