Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

481–490 of 497 posts

Re: Tell HN: AWS appears to be down again

#481

Post by kortex »

jedberg wrote: post #454
jjav wrote: post #445

Earlier quoted context omitted.

> * Free chaos monkey installed in every AZ Only during this beta period, AWS will start charging for this feature soon enough.

We (Netflix) begged them for years to create a Chaos Monkey that we could pay for. There were things we just couldn't do ourselves, like simulate a power pull or just drop all network packets on the bare metal. I guess not enough people asked.

CMaaS sounds amazing for resiliency engineering. There's so much I want to be doing to perturb our stack, but I don't know all the ways stuff can go wrong. Sure I can ddos it, kick services and servers offline, etc, but that's what, a few dozen failure modes? Expertise in chaos would be valuable by itself. Not to mention being able to shake parts of the system I normally can't touch.

Side note: terraform is pretty good for causing various kinds of chaos, deliberately or otherwise.

Re: Tell HN: AWS appears to be down again

#482

Post by ldoughty »

vineyardmike wrote: post #408
tyingq wrote: post #306

Earlier quoted context omitted.

West-2 also launched many years after us-east-1, so less legacy to deal with.

1.US-East-1 wasn't involved today. 2. They don't really have much "legacy" stuff to deal with since they likely turn over racks quickly across their whole fleet and software deployments should be standardized, so any US-east-1 flakiness has to do with the fact that its where amazon houses their control planes often.

There's at least one AZ in East-1 that doesn't support nitro, and that's been around for 4ish years now...

I agree in principle, but clearly something is hobbling them because of (probably) legacy stuff

Re: Tell HN: AWS appears to be down again

#483

Post by ldoughty »

NobodyNada wrote: post #405
drcongo wrote: post #150

Earlier quoted context omitted.

What does downdetector run on?

User reports — i.e. the number of people who google “is X down” and then click a Down Detector link. It’s a clever way of getting reasonably accurate data very quickly and easily, though it does have it’s flaws — the data is pretty noisy and users often attribute outages to the wrong service (e.g. blaming their ISP or Microsoft or something when YouTube is down, or vice versa).

I would guess the user is asking what are down detector's dependencies... E.g. can their website function I'd us-east-2 goes down? Or a GCP equivalent? Or are they on a self-hosted server ? What would cause the metrics to be "off"

Re: Tell HN: AWS appears to be down again

#484

Post by jethro_tell »

reaperducer wrote: post #446
jethro_tell wrote: post #412

Earlier quoted context omitted.

Apple is not a competitor to AWS or Azure in any way. They offer not infrastructure/platform as a service that I am aware of.

Apple and Amazon are competitors. Apple and Microsoft competitors. The postulation was that Apple and Amazon weren't competitors. Not that they're not competitors in a specific niche.

But the idea that Amazon or Microsoft or Google would host anything at apple is pretty out there.

Apple uses their competitor's services because they can't build their own cloud and host their own shit. The big boys don't use competitors for services they are capable of building themselves.

Re: Tell HN: AWS appears to be down again

#485

Post by brewdad »

dogecoinbase wrote: post #471
gizmodo59 wrote: post #462

Earlier quoted context omitted.

It doesn’t matter how much of resiliency engineering you do. Having everything in a single AZ is a risk. If this is acceptable then it’s fine if not you need to think of multi az from day 1.

Auth0 ran in six AZs in two regions[1] and went down today[2], because they picked the wrong two regions. How many regions and AZs should someone pay for before they get reliability? 1: https://auth0.com/blog/auth0-architecture-running-in-multipl... 2: https://twitter.com/auth0/status/1471159935597793290

At a minimum they should have chosen regions not in the same time zone or general geographic area. US-West 1 and US-West 2 might well be safeguarding against a server failure but is not a disaster plan. If your customers are global, choosing multiple continents is probably prudent.

Re: Tell HN: AWS appears to be down again

#486

Post by ec109685 »

jjav wrote: post #450
orourkek wrote: post #424

Earlier quoted context omitted.

> Taking badges out of the cloud reduces points of failure by several orders of magnitude. I'm not convinced that's true, or at least certainly not an order of magnitude. Wouldn't a badge system hosted on-prem also need a user management system (database), a hosted management interface, have a dependency on the LAN, and need most of the same hardware? Such a system would also need to be running on a local server(s),…

The remote solution requires all of those same things, plus in addition it requires internet connectivity to be up and reliable, the cloud provider be available and the third party company be up and still in business. Adding complexity and moving parts never reduces points of failure. It can reduce daily operating worries as long as everything works, but it can't reduce points of failure. It also means than someday w…

Within the building’s on premise hosted infrastructure, are they going to buy multiple racks and multiple servers spread far enough apart so that there aren’t many single points of failure that will bring the badge machine down if they fail?

Re: Tell HN: AWS appears to be down again

#487

Post by Corrado »

gizmodo59 wrote: post #462
kenhwang wrote: post #449

Earlier quoted context omitted.

If my company is any indication, they're going to make more money since everyone will simply check the multi-AZ or multi-region checkboxes they didn't before and throw more money at the problem instead of doing proper resiliency engineering themselves.

It doesn’t matter how much of resiliency engineering you do. Having everything in a single AZ is a risk. If this is acceptable then it’s fine if not you need to think of multi az from day 1.

Whelp, I guess you're not using Cognito then. It has no user account syncing feature so you can't have a user group in more than one region. Grrrrr!

Re: Tell HN: AWS appears to be down again

#489

Post by rfoo »

elasticventures wrote: post #480
staticassertion wrote: post #167

Earlier quoted context omitted.

> As annoying as those Rust evangelists on every single memory corruption bug. First of all, how dare you! Second, shoulda used rust ¯\_(ツ)_/¯

I could have written the OP message a year ago -- I used to feel the same way. Plz don't disparage Rust evangelism! Rust is awesome. yes it is complex, frequently annoying, easy to learn difficult to master. I'm speaking from a 30 year dev career. a few months ago I intended to do a quick investigation into RUST to validate my "i really don't need to learn this" specifically for an embedded project. Within a few hour…

While I'm not sure whether you're serious ;), to be clear: what annoys me is they don't really understand why we are having "someone pwned your phone via a series of memory corruption bugs" daily.

Until those Rust evangelists managed to rewrite the world with Rust (and I promise you there still will be a lot of security bugs), we still have to fix our shit in a low-cost way and their evangelism does not help at all and is pure annoyance.

Re: Tell HN: AWS appears to be down again

#490

Post by iso1631 »

dilyevsky wrote: post #453
electroly wrote: post #321

Earlier quoted context omitted.

We don't have any external customers; they are all internal. We're all on the same side of the table.

Sounds like even worse deal for the customer since there is no refund

Depends on the service are

I have some services which can cope with a 98.5% downtime, as long as they are available the specific 1.5% of the time we need them to run, as such "the cloud" is useless for that service

Post reply on HN