Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How when AWS was down, we were not

authress.io

Post reply on HN
Print view View original

61–70 of 79 posts

Re: How when AWS was down, we were not

#61

Post by wparad »

lorrin wrote: post #45
scottlamb wrote: post #43

I'm surprised the section about retries doesn't mention correlations. They say: > P_{total}(Success) = 1 - P_{3rdParty}(Failure)^{RetryCount} By treating P_{3rdParty}(Failure) as fixed, they're assuming a model in which each each try is completely independent: all the failures are due to background noise. But that's totally wrong, as shown by the existence of big outages like the one they're describing, and not consi…

Agreed, I think the introduction is wrong and detracts from the rest of the article.

Hmmm, which part of the intro did you find an issue with? I want to see if I can fix it.

Re: How when AWS was down, we were not

#62

Post by wparad »

bostik wrote: post #56
thisnullptr wrote: post #49

It’s fascinating to me people think their services are so important they can’t survive any downtime. Can we all admit that, while annoying, nothing really bad happened even when us-east-1 was down for almost half a working day?

As other posters have commented, an external auth service is a very special thing indeed. In modern and/or zero-trust systems if auth doesn't work, then effectively nothing works. My rule of thumb from the past experiences is that if you demand a 99.9% uptime for your own systems and you have an in-house auth, then that auth system must have 99.99% reliability. If you are serving auth for OTHERS, then you have a syst…

The sad truth of the world is that in many cases latency isn't the most critical aspect for tracking. We absolutely do track it because we have the expectation that authentication requests complete. But there are many moving parts to this that make reliable tracking not entirely feasible: * end location of user * end location of customer service * third party login components (login with google, et al) * corporate identity providers * webauthn * customer specific login mechanism workflows * custom integrations for those login mechanisms * user's user agent * internet connectivity

All of those significantly influence the response capability in a way which makes tracking latency next to useless. Maybe there is something we can be doing though. In more than a couple scenarios we do have tracking in place, metrics, and alerting, it just doesn't end up in our SLA.

Re: How when AWS was down, we were not

#63

Post by wparad »

shoo wrote: post #54
thisnullptr wrote: post #49

It’s fascinating to me people think their services are so important they can’t survive any downtime. Can we all admit that, while annoying, nothing really bad happened even when us-east-1 was down for almost half a working day?

In many contexts you are correct & further, as someone in that earlier thread about the AWS us-east-1 outage mentioned, customers can be more forgiving of outages if you as the vendor can point to a widespread AWS us-east-1 outage and note that us-east-1 is down for everyone. But, as JSR_FDED's sibling comment notes & as is spelled out in the article, authress' business model offering an auth service means that their…

Absolutely, part of the problem is that a whole region being down is often less of a problem, then just one critical service. And as you point out the blast radius of a critical dependency is huge.

Re: How when AWS was down, we were not

#66

Post by darkwater »

yearolinuxdsktp wrote: post #38
xyzzy123 wrote: post #33

Earlier quoted context omitted.

I actually like terraform for its LACK of power (tho yeah these days when I have a choice I use a lot of small states and orchestrate with tg). Pulumi or CDK are for sure more powerful (and great tools) but when I need to reach for them I also worry that the infra might be getting too complex.

IMO Pulumi and CDK are an opportunity to simplify your infra by capturing what you’re working with using higher-level abstractions and by allowing you to refactor and extract reusable pieces at any level. You can drive infra definitions easily from typed data structures, you can add conditionals using natural language syntax, and stop trying to program in a configuration language (Terraform HCL with surprises like no…

> and stop trying to program in a configuration language

Many people don't program with a configuration language like HCL. We use it as what it is - a DSL - that covers its main use case in an elegant manner. Maybe I never touched complex enough infra that twists a DSL into a general-use language, but in my experience there are simply no real benefits when using something like CDK (I never tried Pulumi to be fair).

Re: How when AWS was down, we were not

#67

Post by loadbalancer »

indigodaddy wrote: post #29
indigodaddy wrote: post #12

Back in the day (10-12 years ago) at a telecom/cable we accomplished this with F5 Big IP GSLB DNS (and later migrated to A10's GSLB equivalent devices) as the auth DNS server for services/zones that required or were suitable for HA. (I can't totally remember but I'm guessing we must have had a pretty low TTL for this). Had no idea that Route 53 had this sort of functionality

Speaking of F5 Big IP DNS devices, does anyone know of any auth DNS software solution for GSLB/health checking for DNS (I guess excluding Route 53 or other cloud/SaaS). Last I looked all I could find was the polaris-gslb addon for PowerDNS, but the GitHub for that has no activity in 8 years.

We've been using Polaris for the Loadbalancer.org GSLB for a few years now, and we've found it fast and stable. Its so simple its never needed any updates:

https://www.loadbalancer.org/blog/gslb-why-global-server-loa...

Although we did patch a dynamic health check a while back, which will be open source of course. But I'll get someone to check if we actually gave it back to the community or not...

Re: How when AWS was down, we were not

#68

Post by wparad »

sam-cop-vimes wrote: post #59

What a well written article! Nothing complex is built overnight, so it is interesting to see how their defenses have evolved to their current state. Requires an engineering team which actually cares about all this and consistency of approach across what seems like 6 years? Impressive.

Thank you.
Post reply on HN