TLDR: they use dynamic DNS routing and have fail over regions.
Can't believe they didn't just mention that in the beginning. There was a lot of show off imo. None of those would have save them on 20th oct if they didn't have the dynamic dns routing
How when AWS was down, we were not
71–79 of 79 posts
Re: How when AWS was down, we were not
#72It’s fascinating to me people think their services are so important they can’t survive any downtime. Can we all admit that, while annoying, nothing really bad happened even when us-east-1 was down for almost half a working day?
That's a bit of a naive perspective. There are plenty of situations and industries where access being down has an impact far beyond inconvenience. For example, access to medical files for treatment, allergies and surgery. Or access to financial services.
Re: How when AWS was down, we were not
#73TLDR: they use dynamic DNS routing and have fail over regions.
Can't believe they didn't just mention that in the beginning. There was a lot of show off imo. None of those would have save them on 20th oct if they didn't have the dynamic dns routing
Actually this isn't the only thing that exists. As I pointed out that only exists for resources that are duplicated between regions. There's also the critical fallbacks that exist at the service level to decide which resources to consume. Our usage there is possible both with CloudFront Origin Groups as well as replicated data in our database to multiple regions.
Re: How when AWS was down, we were not
#74I'm interested in how they measure that downtime. If you're down for 200 milliseconds, does that accumulate. How do you even measure that you're down for 200ms. (For what it's worth, for some of my services, 200ms is certainly an impact, not as bad as 2 seconds out outage but still noticable and reportable)
I think a lot of web services talk about reliability in terms of uptime (e.g. down for less than 5 minutes a year) but in reality operate on failure ratios (less than 0.001% of request to our service fail).
100%
Re: How when AWS was down, we were not
#75Earlier quoted context omitted.
As other posters have commented, an external auth service is a very special thing indeed. In modern and/or zero-trust systems if auth doesn't work, then effectively nothing works. My rule of thumb from the past experiences is that if you demand a 99.9% uptime for your own systems and you have an in-house auth, then that auth system must have 99.99% reliability. If you are serving auth for OTHERS, then you have a syst…
The sad truth of the world is that in many cases latency isn't the most critical aspect for tracking. We absolutely do track it because we have the expectation that authentication requests complete. But there are many moving parts to this that make reliable tracking not entirely feasible: * end location of user * end location of customer service * third party login components (login with google, et al) * corporate id…
While I agree with parts of the above, there are bits that I disagree with. It's true that you cannot control the network conditions for third parties, and therefore can never be in a position where you would guarantee an SLA for round-trip experience. But I object the notion that tracking end-to-end latency is useless. After all, the three Nielsen usability thresholds are all about latency(!)
Funnily enough, looking through your itemisation I spot two groups that would each benefit from their own kinds of latency monitoring. End location and internet connectivity of the client go into the first. Third-party providers go into the second.
For the first, you'd need to have your own probes reporting from the most actively used networks and locations around the world - that would give you a view into the round-trip latency per major network path. For the second, you'd want to track the time spent between the steps that you control - which in turn would give you a good view into the latency-inducing behaviour of the different third-party providers. Neither are SLA material but they certainly would be useful during enterprise contract negotiations. (Shooting impossible demands down by showing hard data tends to fend off even the most obstinate objections.)
User-agent and bespoke integrations/workflows are entirely out of your hands, and I agree it's useless to try to measure latency for them specifically.
Disclaimer: I have worked with systems where the internal authX roundtrip has to complete within 1ms, and the corresponding client-facing side has to complete its response within 3ms.
Re: How when AWS was down, we were not
#76Earlier quoted context omitted.
> the section about retries doesn't mention correlations. [...] By treating P_{3rdParty}(Failure) as fixed, they're assuming a model in which each each try is completely independent: all the failures are due to background noise. But that's totally wrong, as shown by the existence of big outages like the one they're describing Yes, that jumped out at me as well. A slightly more sophisticated model could be to assume t…
This is absolutely true, but the end result is the same. The assumption is "We can fix a third party component behaving temporarily incorrectly, and therefore we can do something about it". If the third party component never behaves correctly, then nothing we can do to fix it. Correlations don't have to be talked about, because they don't increase the likelihood for success, but rather the likihood of failure, meanin…
I think the reasoning matters as much as the answer, and you had to make at least a couple strange turns to get the "right answer" that retries don't solve the problem:
* the 3rd-party component offering only 90% success—I've never actually seen a system that bad. 99.9% success SLA is kind of the minimum, and in practice any system that has acceptable mean and/or 99%/99.9% latency for a critical auth path also has >=99.99% success in good conditions (even if they don't promise refunds based on that).
* the whole "really reliable retry handler" thing—as mentioned in my first comment, I don't understand what you were getting at here.
I would go a whole other way with this section—more realistic, much shorter. Let's say you want to offer 99.999% success within 1 second, and the third-party component offers 99.9% success per try. Then two tries gives you 99.9999% success if the failures are all uncorrelated but retries do not help at all when the third-party system is down for minutes or hours at a time. [1] Thus, you need to involve an alternative that is believed to be independent of the faulty system—and the primary tool AWS gives you for that is regional independence. This sets up the talk about regional failover much more quickly and with less head-scratching. I probably would have made it through the whole article yesterday even in my feverish state.
[1] unless this request can be done asynchronously, arbitrarily later, in which case the whole chain of thought afterward goes a different way.
Re: How when AWS was down, we were not
#77Earlier quoted context omitted.
As other posters have commented, an external auth service is a very special thing indeed. In modern and/or zero-trust systems if auth doesn't work, then effectively nothing works. My rule of thumb from the past experiences is that if you demand a 99.9% uptime for your own systems and you have an in-house auth, then that auth system must have 99.99% reliability. If you are serving auth for OTHERS, then you have a syst…
The sad truth of the world is that in many cases latency isn't the most critical aspect for tracking. We absolutely do track it because we have the expectation that authentication requests complete. But there are many moving parts to this that make reliable tracking not entirely feasible: * end location of user * end location of customer service * third party login components (login with google, et al) * corporate id…
I imagine you exclude failures of customer systems from your reliability measurements—for example, if you send a backend request to or redirect the user's browser to the customer's corporate identity provider and that persistently fails, you don't call it your own outage.
The same can apply to latency. What is the latency of requests to your system—including dependencies you choose, excluding dependencies the customer chooses. The network leg from the customer or user to your system is a bit of a gray area. The simplest thing to do is measure each request's latency from the point of view of your backend rather than the initiator. This is probably good enough, although in theory it lets you off the hook a bit too easily—to some extent you can choose whether you run near the initiator or not and how many round trips are required, and servers can underestimate their own latency or entirely miss requests during failures. But it's not fair to fail your SLA because of end-user bufferbloat or bad wifi or a crappy ancient Chromebook with too many open tabs or customer webapp server's GC spiral or whatever. Basically impossible to make any 99.999% promises when those things are in play.
My preferred form of SLO is: x% of requests given y ms succeed within y ms, measured by my server. ("given" meaning "does not have an upfront timeout shorter than" and "isn't aborted by the client before".) I might offer a few such guarantees for a particular request type, e.g.:
* 50% of lookups given 1 ms succeed within 1 ms.
* 99% of lookups given 10 ms succeed within 10 ms.
* 99.999% of lookups given 500 ms succeed within 500 ms.
I like to also have client-side and whole-flow measurements but I'm much more cautious about promising anything about them.
Re: How when AWS was down, we were not
#78Earlier quoted context omitted.
This is absolutely true, but the end result is the same. The assumption is "We can fix a third party component behaving temporarily incorrectly, and therefore we can do something about it". If the third party component never behaves correctly, then nothing we can do to fix it. Correlations don't have to be talked about, because they don't increase the likelihood for success, but rather the likihood of failure, meanin…
I think the reasoning matters as much as the answer, and you had to make at least a couple strange turns to get the "right answer" that retries don't solve the problem: * the 3rd-party component offering only 90% success—I've never actually seen a system that bad. 99.9% success SLA is kind of the minimum, and in practice any system that has acceptable mean and/or 99%/99.9% latency for a critical auth path also has >=…
Hmm, I never considered potentially using an SLA on latency as a potential way to justify the argument. If I pull this content into a future article or talk, I will definitely consider reframing it for easier understanding.
Re: How when AWS was down, we were not
#79> We test before deployment. There is no better time to test. Love the deadpan delivery.
On the other hand, saying “Untested code is never released” is a pretty bold statement, even if I understand the good intent
The point here is that, it's important to clarify what you mean by "untested code". Some companies release untested code all the time to production by hide their usage behind feature flags. They admit that the code has not been tested.
Code that we release behind feature flags has been tested, the only reason something goes out on a flag is when we don't want to release it to everyone yet, for product reasons, not technical ones.
Again going back to "untested", when code gets merged, no one ever says "well taht was untested". But what does tested mean? Of course it means that everyone who is accountable for the code believes it was tested. It's subjective, so there can be no other answer.