Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How when AWS was down, we were not

authress.io

Post reply on HN
Print view View original

31–40 of 79 posts

Re: How when AWS was down, we were not

#31

Post by wparad »

indigodaddy wrote: post #27
wparad wrote: post #16

Earlier quoted context omitted.

If the check can't be done, then everything stays stable, so I'm guessing the question is, "What happens if Route 53 does the check and incorrectly reports the result?" In that case, no matter what we are using there is going to be a critical issue. I think the best I could suggest at that point would be to have records in your zone that round robin different cloud providers, but that comes with its own challenges. I…

Have you considered the scenario of "everything is so dead in aws", that the check doesn't happen, plus the backends are dead too (this is assuming the backend services live in aws as well) ? But I'd guess in that case you'd know quickly enough from supplementary alerting (you guys don't seem the type to not have some sort of awesome monitoring in place) and you have a different/worse DR problem on your hands. As far…

I think there are two worlds of thought to the "AWS is totally dead everywhere". And that's: * It is never going to happen due to the way AWS is designed (or at least told to us, which explains why it is so hard to execute actions across regions.) * It will happen but then everything else is going to be dead, so what's the point?

One problem we've run into, which is the "DNS is single point of failure" is that there isn't a clear best strategy to deal with "failover to a different cloud at the DNS routing level."

I'm not the foremost expert when it comes to ASNs and BGPs, but from my understanding that would require some multi-cloud collaboration to get multiple CDNs to still resolve, something that feels like it would require both multiple levels of physical infrastructure as well as significant cost to actually implement correctly compared to the ROI for our customers.

There's a corollary here for me, which is, still as simple as possible to achieve the result. Maybe there is a multi-cloud strategy, but the strategies I've seen still rely on having the DNS zone in one provider that fail-overs or round-robins specific infra in specific locations.

Third party health checks have less of a problem of "tainting" and more just cause further complications, as you add in complexity to resolving your real state, the harder it is to get it right.

For instance, one thing we keep going back and forth on is "After the incident is over, is there a way for us to stay failed-over and not automatically fail back".

And the answer for us so far is "not really". There are a lot of bad options, which all could have catastrophic impacts if we don't get it exactly correct, and haven't come with significant benefits, yet. But I like to think I have an open mind here.

Re: How when AWS was down, we were not

#32

Post by wparad »

rdoherty wrote: post #24

This is probably one of the best summarizations of the past 10 years of my career in SRE. Once your systems get complex enough, something is always broken and you have to prepare for that. Detection & response become just as critical as pre-deploy testing. I do worry about all the automation being another failure point, along with the IaC stuff. That is all software too! How do you update that safely? It's turtles al…

Thank you!

One of the question I frequently get is "do you automatically rollback". And I have hide in the corner and say "not really". Often, if you knew a rollback would work, you probably could also have known to not roll out in the first place. I've seen a lot of failures that only got worse when automation attempted to turn the thing on and off again.

Luckily from an automation roll-out standpoint, it's not that much harder to test in isolation. The harder parts to validate are things like "Does a Route 53 Failover Record really work in practice at the moment we actually need it to work?"

Usually the answer is yes, but then there's always the "but it too could be broken", and as you said, it's turtles all the way down.

The nice part is realistically, the automation for dealing with rollout and IaC is small and simple. We've split up our infrastructure to go with individual services, so each piece of infra is also straight forward.

In practice, our infra is less DRY and more repeated, which has the benefit of avoiding complexity that often comes from attempting to reduce code duplication. The ancillary benefit is that, simple stuff changes less frequently. Less frequent changes because less opportunity for issues.

Not-surprisingly, most incidents comes from changes humans make. Where the second most amount of incidents come from assumptions humans make about how a system operates in edge conditions. If you know these two things to be 100% true, you spend more time designing simple systems and attempting to avoid making changes as much as possible, unless it is absolutely required.

Re: How when AWS was down, we were not

#33

Post by xyzzy123 »

evanmoran wrote: post #28
rdoherty wrote: post #24

This is probably one of the best summarizations of the past 10 years of my career in SRE. Once your systems get complex enough, something is always broken and you have to prepare for that. Detection & response become just as critical as pre-deploy testing. I do worry about all the automation being another failure point, along with the IaC stuff. That is all software too! How do you update that safely? It's turtles al…

Iac is definitely a failure point, but the manual alternative is much worse! I’ve had a lot of benefit from using pulumi, simply because the code can be more compact than the terraform hcl was. For example, for the fall over regions (from the article) you could make a pulumi function that parameterizes only the n things that are different per fall over env and guarantee / verify the scripts are nearly identical. Of c…

I actually like terraform for its LACK of power (tho yeah these days when I have a choice I use a lot of small states and orchestrate with tg).

Pulumi or CDK are for sure more powerful (and great tools) but when I need to reach for them I also worry that the infra might be getting too complex.

Re: How when AWS was down, we were not

#34

Post by wparad »

evanmoran wrote: post #28
rdoherty wrote: post #24

This is probably one of the best summarizations of the past 10 years of my career in SRE. Once your systems get complex enough, something is always broken and you have to prepare for that. Detection & response become just as critical as pre-deploy testing. I do worry about all the automation being another failure point, along with the IaC stuff. That is all software too! How do you update that safely? It's turtles al…

Iac is definitely a failure point, but the manual alternative is much worse! I’ve had a lot of benefit from using pulumi, simply because the code can be more compact than the terraform hcl was. For example, for the fall over regions (from the article) you could make a pulumi function that parameterizes only the n things that are different per fall over env and guarantee / verify the scripts are nearly identical. Of c…

I think some people are going to scream when I say this, but we're using mostly CloudFormation templates.

We don't use the CDK because it introduces complexity into the system.

However to make CloudFormation usable, it is written in typescript, and generates the templates on the fly. I know that sounds like the CDK, but given the size of our stacks, adding an additional technology in, doesn't make things simpler, and there is a lot of waste that can be removed, by using a software language rather than using json/yaml.

There are cases we have some OpenTofu, but for infrastructure resources that customer specific, we have deployments that are run in typescript using the AWS SDK for javascript.

It would be nice if we could make a single change and have it roll-out everywhere. But the reality is that there are many more states in play then what is represented by a single state file. Especially when it comes to interactions between—our infra, our customer's configuration, and the history of requests to change the configuration, as well as resources with mutable states.

One example of that is AWS certificates. They expire. We need them expiring. But expiring certs don't magically update state files or stacks. It's really bad to make assumptions about a customer's environment based on what we thought we knew the last time a change was rolled out.

Re: How when AWS was down, we were not

#35

Post by wparad »

xyzzy123 wrote: post #33
evanmoran wrote: post #28

Earlier quoted context omitted.

Iac is definitely a failure point, but the manual alternative is much worse! I’ve had a lot of benefit from using pulumi, simply because the code can be more compact than the terraform hcl was. For example, for the fall over regions (from the article) you could make a pulumi function that parameterizes only the n things that are different per fall over env and guarantee / verify the scripts are nearly identical. Of c…

I actually like terraform for its LACK of power (tho yeah these days when I have a choice I use a lot of small states and orchestrate with tg). Pulumi or CDK are for sure more powerful (and great tools) but when I need to reach for them I also worry that the infra might be getting too complex.

Agreed, it is much too easy to fall into bad habits. The whole goal of OpenTofu is declarative infrastructure. With CDK and pulumi, it's very easy to end up in a place where you lose that.

But if you need to do something in a particular way, the tools should never be an obstacle.

Re: How when AWS was down, we were not

#36

Post by wparad »

hartator wrote: post #26

Interesting how engineers like to nerd out about SLAs, but never claim or issue credits when something does occur.

In the last decade, there has been at least one time where we did issue credits to our customers when there was a problem. Issues credits back to our customers is a small compensation for any issue we're responsible for, and doing so is part of our Terms of Service.

Re: How when AWS was down, we were not

#37

Post by dang »

wparad wrote: post #20
dang wrote: post #3

Earlier quoted context omitted.

Re-how'd, plus I've resisted the temptation to insert a comma that feels missing to me.

I spent a long time, trying to figure out, what the title of the article, should be. I'm terrible at SEO and generating click-bait titles, it is unfortunately, what, it, is.

You did fine! The title is clear. I was just being playful.

Re: How when AWS was down, we were not

#38

Post by yearolinuxdsktp »

xyzzy123 wrote: post #33
evanmoran wrote: post #28

Earlier quoted context omitted.

Iac is definitely a failure point, but the manual alternative is much worse! I’ve had a lot of benefit from using pulumi, simply because the code can be more compact than the terraform hcl was. For example, for the fall over regions (from the article) you could make a pulumi function that parameterizes only the n things that are different per fall over env and guarantee / verify the scripts are nearly identical. Of c…

I actually like terraform for its LACK of power (tho yeah these days when I have a choice I use a lot of small states and orchestrate with tg). Pulumi or CDK are for sure more powerful (and great tools) but when I need to reach for them I also worry that the infra might be getting too complex.

IMO Pulumi and CDK are an opportunity to simplify your infra by capturing what you’re working with using higher-level abstractions and by allowing you to refactor and extract reusable pieces at any level. You can drive infra definitions easily from typed data structures, you can add conditionals using natural language syntax, and stop trying to program in a configuration language (Terraform HCL with surprises like non-short-circuited AND evaluation).

You still end up having IaaC. You can still have a declarative infrastructure.

Re: How when AWS was down, we were not

#39

Post by smadge »

iso1631 wrote: post #14

I'm interested in how they measure that downtime. If you're down for 200 milliseconds, does that accumulate. How do you even measure that you're down for 200ms. (For what it's worth, for some of my services, 200ms is certainly an impact, not as bad as 2 seconds out outage but still noticable and reportable)

I think a lot of web services talk about reliability in terms of uptime (e.g. down for less than 5 minutes a year) but in reality operate on failure ratios (less than 0.001% of request to our service fail).

Re: How when AWS was down, we were not

#40

Post by toast0 »

wparad wrote: post #31
indigodaddy wrote: post #27

Earlier quoted context omitted.

Have you considered the scenario of "everything is so dead in aws", that the check doesn't happen, plus the backends are dead too (this is assuming the backend services live in aws as well) ? But I'd guess in that case you'd know quickly enough from supplementary alerting (you guys don't seem the type to not have some sort of awesome monitoring in place) and you have a different/worse DR problem on your hands. As far…

I think there are two worlds of thought to the "AWS is totally dead everywhere". And that's: * It is never going to happen due to the way AWS is designed (or at least told to us, which explains why it is so hard to execute actions across regions.) * It will happen but then everything else is going to be dead, so what's the point? One problem we've run into, which is the "DNS is single point of failure" is that there…

It's painful, but you can split your DNS across multiple providers. It's not usually done other than during migrations, but if you put two NS names from providerA and two from providerB, you'll get a mix of resolution (most high profile domains have 4 NS names; sometimes based on research/testing, sometimes based on cargo culting; I assume you want to fit in... but amazon.com has 8, and the DNS root and some high profile tlds have 13, so you do you :)). If either provider fails and stops responding, most resolvers will use the other provider. If one provider fails and returns bad data (including errors) or even can no longer be updated [1], the redundancy doesn't really help --- you probably went from a full outage that's easy to diagnose to a partial outage that's much harder to diagnose; and if both providers are equally reliable, you increased your chances of having an outage.

[1] But, it's DNS; the expectation is that some resolvers, hopefully very few of them, will cache data as if your TTL value was measured in days. IMHO, If you want to move all your traffic in a defined timeframe, DNS is not sufficient.

Post reply on HN