Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cross-Account and Cross-Region Backups with AWS Backup (and Friends)

tylerrussell.dev

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Cross-Account and Cross-Region Backups with AWS Backup (and Friends)

#11

Post by everfrustrated »

deathanatos wrote: post #10
deepsun wrote: post #5

Earlier quoted context omitted.

The most common cause of the outages right now is configuration errors. Even when procedurally they must be limited to AZ only, there is always some region-shared infrastructure that can bring down the whole region altogether.

"Configuration errors" — I'm going to include "bugs" in that —, IME, tend to be global outages more often than regional. If I recount the outages >AZ that I've seen, I think the most recent ones were: GCP, IAM (global; just like a week and a half ago!) GCP, VMs etc. (regional!¹) Azure, application GW (global) Cloudflare (global) Azure, IAM (global) Azure, IAM (global) You can tell IAM is a point of weakness. (As it k…

Notable you don't have AWS on that list.

AWS's definitions for AZ & Regions are by far the strongest in the industry.

GCP has AZ in the same physical complex. Azure Regions would be AZ's under AWS's definition.

Re: Cross-Account and Cross-Region Backups with AWS Backup (and Friends)

#12

Post by timewizard »

everfrustrated wrote: post #11
deathanatos wrote: post #10

Earlier quoted context omitted.

"Configuration errors" — I'm going to include "bugs" in that —, IME, tend to be global outages more often than regional. If I recount the outages >AZ that I've seen, I think the most recent ones were: GCP, IAM (global; just like a week and a half ago!) GCP, VMs etc. (regional!¹) Azure, application GW (global) Cloudflare (global) Azure, IAM (global) Azure, IAM (global) You can tell IAM is a point of weakness. (As it k…

Notable you don't have AWS on that list. AWS's definitions for AZ & Regions are by far the strongest in the industry. GCP has AZ in the same physical complex. Azure Regions would be AZ's under AWS's definition.

AWS had a console login issue a while back due to the default region being us-east-1. There are a handful of other services that are exclusively available in that region as well.

Re: Cross-Account and Cross-Region Backups with AWS Backup (and Friends)

#15

Post by wiether »

nodesocket wrote: post #14

What’s the benefits of using AWS Backup? If your infrastructure is already defined using Terraform then RDS, EBS snapshots, ElastiCache, S3 already have backup configuration options.

As the article shows how to do it, with AWS Backup you can do things like cross-account and cross-region backups.

Moreover, AWS Backup is the _Terraform_ of backup in AWS. You can control all your backups through a single interface, with various policies (scheduling, retention, access...)

For instance, by default, you are limited to 100 Manual RDS Snapshots per account. With AWS Backup, you can do what you want. You can define dozens of different rules for the same services/resources.

So you can let teams manage their resources as they want, and have a backup team manage backuping everything from AWS Backup without having to interact with the services/resources themselves.

Re: Cross-Account and Cross-Region Backups with AWS Backup (and Friends)

#16

Post by time0ut »

Nice write up. I did something similar at a company recently. The ransomware use case was the primary driver. AWS Backup felt kind of half baked. It also took a lot of work to ensure we could bring the apps up in the recovery account smoothly. Trying to retrofit this into existing stacks was kind of a pain.

There is a YC company called Arpio [0] that does this sort of thing as a service. It can replicate a ton of stuff beyond what Backup does (it also uses Backup for certain things from what I remember). It works as advertised and for most companies is probably worth it vs doing this yourself. I am not affiliated, just worked with it at a customer.

[0] https://arpio.io/

Re: Cross-Account and Cross-Region Backups with AWS Backup (and Friends)

#17

Post by tgmatt »

Be aware that AWS Backup is _very_ expensive. We recently stopped using it and switched to AWS DataSync, which is an order of magnitude cheaper. If you want to go even cheaper, S3 replication (not delete markers) will do it for even less.

Backup to S3, use the above to copy it elsewhere.

Re: Cross-Account and Cross-Region Backups with AWS Backup (and Friends)

#18

Post by deepsun »

deathanatos wrote: post #10
deepsun wrote: post #5

Earlier quoted context omitted.

The most common cause of the outages right now is configuration errors. Even when procedurally they must be limited to AZ only, there is always some region-shared infrastructure that can bring down the whole region altogether.

"Configuration errors" — I'm going to include "bugs" in that —, IME, tend to be global outages more often than regional. If I recount the outages >AZ that I've seen, I think the most recent ones were: GCP, IAM (global; just like a week and a half ago!) GCP, VMs etc. (regional!¹) Azure, application GW (global) Cloudflare (global) Azure, IAM (global) Azure, IAM (global) You can tell IAM is a point of weakness. (As it k…

I mostly remember AWS S3 outages, usually limited to a region, but the one in 2017 was supposed to be a regional update (US-EAST-1 region), brought down like a half of AWS, because they depended on S3 in US-EAST-1 [1]

Note that even the intended configuration change was designed to be Regional, not just limited to one AZ.

https://aws.amazon.com/message/41926/

Re: Cross-Account and Cross-Region Backups with AWS Backup (and Friends)

#19

Post by deathanatos »

everfrustrated wrote: post #11
deathanatos wrote: post #10

Earlier quoted context omitted.

"Configuration errors" — I'm going to include "bugs" in that —, IME, tend to be global outages more often than regional. If I recount the outages >AZ that I've seen, I think the most recent ones were: GCP, IAM (global; just like a week and a half ago!) GCP, VMs etc. (regional!¹) Azure, application GW (global) Cloudflare (global) Azure, IAM (global) Azure, IAM (global) You can tell IAM is a point of weakness. (As it k…

Notable you don't have AWS on that list. AWS's definitions for AZ & Regions are by far the strongest in the industry. GCP has AZ in the same physical complex. Azure Regions would be AZ's under AWS's definition.

I haven't worked with them in quite some time. (That's changing, so uh … looking forward to my next AWS outage?) This was more to show regional vs. global than any specific cloud provider. AWS is skating by here on account of not being sampled¹.

If I go waaaaay back (like mid 2010s), we did have an S3 outage. It was regional, even!

> GCP has AZ in the same physical complex.

I can't say if that's correct or not; GCP says,

> Zones should be considered a single failure domain within a region. To deploy fault-tolerant applications with high availability and help protect against unexpected failures, deploy your applications across multiple zones in a region.

That's an AZ, to me. (Or, alternatively & synonymously, a failure domain.)

¹IME over my career, though, AWS is fairly stable. GCP is too. AWS has its foibles, though. When last I worked with RDS (circa 2019), there were bugs.

Post reply on HN