Making S3 More Resilient Using Lambda Edge
contentful.com
Making S3 More Resilient Using Lambda Edge
1–10 of 25 posts
Re: Making S3 More Resilient Using Lambda Edge
#2If the "Cross-region replication" line in the picture is talking about the native S3 cross-region replication (as I assume it is), beware the replication latency in this setup. AWS recently released "replication with an SLA" for S3 [0], but at "99.99% of the objects will be replicated within 15 minutes", it's not a good enough SLA to rely on in setups like this.
Presumably Part 2 of this post will address this limitation, or maybe their product isn't affected by it. (I've never looked into Contentful, though maybe I will now -- blog post purpose achieved?)
I'm also not sure if "active-active" is the best name for this setup, since objects can't be written to the 2nd bucket (replication only goes one direction). Generally I associate "active-active" with "writes can happen anywhere", though maybe I'm wrong?
[0] https://aws.amazon.com/blogs/aws/s3-replication-update-repli...
Re: Making S3 More Resilient Using Lambda Edge
#3Before they'd be affected by Route 53 outages, Cloudfront outages, and S3 outages. Now they can add Lambda outages to that list too.
It's also unclear how this actually solves the problem. Now if S3 in _either_ region is unavailable they'll start to fail 50% of uncached requests. I'm guessing they're using Route 53 health checks with some cloudwatch alarm to cut over to one region when they think the other is unhealthy. Presumably this is covered in the unavailable part 2.
I'm mildly skeptical that this is worth the increased risks plus the increased cost from running Lambda@Edge on cache misses.
Re: Making S3 More Resilient Using Lambda Edge
#4Confused - why not use CloudFront Origin Groups? https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
Full disclosure, I've never used, but pretty sure this feature was created for the scenario you are trying to solve.
Re: Making S3 More Resilient Using Lambda Edge
#5Confused - why not use CloudFront Origin Groups? https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope... Full disclosure, I've never used, but pretty sure this feature was created for the scenario you are trying to solve.
That's a relatively new feature (from November 2018... wow, has it been a year already?). My guess is that they implemented this stuff before that existed or maybe near to its release.
I have not used origin failover either, though I'm pretty sure you're right that this is its exact use-case.
Re: Making S3 More Resilient Using Lambda Edge
#6Before they'd be affected by Route 53 outages, Cloudfront outages, and S3 outages. Now they can add Lambda outages to that list too. It's also unclear how this actually solves the problem. Now if S3 in _either_ region is unavailable they'll start to fail 50% of uncached requests. I'm guessing they're using Route 53 health checks with some cloudwatch alarm to cut over to one region when they think the other is unhealt…
I think it's still a reduction in risk overall. In the old model, they were vulnerable to S3 failing in one region, a thing that's happened many times. Now they've mitigated the S3-failure-in-one-region issue, at least mostly (though as you point out, how they do so is unknown), and in exchange they've picked up a dependency on Lambda@Edge. But Lambda@Edge, like CloudFront, is a global service distributed across many regions, and to my knowledge AWS has never had a global Lambda@Edge outage.
It's not impossible, of course. Some kind of control plane error could probably knock the whole global service offline. But I'd rather bet on a multi-region service than have all my eggs in one regional basket.
Re: Making S3 More Resilient Using Lambda Edge
#7Before they'd be affected by Route 53 outages, Cloudfront outages, and S3 outages. Now they can add Lambda outages to that list too. It's also unclear how this actually solves the problem. Now if S3 in _either_ region is unavailable they'll start to fail 50% of uncached requests. I'm guessing they're using Route 53 health checks with some cloudwatch alarm to cut over to one region when they think the other is unhealt…
lambda edge is do not depend on lambda, ran by CloudFront.
If one region is down, they will get 50% failure but retry will success.
Re: Making S3 More Resilient Using Lambda Edge
#8Confused - why not use CloudFront Origin Groups? https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope... Full disclosure, I've never used, but pretty sure this feature was created for the scenario you are trying to solve.
That's a relatively new feature (from November 2018... wow, has it been a year already?). My guess is that they implemented this stuff before that existed or maybe near to its release. I have not used origin failover either, though I'm pretty sure you're right that this is its exact use-case.
You are correct, this has been implemented before Origin Groups have been released and they might be a viable alternative, but we haven’t tested them yet.
Source: I work at Contentful.
Re: Making S3 More Resilient Using Lambda Edge
#9Google Cloud Storage has multi-region storage classes. Does S3 not have an equivalent of this?
Re: Making S3 More Resilient Using Lambda Edge
#10Before they'd be affected by Route 53 outages, Cloudfront outages, and S3 outages. Now they can add Lambda outages to that list too. It's also unclear how this actually solves the problem. Now if S3 in _either_ region is unavailable they'll start to fail 50% of uncached requests. I'm guessing they're using Route 53 health checks with some cloudwatch alarm to cut over to one region when they think the other is unhealt…
I think it's still a reduction in risk overall. In the old model, they were vulnerable to S3 failing in one region, a thing that's happened many times. Now they've mitigated the S3-failure-in-one-region issue, at least mostly (though as you point out, how they do so is unknown), and in exchange they've picked up a dependency on Lambda@Edge. But Lambda@Edge, like CloudFront, is a global service distributed across many…
The most famous s3 outage has been operator error from a well-meaning privileged user. The fact that it hasn’t happened for Lambda is just betting on luck. Shit happens, we can’t go designing ever more complicated solutions. May be our services should have some graceful degradation when shit happens instead of trying to create a big-bang and spawn an alternate universe.