Live data from Hacker News

Making S3 More Resilient Using Lambda Edge

contentful.com

11–20 of 25 posts

Re: Making S3 More Resilient Using Lambda Edge

#11
post #10
post #6

Earlier quoted context omitted.

I think it's still a reduction in risk overall. In the old model, they were vulnerable to S3 failing in one region, a thing that's happened many times. Now they've mitigated the S3-failure-in-one-region issue, at least mostly (though as you point out, how they do so is unknown), and in exchange they've picked up a dependency on Lambda@Edge. But Lambda@Edge, like CloudFront, is a global service distributed across many…

The most famous s3 outage has been operator error from a well-meaning privileged user. The fact that it hasn’t happened for Lambda is just betting on luck. Shit happens, we can’t go designing ever more complicated solutions. May be our services should have some graceful degradation when shit happens instead of trying to create a big-bang and spawn an alternate universe.

I mean, I agree in spirit, but everyone has a different sense of cost/complexity vs. return.

I don't advocate for ever-more-complicated solutions as a rule. e.g. I think multi-cloud setups are probably way more trouble than they're worth for most companies.

I certainly agree that graceful degradation where possible and not too expensive is ideal. For example, if S3 is having problems in one region, being able to fall back (gracefully degrade) into read-only mode might be a nice thing to have.

(In this particular case having a secondary region also probably helps with disaster recovery, which is pretty much mandatory in B2B, for better or worse.)

Re: Making S3 More Resilient Using Lambda Edge

#12
post #3

Before they'd be affected by Route 53 outages, Cloudfront outages, and S3 outages. Now they can add Lambda outages to that list too. It's also unclear how this actually solves the problem. Now if S3 in _either_ region is unavailable they'll start to fail 50% of uncached requests. I'm guessing they're using Route 53 health checks with some cloudwatch alarm to cut over to one region when they think the other is unhealt…

> Now they can add Lambda outages to that list too.

Specifically Lambda@Edge though, which changes the math a little.

That said, currently the better solution is CloudFront Origin Groups.

Re: Making S3 More Resilient Using Lambda Edge

#15
post #5

Earlier quoted context omitted.

That's a relatively new feature (from November 2018... wow, has it been a year already?). My guess is that they implemented this stuff before that existed or maybe near to its release. I have not used origin failover either, though I'm pretty sure you're right that this is its exact use-case.

You are correct, this has been implemented before Origin Groups have been released and they might be a viable alternative, but we haven’t tested them yet. Source: I work at Contentful.

My concern is you (contentful) are leading folks down a bad path. Adding complexity, code, cost and a larger surface area of services that need to be up.

Re: Making S3 More Resilient Using Lambda Edge

#16

Google Cloud Storage has multi-region storage classes. Does S3 not have an equivalent of this?

They have cross-region replication.

I think it's worthwhile to look more into GCP's multi-region bucket implementation and how nice it is. It pretty much removes the need to explicitly set up cross-region replication of objects and offers a single endpoint from which to serve objects in the nearest/most available region.

Re: Making S3 More Resilient Using Lambda Edge

#20
post #19

Although it may make sense to this company in _majority_ of companies this would be over-engineering. S3 availability is some of the best in the business. If S3 is down, a good chunk of the internet is down with it.

I disagree.

A single region can go down (and has in the past), no matter how reliable S3 as a whole is. If your business wants to avoid downtime, this is a simple solution to further reduce risks that cause downtime.

Just because other sites might go down, doesn't mean you have to accept it for your own.

Post reply on HN