Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Details of yesterday's Bunny CDN outage

bunny.net

Post reply on HN
Print view View original

41–50 of 87 posts

Re: Details of yesterday's Bunny CDN outage

#41

Post by ing33k »

dejangp wrote: post #32

Dejan here from bunny.net. I was reading some of the comments, but wasn't sure where to reply, so I guess I'll post some additional details here. I tried to keep the blog post somewhat technical, but not overwhelm non-technical readers. So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructi…

hey dejan, we have been using BunnyCDN for quite some time. Thanks for the detailed writeup.

looks like storage zones are still not fully stable ? after experiencing several issues with storage zones earlier, we migrated to pull a zone. we didn't had any major issues after the migration.

what plans do you have to improve your storage zone ?

Re: Details of yesterday's Bunny CDN outage

#42

Post by rapsey »

altmind wrote: post #29
penguinten wrote: post #26

Earlier quoted context omitted.

https://www.fastly.com/blog/summary-of-june-8-outage

fastly RCA is underwhelming - no info on what was the component, what happened and how the situation was tackled.

I think a public company will never dare go into as much detail as bunny did. Or maybe it is just the size of the organisation that discourages that.

Re: Details of yesterday's Bunny CDN outage

#44

Post by aitchnyu »

manishsharan wrote: post #31
string wrote: post #8

Good and clear explanation. This is a risk you take when you use a CDN, I still think the benefits outweigh the occasional downtime. I'm a big fan of BunnyCDN, they've saved me a lot of money over the past few years. I'm sure I'd be fuming if I worked at some multi-million dollar company but as someone that mainly works for smaller businesses it's not the end of the world, I suspect most of my clients haven't even no…

TIL about BunnyCDN. I had been paying $0.08 per GB on AWS Cloudfront whereas BunnyCDN is only $0.01 per GB. Can you comment on you experience with them ? Are the APIs comprehensive e.g. cache invalidation ? do they support cookie base authorization ? Any support Geo-Fencing?

Can this allow me to route x.mydomain.com (more than one wildcard and top level) to x.a.run.app (Google Cloud Run)? Cloud Run (and the Django app behind it) won't approve domain mapping for Mumbai yet so I am looking for transparent domain rewriting. Cloudflare allows it but its kinda expensive.

https://cloud.google.com/run/docs/locations#domains

Re: Details of yesterday's Bunny CDN outage

#45

Post by bovermyer »

dejangp wrote: post #35
bovermyer wrote: post #28

Earlier quoted context omitted.

That's not the impression I got. Yeah, their takeaway was to stop using BinaryPack, which I disagree with. However, it sounded to me like they very much understood that they made the biggest error in putting all of their eggs in one basket. Your system WILL go down eventually. The question is how will you recover from it?

Right, this was our biggest failure (not the only one of course, but we are here to improve). Relying on our own systems to maintain our own systems. We are dropping BinaryPack mainly because we're a small team, and it wasn't really a big benefit anyway, so spending more time than necessary to try and salvage that makes no sense. This was more of a hot-fix since we don't want the same thing repeating in a week.

That makes sense then with the additional context.

I don't know the details of your operation, but keeping your ability to update your systems separate from your systems is something I'd strongly encourage.

Re: Details of yesterday's Bunny CDN outage

#46

Post by jiofih »

dejangp wrote: post #32

Dejan here from bunny.net. I was reading some of the comments, but wasn't sure where to reply, so I guess I'll post some additional details here. I tried to keep the blog post somewhat technical, but not overwhelm non-technical readers. So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructi…

> Unfortunately this deployment went to a system that we believed was not a critical part of infrastructure

Deploying to a different, lower priority system is not a canary. Do you phase deployments to each system, per host or zone?

Re: Details of yesterday's Bunny CDN outage

#47

Post by gcbirzan »

aitchnyu wrote: post #44
manishsharan wrote: post #31

Earlier quoted context omitted.

TIL about BunnyCDN. I had been paying $0.08 per GB on AWS Cloudfront whereas BunnyCDN is only $0.01 per GB. Can you comment on you experience with them ? Are the APIs comprehensive e.g. cache invalidation ? do they support cookie base authorization ? Any support Geo-Fencing?

Can this allow me to route x.mydomain.com (more than one wildcard and top level) to x.a.run.app (Google Cloud Run)? Cloud Run (and the Django app behind it) won't approve domain mapping for Mumbai yet so I am looking for transparent domain rewriting. Cloudflare allows it but its kinda expensive. https://cloud.google.com/run/docs/locations#domains

As the docs say, you can use a LB with this. It'll be 18 dollars a month, though.

Re: Details of yesterday's Bunny CDN outage

#48

Post by dejangp »

jiofih wrote: post #46
dejangp wrote: post #32

Dejan here from bunny.net. I was reading some of the comments, but wasn't sure where to reply, so I guess I'll post some additional details here. I tried to keep the blog post somewhat technical, but not overwhelm non-technical readers. So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructi…

> Unfortunately this deployment went to a system that we believed was not a critical part of infrastructure Deploying to a different, lower priority system is not a canary. Do you phase deployments to each system, per host or zone?

For critical systems (or let's call them services) such as DNS, CDN, optimizer, storage, we usually deploy either on a server to server basis, regional basis or cluster basis before going live. What I mean here was that this was not really a critical service that nobody thought could actually cause any harm, so we didn't do canary testing there as it would add a very high level of complexity.

Re: Details of yesterday's Bunny CDN outage

#49

Post by dejangp »

ing33k wrote: post #41
dejangp wrote: post #32

Dejan here from bunny.net. I was reading some of the comments, but wasn't sure where to reply, so I guess I'll post some additional details here. I tried to keep the blog post somewhat technical, but not overwhelm non-technical readers. So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructi…

hey dejan, we have been using BunnyCDN for quite some time. Thanks for the detailed writeup. looks like storage zones are still not fully stable ? after experiencing several issues with storage zones earlier, we migrated to pull a zone. we didn't had any major issues after the migration. what plans do you have to improve your storage zone ?

Hey, glad to hear that and sorry again about any issues. If you're experiencing any ongoing problems, please message our support team. I'm not aware of anything actively broken, but if there's a problem I'm sure we'll be able to help.

Re: Details of yesterday's Bunny CDN outage

#50

Post by plett »

rapsey wrote: post #42
altmind wrote: post #29

Earlier quoted context omitted.

fastly RCA is underwhelming - no info on what was the component, what happened and how the situation was tackled.

I think a public company will never dare go into as much detail as bunny did. Or maybe it is just the size of the organisation that discourages that.

Cloudflare's RFO blog posts are incredibly detailed. Each time I read one, I feel reasonably confident that they have learnt from the mistakes that lead to that outage and that it shouldn't happen again.

https://blog.cloudflare.com/tag/outage/

Post reply on HN