Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Details of yesterday's Bunny CDN outage

bunny.net

Post reply on HN
Print view View original

51–60 of 87 posts

Re: Details of yesterday's Bunny CDN outage

#51

Post by dejangp »

ing33k wrote: post #40
path411 wrote: post #12

Sounds like they got really lucky they could get it back up so quickly. They must have some very talented engineers working there. My take aways though were they should have tested the update better. They should have their production environment more segmented with staggered updates so they have much more contained disasters. And they should have had much better catastrophic failure plans in place.

they are very a small team. Their CEO codes !

I do! In fact, I work like 90 hours a week. I decided to go the bootstrap way (looking back not sure if that was the best idea, but we are where we are), but we're growing 3X year over year, so things are picking up. :)

Re: Details of yesterday's Bunny CDN outage

#52

Post by dpcx »

ram_rar wrote: post #17

> On June 22nd at 8:25 AM UTC, we released a new update designed to reduce the download size of the optimization database. Unfortunately, this managed to upload a corrupted file to the Edge Storage. I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together. > Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with…

In the post or comments, they claimed using canary; perhaps their canary simply didn't die in the coalmine?

Re: Details of yesterday's Bunny CDN outage

#53

Post by xwolfi »

YetAnotherNick wrote: post #16

They are making it sound like they did everything right and it was a issue of third party library. If we list all the libraries our code depend on, it will be in 1000s. I can't comprehend how a CDN does not have any canary or staging setup and in a update everything could go haywire in seconds. I think it is standard practice in any decent size company to have staging/canary and rollbacks.

I came to post that, yeah. I work in a sensitive system on which people can lose millions for a few minutes downtime, and we are a bit anal about week long pilots where half the prod is in a permanent canary stage.

But also they used their own infra it feels to setup their stuff and if their infra was dead they couldnt rollback, which sounds like a case where people a bit too optimistic.

We had catastrophes too, notably on poison pills in a record stream we cant alter, but this update cascade crash sounds avoidable.

Always easy to judge anyway, always happens to you eventually :D

Re: Details of yesterday's Bunny CDN outage

#54

Post by foobarbazetc »

I like how something is “auto-healing” when it’s like… has `Restart=on-failure` in systemd.

Anyway, it’s always DNS. Always.

“Unfortunately, that allowed something as simple as a corrupted file to crash down multiple layers of redundancy with no real way of bringing things back up.”

You can spend many, many millions of $ on multi-AZ Kubernetes microservices blah blah blah and it’ll still be taken down by a SPOF, which, 99% of the time, is DNS.

Actual redundancy, as opposed to “redundancy”, is extremely difficult to achieve because the incremental costs of one more 9 are almost exponential.

And then a customer updates their configuration and your entire global service goes down for hours ala Fastly.

Or a single corrupt file crashes your entire service.

Re: Details of yesterday's Bunny CDN outage

#55

Post by string »

manishsharan wrote: post #31
string wrote: post #8

Good and clear explanation. This is a risk you take when you use a CDN, I still think the benefits outweigh the occasional downtime. I'm a big fan of BunnyCDN, they've saved me a lot of money over the past few years. I'm sure I'd be fuming if I worked at some multi-million dollar company but as someone that mainly works for smaller businesses it's not the end of the world, I suspect most of my clients haven't even no…

TIL about BunnyCDN. I had been paying $0.08 per GB on AWS Cloudfront whereas BunnyCDN is only $0.01 per GB. Can you comment on you experience with them ? Are the APIs comprehensive e.g. cache invalidation ? do they support cookie base authorization ? Any support Geo-Fencing?

I noticed another user has already commented, sounds like they've had more experience with the things you're interested in than I have, for FWIW, the APIs have been sufficient for my use cases and you can definitely purge a pullzone's cache with them.

My primary use has been for serving image assets, switched over from Cloudfront and have seen probably a >80% cost reduction, and no noticeable performance reduction, but as I mentioned I'm operating at a scale where milliseconds of difference don't mean much.

Re: Details of yesterday's Bunny CDN outage

#56

Post by methyl »

jgrahamc wrote: post #34
ram_rar wrote: post #17

> On June 22nd at 8:25 AM UTC, we released a new update designed to reduce the download size of the optimization database. Unfortunately, this managed to upload a corrupted file to the Edge Storage. I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together. > Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with…

I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together. Oh man, you stirred up a really old Cloudflare memory. Back when I was working on our DNS infrastructure I wrote up a task that says: "RRDNS has no way of knowing how many lines to expect or whether what it is read is valid. This could create an issue where the LB map data is not available inside RRDNS." At t…

For whom, like myself, don't know the story, here it is: https://www.wired.com/story/lee-holloway-devastating-decline...

I'm deeply moved after reading it. Can't imagine how tragic it must be for people who know Lee.

Re: Details of yesterday's Bunny CDN outage

#57

Post by dylanz »

methyl wrote: post #56
jgrahamc wrote: post #34

Earlier quoted context omitted.

I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together. Oh man, you stirred up a really old Cloudflare memory. Back when I was working on our DNS infrastructure I wrote up a task that says: "RRDNS has no way of knowing how many lines to expect or whether what it is read is valid. This could create an issue where the LB map data is not available inside RRDNS." At t…

For whom, like myself, don't know the story, here it is: https://www.wired.com/story/lee-holloway-devastating-decline... I'm deeply moved after reading it. Can't imagine how tragic it must be for people who know Lee.

That was an incredible story, and I went down a rabbit hole of reading more about that disease. Thank you very much for sharing.

Re: Details of yesterday's Bunny CDN outage

#58

Post by patrickbolle »

Great write-up. I've just switched from Cloudinary to Backblaze B2 + Bunny CDN and I am saving a pretty ridiculous amount of money for hosting thousands of customer images.

Bunny has a great interface and service; I'm really surprised how little people know about it, I think I discovered it on some 'top 10 CDNs list' that I usually ignore, but the pricing was too good to pass up.

The team is really on the ball from what I've seen. Appreciate the descriptive post, folks!

Re: Details of yesterday's Bunny CDN outage

#59

Post by slackerIII »

Oh, this is a great writeup. I co-host a podcast on outages, and over and over we see cases where circular dependencies end up making recovery much harder. Also, not using a staged deployment is a recipe for disaster!

We just wrapped up the first season, but I'm going to put this on the list of episodes for the second season: https://downtimeproject.com.

Re: Details of yesterday's Bunny CDN outage

#60

Post by sudhirj »

manishsharan wrote: post #31
string wrote: post #8

Good and clear explanation. This is a risk you take when you use a CDN, I still think the benefits outweigh the occasional downtime. I'm a big fan of BunnyCDN, they've saved me a lot of money over the past few years. I'm sure I'd be fuming if I worked at some multi-million dollar company but as someone that mainly works for smaller businesses it's not the end of the world, I suspect most of my clients haven't even no…

TIL about BunnyCDN. I had been paying $0.08 per GB on AWS Cloudfront whereas BunnyCDN is only $0.01 per GB. Can you comment on you experience with them ? Are the APIs comprehensive e.g. cache invalidation ? do they support cookie base authorization ? Any support Geo-Fencing?

Think the answer is yes to all three questions, depending on the specifics. They've got a nice setup, about ~40+ edge locations compared to Cloudfront's ~200+, but the advantage is they're massively cheaper for very small increase in latency. They also have the ~5 region high-volume tier which is something like another order of magnitude cheaper.

The feature set is pretty full, no edge functions, but there is a rule engine you can run on the edge. Fast config updates, nice console and works well enough for most of my projects.

They also have a nice integrated storage solution that's way easier to configure than S3 + Cloudfront, and lots of origin shielding options.

Post reply on HN