Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Details of yesterday's Bunny CDN outage

bunny.net

Post reply on HN
Print view View original

11–20 of 87 posts

Re: Details of yesterday's Bunny CDN outage

#11

Post by dindresto »

busymom0 wrote: post #10

One of the comments on the post is: > One thing you could do in future is to url redirect any BunnyCDN url back to the clients original url, in essence disabling the CDN and getting your clients own hosts do what they were doing before they connected to BunnyCDN, yes it means our sites won't be as fast but its better than not loading up the files at all. I wonder if that is possible in technical terms? Isn't this a h…

Also, how would this work if their whole infrastructure is down? The same problem that prevented them from fixing the network would also have prevented them from adding such a redirect.

Re: Details of yesterday's Bunny CDN outage

#12

Post by path411 »

Sounds like they got really lucky they could get it back up so quickly. They must have some very talented engineers working there.

My take aways though were they should have tested the update better. They should have their production environment more segmented with staggered updates so they have much more contained disasters. And they should have had much better catastrophic failure plans in place.

Re: Details of yesterday's Bunny CDN outage

#13

Post by christophilus »

Happy BunnyCDN user here. Thanks for the writeup.

> Both SmartEdge and the deployment systems we use rely on Edge Storage and Bunny CDN to distribute data to the actual DNS servers. On the other hand, we just wiped out most of our global CDN capacity.

That’s the TLDR. What a stressful couple of hours that must have been for their team.

Re: Details of yesterday's Bunny CDN outage

#15

Post by PaywallBuster »

busymom0 wrote: post #5
throw1742 wrote: post #3

Earlier quoted context omitted.

They're based in Slovenia, so that was 10:25 AM local time for them.

I am going based off of the map here, Europe and then North America is their biggest market: https://bunny.net/network Seems like they were updating production during work hours for most people which is pretty odd imo. Usually I would expect them to get this done between midnight and 2-3am.

If you have a global infrastructure with worldwide customer base you'd want to do critical upgrades when everyone's at the office ready to jump at issues.

Re: Details of yesterday's Bunny CDN outage

#16

Post by YetAnotherNick »

They are making it sound like they did everything right and it was a issue of third party library. If we list all the libraries our code depend on, it will be in 1000s. I can't comprehend how a CDN does not have any canary or staging setup and in a update everything could go haywire in seconds. I think it is standard practice in any decent size company to have staging/canary and rollbacks.

Re: Details of yesterday's Bunny CDN outage

#17

Post by ram_rar »

> On June 22nd at 8:25 AM UTC, we released a new update designed to reduce the download size of the optimization database. Unfortunately, this managed to upload a corrupted file to the Edge Storage.

I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together.

> Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with a stack overflow exception, bypassing any exception handling and just exiting the process. Within minutes, our global DNS server fleet of close to a 100 servers was practically dead

This is exactly, why one needs a canary based deployments. I have seen umpteen amounts of issues being caught in canary, which has saved my team tons of firefighting time.

Re: Details of yesterday's Bunny CDN outage

#18

Post by ev1 »

busymom0 wrote: post #10

One of the comments on the post is: > One thing you could do in future is to url redirect any BunnyCDN url back to the clients original url, in essence disabling the CDN and getting your clients own hosts do what they were doing before they connected to BunnyCDN, yes it means our sites won't be as fast but its better than not loading up the files at all. I wonder if that is possible in technical terms? Isn't this a h…

Doing this if you are intentionally trying to protect or hide your origin is effectively guaranteed killing on the origin. For example, if Cloudflare unproxied one of my subdomains I'd leave them immediately, and likely have to change all my infrastructure and providers due to attacks.

This is also a terrible idea because of ACLs/firewalls only allowing traffic from CDN (this is extremely common for things like Cloudflare and Akamai) and relying on the CDN for access control.

Post reply on HN