Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Details of yesterday's Bunny CDN outage

bunny.net

Post reply on HN
Print view View original

61–70 of 87 posts

Re: Details of yesterday's Bunny CDN outage

#61

Post by tyingq »

foobarbazetc wrote: post #54

I like how something is “auto-healing” when it’s like… has `Restart=on-failure` in systemd. Anyway, it’s always DNS. Always. “Unfortunately, that allowed something as simple as a corrupted file to crash down multiple layers of redundancy with no real way of bringing things back up.” You can spend many, many millions of $ on multi-AZ Kubernetes microservices blah blah blah and it’ll still be taken down by a SPOF, whic…

>Anyway, it’s always DNS. Always.

Which is disappointing. An infrastructure where the backend is VERY easy to make highly redundant. Thwarted by decisions not to do that easy work, or thwarted by client libraries that don't take advantage of it.

Re: Details of yesterday's Bunny CDN outage

#62

Post by zamalek »

This brings up one of my pet peeves: recursion. Of course there should have been other mitigations in place, but recursion is such a dangerous tool. So far as reasonably possible, I consider its only real purpose to confuse students in 101 courses.

I assume that they are using .Net, as SOEs bring down .Net processes. While that sounds like a strange implementation detail, the philosophy of the .Net team has always been "how do you reasonably recover from an stack overflow?" Even in C++ what happens if, for example, the allocator experiences a stack overflow while deallocating some RAII resource, or a finally block calls a function and allocates stack space, or... you get the idea.

The obvious thing to do here would be to limit recursion in the library (which amounts to safe recursion usage). BinaryPack does not have a recursion limit option, which makes it unsafe for any untrusted data (and that can include data that you produce, as Bunny experienced). Time to open a PR, I guess.

This applies to JSON, too. I would suggest that OP configure their serializer with a limit:

[1]: https://www.newtonsoft.com/json/help/html/MaxDepth.htm

Re: Details of yesterday's Bunny CDN outage

#63

Post by theandrewbailey »

zamalek wrote: post #62

This brings up one of my pet peeves: recursion. Of course there should have been other mitigations in place, but recursion is such a dangerous tool. So far as reasonably possible, I consider its only real purpose to confuse students in 101 courses. I assume that they are using .Net, as SOEs bring down .Net processes. While that sounds like a strange implementation detail, the philosophy of the .Net team has always be…

> So far as reasonably possible, I consider its only real purpose to confuse students in 101 courses.

I had a "high school" level programming class with Python before studying CS. I ran into CPython's recursion limit often and wondered why one would use recursion when for loops were a more reliable solution.

Nowadays, my "recursion" is for-looping over an object's children and calling some function on each child object.

Re: Details of yesterday's Bunny CDN outage

#64

Post by thekonqueror »

dejangp wrote: post #32

Dejan here from bunny.net. I was reading some of the comments, but wasn't sure where to reply, so I guess I'll post some additional details here. I tried to keep the blog post somewhat technical, but not overwhelm non-technical readers. So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructi…

Hi Dejan, we are evaluating Bunny for a long-term multi-tenant project. Today your support mentioned that cdn optimizer strips all origin headers. Is there any way permit some headers on a zone basis?

Re: Details of yesterday's Bunny CDN outage

#65

Post by yabones »

methyl wrote: post #56
jgrahamc wrote: post #34

Earlier quoted context omitted.

I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together. Oh man, you stirred up a really old Cloudflare memory. Back when I was working on our DNS infrastructure I wrote up a task that says: "RRDNS has no way of knowing how many lines to expect or whether what it is read is valid. This could create an issue where the LB map data is not available inside RRDNS." At t…

For whom, like myself, don't know the story, here it is: https://www.wired.com/story/lee-holloway-devastating-decline... I'm deeply moved after reading it. Can't imagine how tragic it must be for people who know Lee.

Wow, that is absolutely tragic. Neurodegenerative diseases are something I fear the most, having seen what Huntington's can do to somebody.

Re: Details of yesterday's Bunny CDN outage

#66

Post by gazby »

dejangp wrote: post #49
ing33k wrote: post #41

Earlier quoted context omitted.

hey dejan, we have been using BunnyCDN for quite some time. Thanks for the detailed writeup. looks like storage zones are still not fully stable ? after experiencing several issues with storage zones earlier, we migrated to pull a zone. we didn't had any major issues after the migration. what plans do you have to improve your storage zone ?

Hey, glad to hear that and sorry again about any issues. If you're experiencing any ongoing problems, please message our support team. I'm not aware of anything actively broken, but if there's a problem I'm sure we'll be able to help.

I have also had problems with storage zones. We experienced multiple periods of timeouts, super long TTFB, and 5xx responses. A ticket was opened (#136096) about the TTFB issue with full headers/curl output with an offer to supply any further useful information, but the response of "can you confirm this is no longer happening?" the following day discouraged me from further time spent there.

To this day US PoPs are still pulling from EU storage servers (our storage zone is in NY, replicated in DE).
We've since moved away from Bunny, but if there's anything I can do to help improve this situation I'd be happy to do it because it is otherwise a fantastic product for the price.

Re: Details of yesterday's Bunny CDN outage

#67

Post by 0des »

slackerIII wrote: post #59

Oh, this is a great writeup. I co-host a podcast on outages, and over and over we see cases where circular dependencies end up making recovery much harder. Also, not using a staged deployment is a recipe for disaster! We just wrapped up the first season, but I'm going to put this on the list of episodes for the second season: https://downtimeproject.com .

This is great! I love these types of podcasts. Adding this one to my subscriptions list right now. A bunny CDN episode would be fun. Thanks for putting this podcast on my radar.

Re: Details of yesterday's Bunny CDN outage

#69

Post by bob1029 »

zamalek wrote: post #62

This brings up one of my pet peeves: recursion. Of course there should have been other mitigations in place, but recursion is such a dangerous tool. So far as reasonably possible, I consider its only real purpose to confuse students in 101 courses. I assume that they are using .Net, as SOEs bring down .Net processes. While that sounds like a strange implementation detail, the philosophy of the .Net team has always be…

> recursion is such a dangerous tool.

The most effective tools for the job are usually the more dangerous ones. Certainly, you can do anything without recursion, but forcing this makes a lot of problems much harder than they need to be.

Re: Details of yesterday's Bunny CDN outage

#70

Post by vfaronov »

zamalek wrote: post #62

This brings up one of my pet peeves: recursion. Of course there should have been other mitigations in place, but recursion is such a dangerous tool. So far as reasonably possible, I consider its only real purpose to confuse students in 101 courses. I assume that they are using .Net, as SOEs bring down .Net processes. While that sounds like a strange implementation detail, the philosophy of the .Net team has always be…

> While that sounds like a strange implementation detail, the philosophy of the .Net team has always been "how do you reasonably recover from an stack overflow?"

Can you expand on this or link to any further reading? I just realized that this affects my platform (Go) as well, but I don't understand the reasoning. Why can't stack overflow be treated just like any other exception, unwinding the stack up to the nearest frame that has catch/recover in place (if any)?

Post reply on HN