Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Details of yesterday's Bunny CDN outage

bunny.net

Post reply on HN
Print view View original

71–80 of 87 posts

Re: Details of yesterday's Bunny CDN outage

#71

Post by kalev »

manishsharan wrote: post #31
string wrote: post #8

Good and clear explanation. This is a risk you take when you use a CDN, I still think the benefits outweigh the occasional downtime. I'm a big fan of BunnyCDN, they've saved me a lot of money over the past few years. I'm sure I'd be fuming if I worked at some multi-million dollar company but as someone that mainly works for smaller businesses it's not the end of the world, I suspect most of my clients haven't even no…

TIL about BunnyCDN. I had been paying $0.08 per GB on AWS Cloudfront whereas BunnyCDN is only $0.01 per GB. Can you comment on you experience with them ? Are the APIs comprehensive e.g. cache invalidation ? do they support cookie base authorization ? Any support Geo-Fencing?

First time using a CDN improved our site [1] performance by a huge amount, thanks to BunnyCDN. Really easy to setup, great dashboard. The image optimizer for a flat rate works really really well. Only missing option is to rotate images, which I opened a feature request for with them.

You can see our CDN usage inspecting the URLs to the product images. Size attributes are added to the URL and Bunny automatically resizes and compresses the images on the fly.

[1] https://www.airsoftbazaar.nl

Re: Details of yesterday's Bunny CDN outage

#72

Post by zamalek »

vfaronov wrote: post #70
zamalek wrote: post #62

This brings up one of my pet peeves: recursion. Of course there should have been other mitigations in place, but recursion is such a dangerous tool. So far as reasonably possible, I consider its only real purpose to confuse students in 101 courses. I assume that they are using .Net, as SOEs bring down .Net processes. While that sounds like a strange implementation detail, the philosophy of the .Net team has always be…

> While that sounds like a strange implementation detail, the philosophy of the .Net team has always been "how do you reasonably recover from an stack overflow?" Can you expand on this or link to any further reading? I just realized that this affects my platform (Go) as well, but I don't understand the reasoning. Why can't stack overflow be treated just like any other exception, unwinding the stack up to the nearest…

> Why can't stack overflow be treated just like any other exception[...]?

Consider the following code:

    func overflows() {
        defer a()
        
        fmt.Println("hello") //
The answer lies in trying to figure out how Go would successfully unwind that stack, it can't: when it calls `a` it will simply overflow again. Something that has been discussed is "StackAboutToOverflowException", but that only kicks the bucket down the road (unwinding could still cause an overflow).

In truth, the problem exists because of implicit calls at the end of methods interacting with stack overflows, whether that's because of defer-like functionality, structured exception handling, or deconstructors.

Re: Details of yesterday's Bunny CDN outage

#74

Post by EricE »

manigandham wrote: post #68

All this focus on redundancy should be replaced with a focus on recovery. Perfect availability is already impossible. For all practical uses, something that recovers within minutes is better than trying to always be online and failing horribly.

A backup vendor once pointed out that backup was the most misnamed product/function in all of computerdom. He argued it should really be referred to as restore, since when the chips are down that's what you really, really care about. That really resonated with the young sysadmin I was at the time.

Very similar to the story about the planes with holes coming back in WWII and the initial analysis of adding more armor to where the holes were, when someone flipped it and pointed out that armor was needed where the holes weren't since planes with holes in those spots weren't the ones coming back.

Re: Details of yesterday's Bunny CDN outage

#75

Post by T4m2 »

gazby wrote: post #66
dejangp wrote: post #49

Earlier quoted context omitted.

Hey, glad to hear that and sorry again about any issues. If you're experiencing any ongoing problems, please message our support team. I'm not aware of anything actively broken, but if there's a problem I'm sure we'll be able to help.

I have also had problems with storage zones. We experienced multiple periods of timeouts, super long TTFB, and 5xx responses. A ticket was opened (#136096) about the TTFB issue with full headers/curl output with an offer to supply any further useful information, but the response of "can you confirm this is no longer happening?" the following day discouraged me from further time spent there. To this day US PoPs are st…

We had the same, super long TTFB and lots of 5xx errors, seems to be mostly fixed now, but there are defiantly things that could be done differently, however given the pricing and feature set I'm happy with the service

Would love additional capabilities within the image optimizer such as methods of crop

Re: Details of yesterday's Bunny CDN outage

#76

Post by sysbot »

dejangp wrote: post #32

Dejan here from bunny.net. I was reading some of the comments, but wasn't sure where to reply, so I guess I'll post some additional details here. I tried to keep the blog post somewhat technical, but not overwhelm non-technical readers. So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructi…

From the article "Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with a stack overflow exception, bypassing any exception handling and just exiting the process. Within minutes, our global DNS server fleet of close to a 100 servers was practically dead." and from your comment "We do the same for the CDN and always use canary testing if possible. We unfortunately never assumed this piece of software could cause all the DNS servers to stack overflow."

This read like the DNS software is being changed. As some people already mentioned is this a corruption where checksum would of been prevented the stack overflow or would a canary detected this? Why would a change to DNS server software not canaried?

Re: Details of yesterday's Bunny CDN outage

#77

Post by unilynx »

sysbot wrote: post #76
dejangp wrote: post #32

Dejan here from bunny.net. I was reading some of the comments, but wasn't sure where to reply, so I guess I'll post some additional details here. I tried to keep the blog post somewhat technical, but not overwhelm non-technical readers. So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructi…

From the article "Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with a stack overflow exception, bypassing any exception handling and just exiting the process. Within minutes, our global DNS server fleet of close to a 100 servers was practically dead." and from your comment "We do the same for the CDN and always use canary testing if possible. We unfortunately…

I read it as "DNS software changed, that worked fine, but it turns out we sometimes generate a broken database - not often enough to see it hit during canary, but devastating when it finally happened"

GP also notes that this database changed perhaps every 30 seconds

Just a few guesses.. if you have a process that corrupts a random byte every 100.000 runs, and you run it every 30 seconds, it might take days before you're at 50% odds of having seen it happening. and if that used to be a text or JSON database, flipping a random bit might not even corrupt anything important. Or if the code swallows the exception at some level, it might even self-heal after 30 seconds when new data comes in, causing an unnoticed blib in the monitoring if at all

Now I don't know what binary pack does exactly, but if you were to replace the above process with something that compresses data, a flipped bit will corrupt a lot more data, often everything from that point forwards (where text or json is pretty self-syncronizing). And if your new code falls over completely if that happens, no more self-healing.

I can totally imagine missing an event like that during canary testing

Re: Details of yesterday's Bunny CDN outage

#78

Post by iJohnDoe »

Slightly off-topic, has anyone else noticed higher latency with internet traffic going in or out of Germany? Just in general?

Frankfurt was mentioned in the post and I immediately thought it would be a bad idea because I’ve always seen USA to Germany traffic have higher latency. Maybe within Europe it’s fine.

Re: Details of yesterday's Bunny CDN outage

#79

Post by maddyboo »

methyl wrote: post #56
jgrahamc wrote: post #34

Earlier quoted context omitted.

I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together. Oh man, you stirred up a really old Cloudflare memory. Back when I was working on our DNS infrastructure I wrote up a task that says: "RRDNS has no way of knowing how many lines to expect or whether what it is read is valid. This could create an issue where the LB map data is not available inside RRDNS." At t…

For whom, like myself, don't know the story, here it is: https://www.wired.com/story/lee-holloway-devastating-decline... I'm deeply moved after reading it. Can't imagine how tragic it must be for people who know Lee.

Sounds similar to what happened to Nietzsche:

https://en.wikipedia.org/wiki/Friedrich_Nietzsche#Mental_ill...

Re: Details of yesterday's Bunny CDN outage

#80

Post by Matthias247 »

ram_rar wrote: post #17

> On June 22nd at 8:25 AM UTC, we released a new update designed to reduce the download size of the optimization database. Unfortunately, this managed to upload a corrupted file to the Edge Storage. I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together. > Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with…

That doesn't protect against a file already being generated in a broken fashion. Or that it's content is not compatible to the newest schema you are using for deserialization.

For serialization in a distributed system you always want to have a parser which can detect invalid data and has means to support forward and backward compatibility.

Post reply on HN