Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Details of yesterday's Bunny CDN outage

bunny.net

Post reply on HN
Print view View original

81–87 of 87 posts

Re: Details of yesterday's Bunny CDN outage

#81

Post by throwdbaaway »

dejangp wrote: post #32

Dejan here from bunny.net. I was reading some of the comments, but wasn't sure where to reply, so I guess I'll post some additional details here. I tried to keep the blog post somewhat technical, but not overwhelm non-technical readers. So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructi…

> This piece of data is highly dynamic and processes every 30 seconds or so based on hundreds of thousands of metrics.

Perhaps you guys need a ... database? Relevant HN discussion few months ago when tailscale migrated from json to etcd: https://news.ycombinator.com/item?id=25767128

Re: Details of yesterday's Bunny CDN outage

#82

Post by vfaronov »

zamalek wrote: post #72
vfaronov wrote: post #70

Earlier quoted context omitted.

> While that sounds like a strange implementation detail, the philosophy of the .Net team has always been "how do you reasonably recover from an stack overflow?" Can you expand on this or link to any further reading? I just realized that this affects my platform (Go) as well, but I don't understand the reasoning. Why can't stack overflow be treated just like any other exception, unwinding the stack up to the nearest…

> Why can't stack overflow be treated just like any other exception[...]? Consider the following code: func overflows() { defer a() fmt.Println("hello") // The answer lies in trying to figure out how Go would successfully unwind that stack, it can't: when it calls `a` it will simply overflow again. Something that has been discussed is "StackAboutToOverflowException", but that only kicks the bucket down the road (unwi…

But doesn’t this apply to “normal” panics as well? When unwinding the stack of a panicking goroutine, any deferred call might panic again, in which case Go keeps walking up the stack with the new panic. In a typical server situation, it will eventually reach some generic “log and don’t crash” function, which is unlikely to panic or overflow.

Perhaps one difference is that, while panics are always avoidable in a recovery function, stack overflows are not (if it happens to be deep enough already). Does the argument go “even a seemingly safe recovery function can’t be guaranteed to succeed, so prevent the illusion of safety”?

(To be clear: I’m not arguing, just trying to understand.)

Re: Details of yesterday's Bunny CDN outage

#83

Post by listenallyall »

ing33k wrote: post #40
path411 wrote: post #12

Sounds like they got really lucky they could get it back up so quickly. They must have some very talented engineers working there. My take aways though were they should have tested the update better. They should have their production environment more segmented with staggered updates so they have much more contained disasters. And they should have had much better catastrophic failure plans in place.

they are very a small team. Their CEO codes !

...and provides support too. Even to a low-traffic, low-spend user whose site generates just a few pennies of revenue for Bunny. Yet every time I made a support request, I got a response in minutes, often from Dejan himself. Highly recommend.

Re: Details of yesterday's Bunny CDN outage

#84

Post by ksec »

manishsharan wrote: post #31
string wrote: post #8

Good and clear explanation. This is a risk you take when you use a CDN, I still think the benefits outweigh the occasional downtime. I'm a big fan of BunnyCDN, they've saved me a lot of money over the past few years. I'm sure I'd be fuming if I worked at some multi-million dollar company but as someone that mainly works for smaller businesses it's not the end of the world, I suspect most of my clients haven't even no…

TIL about BunnyCDN. I had been paying $0.08 per GB on AWS Cloudfront whereas BunnyCDN is only $0.01 per GB. Can you comment on you experience with them ? Are the APIs comprehensive e.g. cache invalidation ? do they support cookie base authorization ? Any support Geo-Fencing?

>AWS Cloudfront

AWS Cloudfront is neither the fastest or cheapest, in both bulk and small file transfer. I cant think of a single technical reason it is better. Fastly, Akamai, Limelight, Cloudflare or even good old EdgeCast. They all have their strong point in some of their niche services or domain.

Any reason for using AWS Cloudfront Enterprise purchase reason? Or are there some technical superiority I am not seeing?

Re: Details of yesterday's Bunny CDN outage

#85

Post by zamalek »

vfaronov wrote: post #82
zamalek wrote: post #72

Earlier quoted context omitted.

> Why can't stack overflow be treated just like any other exception[...]? Consider the following code: func overflows() { defer a() fmt.Println("hello") // The answer lies in trying to figure out how Go would successfully unwind that stack, it can't: when it calls `a` it will simply overflow again. Something that has been discussed is "StackAboutToOverflowException", but that only kicks the bucket down the road (unwi…

But doesn’t this apply to “normal” panics as well? When unwinding the stack of a panicking goroutine, any deferred call might panic again, in which case Go keeps walking up the stack with the new panic. In a typical server situation, it will eventually reach some generic “log and don’t crash” function, which is unlikely to panic or overflow. Perhaps one difference is that, while panics are always avoidable in a recov…

I'm not actually sure what Go would do in a double-fault scenario (that's when a panic causes a panic), but assuming it can recover from that:

In the absolute worst case scenario: stack unwinding is itself a piece of code[1]. In order to initiate the stack unwind, and deal with SEH/defer/dealloc, the Go runtime would need stack space to call that method. Someone might say, "freeze the stack and do the unwind on a different thread." The problem is the bit in the quotes is, again, at least one stack frame and needs stack space to execute.

I just checked the Go source, and it basically uses a linked list of stack frames in the heap[2]. If a stack is about to overflow, it allocates a new stack and continues in that stack. This does have a very minor performance penalty. So you're safe from this edge case :).

[1]: https://www.nongnu.org/libunwind/ [2]: https://golang.org/src/runtime/stack.go

Re: Details of yesterday's Bunny CDN outage

#86

Post by cutemonster »

Matthias247 wrote: post #80
ram_rar wrote: post #17

> On June 22nd at 8:25 AM UTC, we released a new update designed to reduce the download size of the optimization database. Unfortunately, this managed to upload a corrupted file to the Edge Storage. I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together. > Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with…

That doesn't protect against a file already being generated in a broken fashion. Or that it's content is not compatible to the newest schema you are using for deserialization. For serialization in a distributed system you always want to have a parser which can detect invalid data and has means to support forward and backward compatibility.

> forward and backward compatibility

Also for HTTP requests I suppose

Re: Details of yesterday's Bunny CDN outage

#87

Post by JetSpiegel »

plett wrote: post #50
rapsey wrote: post #42

Earlier quoted context omitted.

I think a public company will never dare go into as much detail as bunny did. Or maybe it is just the size of the organisation that discourages that.

Cloudflare's RFO blog posts are incredibly detailed. Each time I read one, I feel reasonably confident that they have learnt from the mistakes that lead to that outage and that it shouldn't happen again. https://blog.cloudflare.com/tag/outage/

Yeah, but Cloudflare is accountable to the US Congress, they better be transparent.
Post reply on HN