Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to implement a multi-CDN strategy

blog.streamroot.io

Post reply on HN
Print view View original

51–59 of 59 posts

Re: How to implement a multi-CDN strategy

#51

Post by eldod »

zzzcpan wrote: post #24
howdjisjje wrote: post #19

Earlier quoted context omitted.

You solution to bandwidth congestion is for everyone to use 3x+ more bandwidth than they need?

For video you can fetch different chunks from different endpoints simultaneously, not the same chunk, therefore not wasting bandwidth at all.

This is more or less what we do with our our client-side switching solution at Streamroot: we first make sure the user has enough video segments in its buffer, and then we try to get the next video segments for different CDNs, so we're able to compare the download speeds, latency and RTT between the different CDNs without adding any overhead. You don't necessarily download the segments at the same time, but with some estimation and smoothing algoriths you're able to have some meaningful scores for each CDN. The concepts and problematics here are very close to the problematic of bandwidth estimation for HTTP Adaptive Bitrate streaming formats like HLS & DASH, because you have an instable network, and you are only able to estimate the bandwidth from discrete segment measurements. [0]

If you want to do it on a sub-asset (video segment, or image or JS file) level, it's possible by doing byte-range requests (ask bytes 0-100 from CDN A and 101-200 from CDN B), but in that case you still add some overhead for establishing the TCP connection, and in the end as you need the whole asset to use it, you'll just limit the download speed to the minimum of the two.

[0]https://blog.streamroot.io/abr-algorithms-work-optimize-stac...

Re: How to implement a multi-CDN strategy

#53

Post by gstaro »

toast0 wrote: post #32
gstaro wrote: post #7

Earlier quoted context omitted.

Can't you do that with DNS records where there are multiple IPs on a A record? Basically, we're already doing this for fault tolerance and load balancing within a single CDN. Except that currently we randomize the IPs. To enforce priorities, you'd want the IPs in the A record at least partially ordered by provider.

Multiple IPs on an A record works to some extent, most (many?) browsers will silently retry another ip from the list if some of the IPs don't accept a connection; I don't know if they'll try another IP on timeout though. But you can't actually expect any ordering to make it through to the client. Your authoritative server may reorder the records, their recursive server may reorder the records, and the client resoluti…

Thanks for clarifying!

Re: How to implement a multi-CDN strategy

#54

Post by toast0 »

toast0 wrote: post #32
gstaro wrote: post #7

Earlier quoted context omitted.

Can't you do that with DNS records where there are multiple IPs on a A record? Basically, we're already doing this for fault tolerance and load balancing within a single CDN. Except that currently we randomize the IPs. To enforce priorities, you'd want the IPs in the A record at least partially ordered by provider.

Multiple IPs on an A record works to some extent, most (many?) browsers will silently retry another ip from the list if some of the IPs don't accept a connection; I don't know if they'll try another IP on timeout though. But you can't actually expect any ordering to make it through to the client. Your authoritative server may reorder the records, their recursive server may reorder the records, and the client resoluti…

For reference, RFC 3484 [1] is the misguided RFC that tells people they should sort their DNS responses to maximize the common prefix between the source and destination address. This is probably helpful when the common prefix is meaningful, but when numbering agencies give out neighboring /24's to unrelated networks, and related networks often have ips widely distributed across the overall address space, it's not actually useful.

[1] https://www.ietf.org/rfc/rfc3484.txt

Re: How to implement a multi-CDN strategy

#55

Post by jloveless »

eldod wrote: post #48
jloveless wrote: post #22

This is a nicely written article, however it's worth noting that the performance/reliability/availability differences across CDNs at a particular moment in time are pretty much non existent. These providers share the same backbone networks, same IX PoPs etc and thereby have little diversification benefit. See https://blog.edgemesh.com/understanding-diversification-netw... ) Where multi-CDN really shines is helping wi…

Have to disagree here, you can have very different performances from different CDNs for the same user at the same time. Some CDNs have their own backbone (or at least partially), a lot of them are using different routes, and a lot of the time, the issue is not with the backbone, but with peering inter-connections, which can be different between each ISP & CDN. And a CDN's capacity is shared between all its customers,…

In this case - given this and the other comments - I stand corrected! I would love to see some data on examples of this occurring in the wild - and it 100% makes sense that a congested CDN provider would impact neighbors. It would be great if someone could do a writeup on examples and the detection/mitigation strategies. Perhaps issues like these (alongside cost) are driving the DIY CDN adoptions (Apple being the exemplar but also Tesla etc)? Also the Pinterest example is a great real world example - and they do an awesome job especially given the size of that cache - so there must be so real value from a performance standpoint! Out of curiosity does it seem like these dynamic switching decisions are better at the server level or client level?

Re: How to implement a multi-CDN strategy

#56

Post by [deleted] »

jloveless wrote: post #22

This is a nicely written article, however it's worth noting that the performance/reliability/availability differences across CDNs at a particular moment in time are pretty much non existent. These providers share the same backbone networks, same IX PoPs etc and thereby have little diversification benefit. See https://blog.edgemesh.com/understanding-diversification-netw... ) Where multi-CDN really shines is helping wi…

[deleted]

Re: How to implement a multi-CDN strategy

#57

Post by wtracy »

howdjisjje wrote: post #19
esotericn wrote: post #17

Earlier quoted context omitted.

"If that doesn't work" isn't the problem. As a silly limiting example, imagine that you host Netflix on your dial-up connection as url A. It works. Oh, okay, right, let's set a timeout then, if it takes more than 1 second to load, we try url B. That works, but now we've got a 1 second delay on everything. Okay, we'll update the default to be url B. Conditions are changing all the time as a result of bottlenecks in th…

You solution to bandwidth congestion is for everyone to use 3x+ more bandwidth than they need?

They're not consuming 3x bandwidth if they're bailing out after downloading less then a kilobyte of a video that's tends or hundreds of megabytes in size.

That extra bandwidth is a rounding error in the grand scheme of things.

It could be important, though, for the client to signal the server to close the connection. Theoretically the connection would drop after several seconds and the server would stop transmitting, but I could imagine some middleware cheerfully downloading the whole stream and throwing it away.

Re: How to implement a multi-CDN strategy

#58

Post by axeldelmas »

wtracy wrote: post #57
howdjisjje wrote: post #19

Earlier quoted context omitted.

You solution to bandwidth congestion is for everyone to use 3x+ more bandwidth than they need?

They're not consuming 3x bandwidth if they're bailing out after downloading less then a kilobyte of a video that's tends or hundreds of megabytes in size. That extra bandwidth is a rounding error in the grand scheme of things. It could be important, though, for the client to signal the server to close the connection. Theoretically the connection would drop after several seconds and the server would stop transmitting,…

The problem with with this approach is that you're only considering time to first byte, which is part of the equation especially in the case of smaller files like scripts, but in case of larger files like video segments, throughput is more important. If you only wait for 1kB to download, then you essentially measure time to first byte.

Then the instruction to stop the download is not instantaneous, so by the time you realize you have downloaded 1kB on the client side, the server might already have sent the whole video segment on the other side, so this is not the way to go in order to optimize congestion

Re: How to implement a multi-CDN strategy

#59

Post by Heag3aec »

toomuchtodo wrote: post #8
Heag3aec wrote: post #3

Earlier quoted context omitted.

Combine with SRI and some convention to just ask one of several hosts for it based on the hash(es) and we have concent-addressable loading.

So IPFS?

IPFS would be a particular implementation of those more general concepts. SRI + multiple HTTP sources would be another more incremental approach.
Post reply on HN