I am curious, if anyone knows how well does Akamai work in the CDN world?
Don't use them unless you have to. The vendor lock-in is strong.
Building Your Own CDN for Fun and Profit
11–20 of 65 posts
Re: Building Your Own CDN for Fun and Profit
#12> Second, BGP routes are not that stable. While DNS requests only require a single packet to be sent in both directions, HTTP (web) requests require establishing a connection to download the content. If the route changes, the HTTP connection is broken. I thought Cloudflare uses Anycast to avoid targeted DDOS? How do they handle changing routes during HTTP requests?
Anycast means there are multiple routes going to the same destination. You get the route that is the shortest path via BGP to the anycast IP (least number of BGP hops). Once you have an established TCP session via one route, it will remain established through that route, as long as that route is still the “shortest” between your IP and the anycast IP. The route will not “change” unless cloudflare changes their routin…
I think it should be clarified that "destination" refers to an IP address, not an individual host. My understanding is that anycasting means a single address corresponds to multiple hosts achieved by simply advertising it from several sources with BGP, and you will often still have multiple redundant routes to any of the individual hosts behind the anycast IP because most locations will have redundant internet links.
Depending on how the routing is set up, it doesn't matter if the route changes so long as you end up on the same host consistently (or one that can at least pretend it's the same host if you do some kind of fancy session mirroring, perhaps)
Re: Building Your Own CDN for Fun and Profit
#13> Second, BGP routes are not that stable. While DNS requests only require a single packet to be sent in both directions, HTTP (web) requests require establishing a connection to download the content. If the route changes, the HTTP connection is broken. I thought Cloudflare uses Anycast to avoid targeted DDOS? How do they handle changing routes during HTTP requests?
There's alot of fear around the possibilities of flapping routes, but alot of real world data seems to show it doesn't happen seem to impact web traffic that often.
People often mix and match anycast/Geo DNS and anycast/unicast http.
Some even go a step further and, for video files, anycast to a node that 302s to it's own unicast address.
Re: Building Your Own CDN for Fun and Profit
#14I don't understand how his use of Traefik gets round the SSL pain point? > Using SSL/TLS certificates > The next pain point is using SSL/TLS certificates. Actually, let’s call them what they are: x509 certificates. Each of your edge locations needs to have a valid certificate for your domain. The simple solution, of course, is to use LetsEncrypt to generate a different certificate for each, but you have to be careful…
I guess they can use Traefik to distribute the certificates, so instead of having each node request their own set of certificates, he can instead request them once and distribute the certificate to all the other nodes, and keep themselves under the limit set by LetsEncrypt.
Re: Building Your Own CDN for Fun and Profit
#15I am curious, if anyone knows how well does Akamai work in the CDN world?
They're good but only at $100k per month and above. You really need to use their full suite of products to get the full benefit, and by that time you'll be at $100k per month.
They are OK not great for smaller accounts.
Re: Building Your Own CDN for Fun and Profit
#16> Second, BGP routes are not that stable. While DNS requests only require a single packet to be sent in both directions, HTTP (web) requests require establishing a connection to download the content. If the route changes, the HTTP connection is broken. I thought Cloudflare uses Anycast to avoid targeted DDOS? How do they handle changing routes during HTTP requests?
There's alot of fear around the possibilities of flapping routes, but alot of real world data seems to show it doesn't happen seem to impact web traffic that often. People often mix and match anycast/Geo DNS and anycast/unicast http. Some even go a step further and, for video files, anycast to a node that 302s to it's own unicast address.
Indeed. There are also right ways to setup anycasting and wrong ways.
Right way: 1-2 major Tier1 carriers across all of your PoPs with local peering for regional eyeball networks.
Wrong way: Using a different set of transit carriers at each location.
You really don't want that many AS paths to reach your content from a given location (3-4 is more than enough). What you're really going for with BGP anycasting is that your local ISP has a direct route to the closest PoP via exchange peering, or that the Tier1 path drop you off to the "closest" route. Transit carriers do this for a living, and they're usually quite good at figuring out route weighting inside their own network.
Yes, I know Netflix does it differently but they use a lot more smart geo DNS routing than anycasting.
Edit: IMHO it's also better to choose a Tier1 with a moderate sized network that values stability and performance over size. So someone like NTT over say Level3.
Re: Building Your Own CDN for Fun and Profit
#17I don't understand how his use of Traefik gets round the SSL pain point? > Using SSL/TLS certificates > The next pain point is using SSL/TLS certificates. Actually, let’s call them what they are: x509 certificates. Each of your edge locations needs to have a valid certificate for your domain. The simple solution, of course, is to use LetsEncrypt to generate a different certificate for each, but you have to be careful…
Traefik can simply request certificates using the DNS verification method, as opposed to the certbot HTTP verification. (HTTP would not work with a distributed setup like this.) Alternatively, Traefik can also synchronize certificate requests using one of the many key-value stores supported (untested as of yet).
The drawback of the DNS method without synchronization between the nodes is that you run into the LetsEncrypt rate limit quite easily. My expansion to ap-southeast-1 and sa-east-1 is waiting for the LE cooldown.
Disclaimer: I'm the author of the article.
Re: Building Your Own CDN for Fun and Profit
#18if you have a specialized application knowing how to do this can be quite useful. CDN pops are almost not existent across much of the Middle East and Africa. Sometimes building your own is the only way until a commercial offering becomes available.
Also, PoPs in some regions are often nearly useless even if they exist on paper.
For example, Cloudflare has a PoP in Seoul, but it has such limited bandwidth that most sites using Cloudflare are routed to Tokyo, Hong Kong, and even Los Angeles. Several of my clients in Korea signed up for Cloudflare a few years ago when the local PoP was still usable, but now all but two of them have canceled their subscriptions. Instead, I've been building a lot of caching proxies for them lately.
If anyone is here for the Winter Olympics right now and some of your favorite sites don't seem to be living up to Korea's reputation for ultra-fast internet, Cloudflare might be one reason. (Meanwhile, Amazon's PoP in Seoul is perfectly fine, albeit expensive.)
Re: Building Your Own CDN for Fun and Profit
#19> Second, BGP routes are not that stable. While DNS requests only require a single packet to be sent in both directions, HTTP (web) requests require establishing a connection to download the content. If the route changes, the HTTP connection is broken. I thought Cloudflare uses Anycast to avoid targeted DDOS? How do they handle changing routes during HTTP requests?
Anycast means there are multiple routes going to the same destination. You get the route that is the shortest path via BGP to the anycast IP (least number of BGP hops). Once you have an established TCP session via one route, it will remain established through that route, as long as that route is still the “shortest” between your IP and the anycast IP. The route will not “change” unless cloudflare changes their routin…
> The route will not “change” unless cloudflare changes their routing, or you change your location/IP so that a shorter route exists. Once you’ve changed your IP, you’ve already interrupted any TCP sessions anyway.
That's what I thought, too. But the article explicitly states this as a potential issue.
Re: Building Your Own CDN for Fun and Profit
#20We currently use KeyCDN which works out well, both performance & money wise. You may want to try it out.
Yeah. They are pretty good and very good value for money. We went from Cloudfront -> Edgecast -> KeyCDN and each change reduced our costs. Cloudfront can become really expensive since they charge for each HTTP request in addition to bandwidth.