Building Your Own CDN for Fun and Profit • Hacker News Board

Re: Building Your Own CDN for Fun and Profit

#41

Post by janoszen » Wed, Feb 14, 2018, 6:04 PM UTC

zzzcpan wrote: post #39

Fast nameservers are not as important as author suggests. But either way one extra indirection for nameservers would allow you to choose nameserver records dynamically too. And with large enough TTLs and some traffic, clients won't have to go all the way to find out the closest nameserver, essentially providing clients with the fastest one from the cache. Since redundancy is built in into DNS any nameserver with larg…

(I'm the author.) This whole setup is built for a comparatively low traffic blog, so DNS caching won't help much. (On normal days I get ~100 visitors.) This is compounded by the TTL which is 60s to account for node failures.

The optimization level is in the sub 1 second range, so not having to pay one large RTT penalty for a DNS lookup is quite important. I've measured 300+ms RTT to Australia on the previous box I was using, that impacted the load times quite severely.

Re: Building Your Own CDN for Fun and Profit

#42

Post by lessclue » Wed, Feb 14, 2018, 6:07 PM UTC

kirankn wrote: post #2

We currently use KeyCDN which works out well, both performance & money wise. You may want to try it out.

Same here. Cloudfront was ridiculous. KeyCDN works great for us.

Re: Building Your Own CDN for Fun and Profit

#43

Post by janoszen » Wed, Feb 14, 2018, 6:08 PM UTC

dzolvd wrote: post #34

janoszen wrote: post #33

Earlier quoted context omitted.

Interesting, although I specifically wanted to build a push CDN (where I can push the content) rather than a pull CDN (that works with an origin) to avoid the added latency with cache misses.

Makes sense, I am enjoying looking through the source as we are moving to an ansible and hopefully dockerized deployment model.

Of course it's dockerized, it has to be cool, right? :)

Ansible is running docker-compose up -d on deployment an Traefik is doing the magic. I want to extend it to host multiple sites in the future. (Btw. Ansible ran from a central location is painfully slow because of the large latency to the edge nodes.)

The content itself is deployed using rsync, Ansible was just too painfully slow for that.

Re: Building Your Own CDN for Fun and Profit

#44

Post by ksec » Wed, Feb 14, 2018, 6:56 PM UTC

kirankn wrote: post #2

We currently use KeyCDN which works out well, both performance & money wise. You may want to try it out.

Why KeyCDN ? Why not MaxCDN or Fastly etc?

Re: Building Your Own CDN for Fun and Profit

#45

Post by photonios » Wed, Feb 14, 2018, 7:08 PM UTC

scurvy wrote: post #15

thisisit wrote: post #5

I am curious, if anyone knows how well does Akamai work in the CDN world?

They're good but only at $100k per month and above. You really need to use their full suite of products to get the full benefit, and by that time you'll be at $100k per month. They are OK not great for smaller accounts.

At work we use Akamai to serve a large website and we pay about a 10th of that and it definitely benefits us. But that's only because they simply have a large network and are one of the few CDN's that have POP's close to our customers. Other than that, it's just overkill for small businesses.

Re: Building Your Own CDN for Fun and Profit

#46

Post by davidu » Wed, Feb 14, 2018, 7:15 PM UTC

jgrahamc wrote: post #25

davidu wrote: post #24

"Second, BGP routes are not that stable." This has been disproved for close to ten years empirically and academically[1]. Route flaps generally result in convergence to the exact same destination if it has another path and is still online. If it's offline, then it's working as intended, and that's no different from a server being rotated via a DNS pool going down. 1: Quick search: https://www.google.com/search?q=tcp+…

That's not our experience either. BGP is fine. But it is the case that transit and peering connections are not stable (in the sense of going up and down randomly or suddenly experiencing high levels of packet loss) and active monitoring is a must.

Is your experience that when routes reconverge, they still select the same end POP they did prior to the flap?

Assuming the end POP is still reachable along another regional route, I believe all the data I've seen shows that the client almost always hits the same destination they did before the flap.

Re: Building Your Own CDN for Fun and Profit

#47

Post by lostcolony » Wed, Feb 14, 2018, 7:15 PM UTC

nathan_f77 wrote: post #22

That's a fun project. For production websites and blogs, I'm pretty happy with Netlify, CloudFlare, and CloudFront. But CloudFront charges $600 per month for custom SSL certificates [1], so you could save a lot of money by just spinning up ~10 servers in different AWS regions. I noticed this line at the bottom of the page: "When it comes to picking a solution, I often choose the less traveled road". I don't agree wit…

'It's always better to choose the most-traveled roads, especially in DevOps'

Maybe. I agree it's not as clearcut as always pick the less traveled road, but the difference between the two may include a competitive advantage that you'd be unwise to overlook. I mean, you're on HN; Paul Graham and Common Lisp back in the 90s is an excellent example.

Re: Building Your Own CDN for Fun and Profit

#48

Post by jtl999 » Wed, Feb 14, 2018, 7:19 PM UTC

kijin wrote: post #18

mikerg87 wrote: post #8

if you have a specialized application knowing how to do this can be quite useful. CDN pops are almost not existent across much of the Middle East and Africa. Sometimes building your own is the only way until a commercial offering becomes available.

Also, PoPs in some regions are often nearly useless even if they exist on paper. For example, Cloudflare has a PoP in Seoul, but it has such limited bandwidth that most sites using Cloudflare are routed to Tokyo, Hong Kong, and even Los Angeles. Several of my clients in Korea signed up for Cloudflare a few years ago when the local PoP was still usable, but now all but two of them have canceled their subscriptions. In…

Cloudflare has a Vancouver PoP but Telus Vancouver doesn't use it, all traffic is routed to Seattle. (as an example)

  colo=SEA
  spdy=h2
  http=h2
  loc=CA

Re: Building Your Own CDN for Fun and Profit

#49

Post by rsync » Wed, Feb 14, 2018, 8:32 PM UTC

I have always thought it would be a fun and inspiring project to deploy a global CDN ... my career and my lifelong hobby have both been "UNIX sysadmin" and I love running networks ...

However, I spoke about this to some ISP/NANOG folks that I trust and they said that running a real CDN is a nightmare because all of your links (providers) hate you ... you're producing the exact opposite of the traffic that they want and they will not give you any breaks or help or benefits since you are their worst customer.

How accurate was that assessment ?

Re: Building Your Own CDN for Fun and Profit

#50

Post by sacheendra » Wed, Feb 14, 2018, 8:38 PM UTC

rsync wrote: post #49

I have always thought it would be a fun and inspiring project to deploy a global CDN ... my career and my lifelong hobby have both been "UNIX sysadmin" and I love running networks ... However, I spoke about this to some ISP/NANOG folks that I trust and they said that running a real CDN is a nightmare because all of your links (providers) hate you ... you're producing the exact opposite of the traffic that they want a…

I have also heard the same. An interesting thing to do would be to also be a commercial ISP (i.e., sell to datacenters and businesses). Now that is the traffic all the ISPs want as outgoing traffic goes to such networks.

Running a Global CDN and ISP might be a tad too ambitious.