Proxying the files has a number of benefits.
- The first is that I can have all my API endpoints under one domain. This simplifies downloading as you don't need to make a separate request to figure out where the file is stored.
- The storage servers that Hetzner sells only have 1 Gbps bandwidth. That runs out very quickly when a file goes viral. The 10 Gbps caching servers do a lot of heavy lifting here, this makes sure the disks in the storage nodes last longer.
- I can also decide to switch to a different storage system on my storage nodes when I want. I have been considering to deploy reed-solomon encoding for a while. That would make it impossible to link directly to a single storage server as a single file would also be distributed.
- Sending out this much data uses a lot of RAM for TCP send buffers. Installing this RAM on a single content delivery node is cheaper than installing it on every storage server.
To prevent the bandwidth load from affecting the UI speed I have a rate limiter on the download API which slows down when the uplink reaches 95% capacity. This way there is always some bandwidth left for the HTML and database communications.
With regards to content delivery: I want to use pixeldrain to serve static files. Nothing like the fancy site-wrapping tech that cloudflare uses. The idea is that users can have a file tree on pixeldrain somewhat like dropbox. They can copy the direct download link to that file and use it to embed videos, audio and pictures in their own websites. Because this is a lot simpler than other CDN services I can offer it at a very competitive price.