171–180 of 199 posts
Consider storing the data on Backblaze B2 ($0.005/GB/month) and serving content via Cloudflare (egress from B2 to Cloudflare is free through their Bandwidth Alliance). (No affiliation with either; just a happy customer for a tiny personal project)
Depends on what exactly you want to do with it. Hetzner has very cheap Storage boxes (10TB for $20/month with unlimited traffic) but those are closer to FTP boxes with a 10 connection limit. They are also down semi-regularly for maintenance. For rock-solid public hosting Cloudflare is probably a much better bet, but you're also paying 7 times the price. More than a dedicated server to host the files, but you get more…
support seems non existent cos no one answers emails or web chat…
I helped run a wireless research data archive for a while. We made smaller data sets available via internet download but for the larger data sets we asked people to send us a hard drive to get a copy. Sneakernet can be faster and cheaper than using the internet. Even if you wanted to distribute 10TB of _new_ data every month, mailing hard drives would probably be faster and cheaper, unless all your customers are on I…
Nothing beats the bandwidth of a station wagon full of hard drives:-)
That reminds me of the entertaining "I just want to serve 5 terabytes. Why is this so difficult?" video that someone made inside Google. It satirizes the difficulty of getting things done at production scale. https://www.youtube.com/watch?v=3t6L-FlfeaI
The interesting thing is there also paradoxes of large scale: things that get more difficult with increasing size. Medium- and smaller-scale can often be more flexible because they don't have to incur the pain of nonuniformity as scale increases. While they may not be able to afford optimizations or discounts with larger, standardized purchases, they can provide more personalized services large scale cannot hope to p…
Eventually, all outages are black swan events. If you have 1000 independent instances (i.e., 1000 customers), when the unexpected thing hits, you’re still 99.9% available during the time when the impacted instance is down.
Also, you can probably permanently prevent the black swan from hitting again before it hits again.
Earlier quoted context omitted.
That's if you use their CDN. Cloudflare R2 doesn't charge for egress bandwidth. If you have 100TB/mo to serve, try it and see what happens. I haven't heard of anyone being kicked off of R2 for using too much egress bandwidth yet. At scale, you'll pay a couple thousand dollars for Class B operations on R2, and another bunch for storing the 10 TB in the first place, but that's relatively cheap compared to other offerin…
CF is not particularly fond of non-Enterprise customers serving more than a few TB/mo. Source: $corp serves 150 TB/mo via CF and pays somewhere north of 50k+/yr for it
The whole point of R2 is to remove predatory practices of egregious charging for egress, and if somehow they went back on this promise that would be a very bad PR.
Consider storing the data on Backblaze B2 ($0.005/GB/month) and serving content via Cloudflare (egress from B2 to Cloudflare is free through their Bandwidth Alliance). (No affiliation with either; just a happy customer for a tiny personal project)
Man, thanks so much for this. I’m using Wasabi with a Yarkon front end right now and it’s great, but Backblaze/Cloudflare is looking like a serious contender.
I always assumed having a raspberry pi with a couple HDs in raid1 with IPFS or torrent would be the best way to do this. Giving another one of these raid1 rpis to a friend could make it reasonably available. I am very interested to know if there are good tools around this though, such as a good way to serve a filesystem (nfs-like for example) via torrent/ipfs and if the directories could be password protected in diff…
If you just want to be able to sync a directory between multiple devices with encryption options I'd recommend Syncthing. It's dead easy to set up, I've currently got it on a rpi backing up all my photos from my phone while syncing my Obsidian vault beteen my phone and desktop.
That said above 80 TB is looking hard for eternity, unless you can provide backup power and endure noise of spinning drives.