I would do it in S3 + Cloudfront if your budget supported it. Mostly because I don't want to maintain a server somewhere.
Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
141–150 of 199 posts
Re: Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
#142Cloudflare R2, egress is free. Storing 10TB would be about $150 a month.
Holy shit that's expensive
Re: Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
#143You could do this for about $1k/mo with Linode and Wasabi. For FastComments we store assets in Wasabi and have services in Linode that act as an in-memory+on disk LRU cache. We have terabytes of data but only pay $6/mo for Wasabi, because the cache hit ratio is high and Wasabi doesn't charge for egress until your egress is more than your storage or something like that. The rest of the cost is egress on Linode. The ni…
Actually, with Interserver and Wasabi you could probably get it under $100/mo.
You could just run Varnish with the S3 backend. Popular files will be cached locally on the server, and you'll pay a lot less for egress from Wasabi.
Re: Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
#144I would do it in S3 + Cloudfront if your budget supported it. Mostly because I don't want to maintain a server somewhere.
This is one way, but wouldn't be cost-effective bandwidth wise (which is OPs main concern).
Re: Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
#145Depends on what exactly you want to do with it. Hetzner has very cheap Storage boxes (10TB for $20/month with unlimited traffic) but those are closer to FTP boxes with a 10 connection limit. They are also down semi-regularly for maintenance. For rock-solid public hosting Cloudflare is probably a much better bet, but you're also paying 7 times the price. More than a dedicated server to host the files, but you get more…
Wow! It's the first time I'm hearing of this Hetzner offering. It's ideal for my offsite backup needs. Thanks!
I think rsync.net is much cheaper. They frequently run offers, I think I pay $40/yr for 2 TB or something.
Re: Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
#146Re: Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
#147One way I can think of is to serve files over BitTorrent with a (HTTP) web seed stored across blob stores at Cloudflare R2, Backblaze B2, Wasabi.
I briefly looked at services selling storage on FileCoin / IPFS and Chia, but couldn't find anything that inspired confidence.
Re: Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
#148Earlier quoted context omitted.
Storage Boxes are even cheaper, 20 EUR (or less if you’re outside Europe) for 10TB + unlimited bandwidth.
1gbit is 300T a month, 10g is 3000T a month. There’s always a limit, that might be measured in TB, PB or EB, and may be what you determine practical or not, but it’s there
I'm wonderinng if anyone even gets close to 1gbs from a Hetzner storage box
Re: Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
#149Earlier quoted context omitted.
> Hetzner has very cheap Storage boxes (10TB for $20/month with unlimited traffic) * based on fair use at 250 TB/mo: > In order to continue hosting your servers with us, the traffic use will need to be drastically reduced. Please check your servers and confirm what is using so much traffic, making sure it is nothing abusive, and then find ways of reducing it.
Thanks for finding this. Unlimited rarely is. Looks like backblaze (other post below) has a free bandwidth and cheap storage solution
Backblaze B2 is only free with Bandwidth Alliance partners, otherwise $0.01/GB, individual transactions will also cost you, depending on how much you use.
Re: Ask HN: Suggestions to host 10TB data with a monthly +100TB bandwidth
#150Depends on what exactly you want to do with it. Hetzner has very cheap Storage boxes (10TB for $20/month with unlimited traffic) but those are closer to FTP boxes with a 10 connection limit. They are also down semi-regularly for maintenance. For rock-solid public hosting Cloudflare is probably a much better bet, but you're also paying 7 times the price. More than a dedicated server to host the files, but you get more…
That's if you use their CDN. Cloudflare R2 doesn't charge for egress bandwidth. If you have 100TB/mo to serve, try it and see what happens. I haven't heard of anyone being kicked off of R2 for using too much egress bandwidth yet.
At scale, you'll pay a couple thousand dollars for Class B operations on R2, and another bunch for storing the 10 TB in the first place, but that's relatively cheap compared to other offerings where you'd pay for metered egress bandwidth.
https://developers.cloudflare.com/r2/pricing/ https://r2-calculator.cloudflare.com/