101–110 of 199 posts
I would also like to ask everyone about suggestions for deep storage of personal data, media etc. 10TB with no need for access unless in case of emergency data loss. I'm currently using S3 intelligent tiering.
Tarsnap.
I can't justify colo unless I can get 10U for $300/month with 2kW of PDU, 1500 kWh, and 1 GbE uncapped.
You could do this for about $1k/mo with Linode and Wasabi. For FastComments we store assets in Wasabi and have services in Linode that act as an in-memory+on disk LRU cache. We have terabytes of data but only pay $6/mo for Wasabi, because the cache hit ratio is high and Wasabi doesn't charge for egress until your egress is more than your storage or something like that. The rest of the cost is egress on Linode. The ni…
> You could do this for about $1k/mo with Linode and Wasabi. This is still crazy expensive. Cloud providers have really warped people’s expectations.
Backups probably wouldn't be much more.
That reminds me of the entertaining "I just want to serve 5 terabytes. Why is this so difficult?" video that someone made inside Google. It satirizes the difficulty of getting things done at production scale. https://www.youtube.com/watch?v=3t6L-FlfeaI
Medium- and smaller-scale can often be more flexible because they don't have to incur the pain of nonuniformity as scale increases. While they may not be able to afford optimizations or discounts with larger, standardized purchases, they can provide more personalized services large scale cannot hope to provide.
If it fits your model, WebTorrent[0] can offload a lot of bandwidth to peers. [0] https://github.com/webtorrent/webtorrent
At some point you still need a seed for that 10TB of data with some level of reliability. WebTorrent only solves the monthly bandwidth iff you've got some high capacity seeds (your servers or long-term peers).
One multi-vendor, zero-knowledge, HA solution is https://tahoe-lafs.org/trac/tahoe-lafs
S3 is probably the highest quality. It’s enterprise grade : fast, secure with a lot of tiers and controls. If you recover only small data, it’s also not expensive. The only problem is if you recover large data. That would be a major problem.
Depends on what exactly you want to do with it. Hetzner has very cheap Storage boxes (10TB for $20/month with unlimited traffic) but those are closer to FTP boxes with a 10 connection limit. They are also down semi-regularly for maintenance. For rock-solid public hosting Cloudflare is probably a much better bet, but you're also paying 7 times the price. More than a dedicated server to host the files, but you get more…
you can definitely do this at home on the cheap. As long as you have a decent internet connection, that is ;) 10TB+ harddisks are not expensive, you can put them in an old enclosure together with a small industrial or NUC PC in your basement
I tried this using Pine ROCKPro64 to possibly install Ceph across 2-5 RAID1 NAS enclosures. The problem is I can't get any of their dusty Linux forks to recognize the storage controller, so they're $200 paperweights.
I wrote a SATA HDD "top" utility that brings in data from SMART, mdadm, lvm, xfs, and the Linux SCSI layer. I set monitoring to look for elevated temperature, seek errors, scan errors, reallocation counts, offline reallocation, and probational count.
Assuming the simplest need is making files available :
1) Sync.com provides unlimited hosting and file sharing from it.
Sync is a decent Dropbox replacement with a few more bells and whistles.
2) BackBlaze business let’s you deliver files for free via their CDN. $5/TB per month storage plus free egress via their CDN.
https://www.backblaze.com/b2/solutions/developers.html
Backblaze seems to be 70-80% cheaper than S3 as it claims.
Traditional best practice cloud paths are optimized to be a best practice to generate profit for the cloud provider.
Luckily it’s nice to rarely be alone or the first to have a need.