We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data. Seaweed had much better performance for our use case.
Forgive my ignorance but why is this preferable to a big ZFS pool?
SeaweedFS fast distributed storage system for blobs, objects, files and datalake
11–20 of 128 posts
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#12SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited).
The docs and operational tooling feel a bit janky at first, but they get the job done, and the whole project is surprisingly feature-rich. I've dealt with basic power-outages, hardware-caused data corruption (cheap old SSDs), etc, and it was possible to recover.
In some ways I feel like the surprising thing is that there is such a gap in open source S3 API blob stores. Minio is very simple and great, but is one-file-per-object on disk (great for maybe 90% of use-cases, but not billions of thumbnails). Ceph et al are quite complex. There are a bunch of almost-sort-kinda solutions like base64-encoded bytes in HBase/postgresql/etc, or chunking (like MongoDB), but really you just want to concatenate the bytes like a .tar file, and index in with range requests.
The Wayback Machine's WARC files plus CDX (index files with offset/range) is pretty close.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#13We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data. Seaweed had much better performance for our use case.
Forgive my ignorance but why is this preferable to a big ZFS pool?
Not the only reason, but we have a distributed workload so HTTP is a better protocol than NFS for our use case.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#14I was asking around in my network after experience with self hosting S3 like solutions. One serious user of SeaweedFS recommended looking into min.io instead. Another serious user of min.io recommend looking into SeaweedFS instead…
If your looking for more recommendations, try Garage ( https://garagehq.deuxfleurs.fr/ ), which is on my short list to try in my home lab...
Longhorn is another that I see quite a lot, next to Ceph/Rook and lately SeaweedFS.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#15We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data. Seaweed had much better performance for our use case.
Do you wish it supported Erasure Coding for lower disk usage, or is your workload such that the extra spindles from replication are useful?
That would be nice and that’s why we first tried MinIO.
But with MinIO and erasure coding a single PUT results in more IOPS and we saw lower performance.
Also, expanding MinIO must be done in increments of your original buildout which is annoying. So if you start with 4 servers and 500TB, they recommend you expand by adding another 4 servers with 500TB at least.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#16I was asking around in my network after experience with self hosting S3 like solutions. One serious user of SeaweedFS recommended looking into min.io instead. Another serious user of min.io recommend looking into SeaweedFS instead…
It used to be if you wanted thousands of tiny files give seaweed a go, minio would suck. But minio has since had a revision so you'd have to test it out.
Seaweed has been running my k8s persistent volumes pretty admirably for like a year for about 4 devs.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#17We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data. Seaweed had much better performance for our use case.
why not ceph?
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#18Earlier quoted context omitted.
If your looking for more recommendations, try Garage ( https://garagehq.deuxfleurs.fr/ ), which is on my short list to try in my home lab...
Longhorn is another that I see quite a lot, next to Ceph/Rook and lately SeaweedFS. https://github.com/longhorn/longhorn
Longhorn claims to be a block storage solution so more like EBS than S3.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#19Does anyone know how well the Seaweed Filer works for renaming files or missing files? My use case’s involves writing a lot of data to temporary files that are then renamed to their final name. This is always the Achilles heel for distributed file storage, where files are put into buckets based on the file path… when you rename the path, but keep the data, lookups become more complicated.
(This is HPC work with large processing pipelines. I keep track of if the job was successful based upon if the final file exists. The rename only happens if the job was successful. It’s a great way to track pipeline status, but metadata lookups can be a pain — particularly for missing files. )
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#20What would be the best S3 like storage software with user based access and limits that I can locally host?