SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#1

Post by thushanfernando » Fri, Feb 02, 2024, 10:57 PM UTC

SeaweedFS fast distributed storage system for blobs, objects, files and datalake
github.com

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#2

Post by gorset » Fri, Feb 02, 2024, 11:10 PM UTC

I was asking around in my network after experience with self hosting S3 like solutions. One serious user of SeaweedFS recommended looking into min.io instead. Another serious user of min.io recommend looking into SeaweedFS instead…

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#3

Post by junon » Fri, Feb 02, 2024, 11:12 PM UTC

gorset wrote: post #2

I was asking around in my network after experience with self hosting S3 like solutions. One serious user of SeaweedFS recommended looking into min.io instead. Another serious user of min.io recommend looking into SeaweedFS instead…

Sounds like you should try both and write an article!

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#4

Post by 4by4by4 » Fri, Feb 02, 2024, 11:13 PM UTC

We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data.

Seaweed had much better performance for our use case.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#5

Post by jasonjayr » Fri, Feb 02, 2024, 11:14 PM UTC

gorset wrote: post #2

I was asking around in my network after experience with self hosting S3 like solutions. One serious user of SeaweedFS recommended looking into min.io instead. Another serious user of min.io recommend looking into SeaweedFS instead…

If your looking for more recommendations, try Garage ( https://garagehq.deuxfleurs.fr/ ), which is on my short list to try in my home lab...

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#6

Post by erikaww » Fri, Feb 02, 2024, 11:16 PM UTC

4by4by4 wrote: post #4

We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data. Seaweed had much better performance for our use case.

Any hiccups?

Drop in S3 compatibility with much better performance would be insane

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#7

Post by DrDroop » Fri, Feb 02, 2024, 11:20 PM UTC

I there any reason to use something like this instead of S3 or similar products when you are not running your own infra?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#8

Post by 4by4by4 » Fri, Feb 02, 2024, 11:20 PM UTC

erikaww wrote: post #6

4by4by4 wrote: post #4

We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data. Seaweed had much better performance for our use case.

Any hiccups? Drop in S3 compatibility with much better performance would be insane

Setup is a little obscure, but the developer is responsive on Slack and GH.

We are only a couple months in and haven’t had to add to our cluster yet, storing about 250TB, so it’s still early for us. Promising so far and hardware has already paid for itself.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#9

Post by throwup238 » Fri, Feb 02, 2024, 11:25 PM UTC

DrDroop wrote: post #7

I there any reason to use something like this instead of S3 or similar products when you are not running your own infra?

If you’re running it on AWS? Probably not.

Otherwise: egress costs.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#10

Post by Scaevolus » Fri, Feb 02, 2024, 11:32 PM UTC

4by4by4 wrote: post #4

We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data. Seaweed had much better performance for our use case.

Do you wish it supported Erasure Coding for lower disk usage, or is your workload such that the extra spindles from replication are useful?