Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SeaweedFS fast distributed storage system for blobs, objects, files and datalake

github.com

Post reply on HN
Print view View original

101–110 of 128 posts

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#101

Post by tempest_ »

Already__Taken wrote: post #16
gorset wrote: post #2

I was asking around in my network after experience with self hosting S3 like solutions. One serious user of SeaweedFS recommended looking into min.io instead. Another serious user of min.io recommend looking into SeaweedFS instead…

It used to be if you wanted thousands of tiny files give seaweed a go, minio would suck. But minio has since had a revision so you'd have to test it out. Seaweed has been running my k8s persistent volumes pretty admirably for like a year for about 4 devs.

Lets say instead of 1000s I need to store billions.

So far I have been testing with seaweed and it seems to chug along fine at around ~4B files and it is still increasing.

Has minio improved on that lately ?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#102

Post by magicalhippo »

arccy wrote: post #80
papaver-somnamb wrote: post #65

Tried and rejected SeaweedFS due to Postgres failing to even initialize itself on a POSIX FS volume mounted over SeaweedFS' CSI driver. And that's too bad, because SeaweedFS was otherwise working well! What we need and haven't identified yet is an SDS system that provides both fully-compliant POSIX FS and S3 volumes, is FOSS, a production story where individuals can do all tasks competently/quickly/effectively (manag…

running something like postgres over a networked filesystem sounds very wrong

There was some work done to add a S3 storage backend for ZFS[1], precisely with the goal of running PosgreSQL on effectively external storage.

A key point was to effectively treat S3 as a huge, reliable disk with 10MB "sectors". So the bucket would contain tons of 10MB chunks and ZFS would let S3 handle the redundancy. For performance it was coupled with a large, local SSD-based write-back cache.

Sadly it seems the company behind this figured it needed to keep this closed-source in order to get ROI[2].

[1]: https://youtu.be/opW9KhjOQ3Q

[2]: https://github.com/openzfs/zfs/issues/12119

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#103

Post by chrislusf »

CyberDildonics wrote: post #97
chrislusf wrote: post #49

Earlier quoted context omitted.

> How is that not mmap? The allocated storage is append only. For updates, just allocate another blob. The deleted blobs would be garbage collected later. So it is not really mmap. > Also what is the difference between a file, an object, a blob, a filesystem and an object store? The answer would be too long to fit here. Maybe chatgpt can help. :) > Is all this just files indexed with sql? Sort of yes.

You made the claim: what makes it different is a new way of programming for the cloud era. but you aren't even explaining how anything is different from what a normal file system can do, let alone what makes it a "new way of programming for the cloud era".

Sorry it was not so clear. Previously fallocate just allocate disk space for a local server. Now SeaweeedFS can allocate a blob on a remote storage.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#104

Post by CyberDildonics »

chrislusf wrote: post #103
CyberDildonics wrote: post #97

Earlier quoted context omitted.

You made the claim: what makes it different is a new way of programming for the cloud era. but you aren't even explaining how anything is different from what a normal file system can do, let alone what makes it a "new way of programming for the cloud era".

Sorry it was not so clear. Previously fallocate just allocate disk space for a local server. Now SeaweeedFS can allocate a blob on a remote storage.

What is the difference between a blob and a file and what is the difference between allocating a blob on remote storage or a file on remote storage?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#105

Post by pilgrim0 »

blr_lpm wrote: post #57
pilgrim0 wrote: post #22

Earlier quoted context omitted.

I was quite surprised to discover that minio is one file per object. Having read some papers about object stores, this is definitely not what I expected.

What are the pros/cons of storing one file per object? As a noob in this domain, this made sense to me. It will be great if you can share name or reference of some papers around this. Thank you in advance.

The other commenter already outlined the main trade-offs, which boils down to increased latency and storage overhead for the file-per-object model. As for papers, I like the design of Haystack.

https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#106

Post by mdaniel »

seized wrote: post #24
bnewbold wrote: post #12

SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…

GarageS3 is a nice middle ground, it is not file on disk per object but it's simpler than SeaweedFS as well. https://garagehq.deuxfleurs.fr/

One will want to be cognizant that Garage, like recent MinIO releases, is AGPL https://git.deuxfleurs.fr/Deuxfleurs/garage/src/tag/v0.9.1/L...

I'm not trying to start trouble, only raising awareness because in some environments such a thing matters

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#107

Post by snthpy »

papaver-somnamb wrote: post #65

Tried and rejected SeaweedFS due to Postgres failing to even initialize itself on a POSIX FS volume mounted over SeaweedFS' CSI driver. And that's too bad, because SeaweedFS was otherwise working well! What we need and haven't identified yet is an SDS system that provides both fully-compliant POSIX FS and S3 volumes, is FOSS, a production story where individuals can do all tasks competently/quickly/effectively (manag…

What about JuiceFS?

I've never used it myself and just learned about it from this thread but it seems to fit the bill.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#108

Post by KaiserPro »

SheddingPattern wrote: post #86
KaiserPro wrote: post #84

Things to make sure of when choosing your distributed storage: 1) are you _really_ sure you need it distributed, or can you shard it your self? (hint, distributed anything sucks at least one if not two innovation tokens, if you're using other innovation tokens as well. you're going to have a very bad time) 2) do you need to modify blobs, or can you get away with read/modify/replace? (s3 doesn't support partial writes…

Sounds like you are talking from experience. Are you storage specialist, how did you learn so much about this?

VFX engineer, I have suffered through:

_early_ lustre (its much better now)

GPFS

Gluster (fuck that)

clustered XFS (double fuck that)

Isilon

Nowadays, a single 2u server can realistically support 2x 100gig nics at full bore. So the biggest barrier is density. You can probably get 1pb in a rack now, and linking a bunch of jbods(well NVMEs) is probably easily to do now.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#109

Post by KaiserPro »

flemhans wrote: post #90
KaiserPro wrote: post #84

Things to make sure of when choosing your distributed storage: 1) are you _really_ sure you need it distributed, or can you shard it your self? (hint, distributed anything sucks at least one if not two innovation tokens, if you're using other innovation tokens as well. you're going to have a very bad time) 2) do you need to modify blobs, or can you get away with read/modify/replace? (s3 doesn't support partial writes…

1) only if it removes a "janitor" token of nannying the servers. Right now I just have one big server with a big 160TB ZFS pool, but it's running out. 2) No modifications, just new files and the occasional deletion request. 3) Almost just 1 write and 1 read per file, this is a backing storage for the source files, and they are cached in front. 4) Never 5) Files are written only by one other server, and there will be…

GPFS is pretty sexy nowadays, although its really expensive: https://www.ibm.com/products/storage-scale

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#110

Post by chrislusf »

CyberDildonics wrote: post #104
chrislusf wrote: post #103

Earlier quoted context omitted.

Sorry it was not so clear. Previously fallocate just allocate disk space for a local server. Now SeaweeedFS can allocate a blob on a remote storage.

What is the difference between a blob and a file and what is the difference between allocating a blob on remote storage or a file on remote storage?

A large file can be chunked into blobs.
Post reply on HN