Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SeaweedFS fast distributed storage system for blobs, objects, files and datalake

github.com

Post reply on HN
Print view View original

51–60 of 128 posts

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#51

Post by SOLAR_FIELDS »

Already__Taken wrote: post #16
gorset wrote: post #2

I was asking around in my network after experience with self hosting S3 like solutions. One serious user of SeaweedFS recommended looking into min.io instead. Another serious user of min.io recommend looking into SeaweedFS instead…

It used to be if you wanted thousands of tiny files give seaweed a go, minio would suck. But minio has since had a revision so you'd have to test it out. Seaweed has been running my k8s persistent volumes pretty admirably for like a year for about 4 devs.

I set up seaweed on my home lab a few weeks ago and while it was a bit difficult to initially get everything configured it seems to work really well once I got it running. I’m using it with the CSI driver and have it incrementally backing up to s3 (I originally had it mirrored then realized for my use case I didn’t need to do that).

My one complaint is that I could not really get it to work with an S3 compatible api that wasn’t officially supported in the list of S3 backends, even though that should have been theoretically possible. I ended up picking a supported backend instead.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#53

Post by KingOfCoders »

Guthur wrote: post #47
osigurdson wrote: post #37

Earlier quoted context omitted.

>> and indeed in 2013, he cited the failure of WinFS as his greatest disappointment at Microsoft, Failing to capture any of the mobile handset market while missing out almost entirely on search and social media businesses would be higher on my list if I were in BG's shoes.

Microsoft has never been good in either consumer electronics or advertising (social media). MS carved out economic rent on business with Windows and Office, Apple actually failed at that.

"Gaming is now the third largest business at Microsoft." After Office and Azure, before Windows.

https://www.theverge.com/2024/1/30/24055445/microsoft-q2-202...

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#54

Post by maayank »

no_wizard wrote: post #32
bnewbold wrote: post #12

SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…

Written in Go no less, a GC language! I was expecting C/C++ or Rust, pleasantly surprised to see Go.

Why pleasantly surprised compared to Rust? What’s the significance of GCing?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#55

Post by spockz »

KingOfCoders wrote: post #53
Guthur wrote: post #47

Earlier quoted context omitted.

Microsoft has never been good in either consumer electronics or advertising (social media). MS carved out economic rent on business with Windows and Office, Apple actually failed at that.

"Gaming is now the third largest business at Microsoft." After Office and Azure, before Windows. https://www.theverge.com/2024/1/30/24055445/microsoft-q2-202...

Sure. But windows was/is the loss leader for that. No gaming or office without windows. Even Azure benefits from it as it ties into Active Directory with Azure AD. This makes it a completely integrated story. So that they are even still making money off of windows directly is just a direct benefit.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#56

Post by [deleted] »

pilgrim0 wrote: post #22
bnewbold wrote: post #12

SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…

I was quite surprised to discover that minio is one file per object. Having read some papers about object stores, this is definitely not what I expected.

[deleted]

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#57

Post by blr_lpm »

pilgrim0 wrote: post #22
bnewbold wrote: post #12

SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…

I was quite surprised to discover that minio is one file per object. Having read some papers about object stores, this is definitely not what I expected.

What are the pros/cons of storing one file per object? As a noob in this domain, this made sense to me.

It will be great if you can share name or reference of some papers around this. Thank you in advance.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#58

Post by XorNot »

blr_lpm wrote: post #57
pilgrim0 wrote: post #22

Earlier quoted context omitted.

I was quite surprised to discover that minio is one file per object. Having read some papers about object stores, this is definitely not what I expected.

What are the pros/cons of storing one file per object? As a noob in this domain, this made sense to me. It will be great if you can share name or reference of some papers around this. Thank you in advance.

I imagine very large objects you'd like to be able to shard across multiple servers.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#59

Post by anonzzzies »

seized wrote: post #24
bnewbold wrote: post #12

SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…

GarageS3 is a nice middle ground, it is not file on disk per object but it's simpler than SeaweedFS as well. https://garagehq.deuxfleurs.fr/

Yes, garage sourcecode is very easy to read and understand. Didn’t read seaweed yet.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#60

Post by anonzzzies »

maayank wrote: post #54
no_wizard wrote: post #32

Earlier quoted context omitted.

Written in Go no less, a GC language! I was expecting C/C++ or Rust, pleasantly surprised to see Go.

Why pleasantly surprised compared to Rust? What’s the significance of GCing?

A lot of people regard GCs as something one should not use for low level components like file systems and databases. So that this performs so well might be the surprise for GP.
Post reply on HN