Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SeaweedFS fast distributed storage system for blobs, objects, files and datalake

github.com

Post reply on HN
Print view View original

121–128 of 128 posts

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#121

Post by [deleted] »

snthpy wrote: post #107
papaver-somnamb wrote: post #65

Tried and rejected SeaweedFS due to Postgres failing to even initialize itself on a POSIX FS volume mounted over SeaweedFS' CSI driver. And that's too bad, because SeaweedFS was otherwise working well! What we need and haven't identified yet is an SDS system that provides both fully-compliant POSIX FS and S3 volumes, is FOSS, a production story where individuals can do all tasks competently/quickly/effectively (manag…

What about JuiceFS? I've never used it myself and just learned about it from this thread but it seems to fit the bill.

[deleted]

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#122

Post by ddorian43 »

magicalhippo wrote: post #102
arccy wrote: post #80

Earlier quoted context omitted.

running something like postgres over a networked filesystem sounds very wrong

There was some work done to add a S3 storage backend for ZFS[1], precisely with the goal of running PosgreSQL on effectively external storage. A key point was to effectively treat S3 as a huge, reliable disk with 10MB "sectors". So the bucket would contain tons of 10MB chunks and ZFS would let S3 handle the redundancy. For performance it was coupled with a large, local SSD-based write-back cache. Sadly it seems the c…

Neon does this for PostgreSQL and it's open source (more like code-dump though)

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#123

Post by KaiserPro »

nh2 wrote: post #116
KaiserPro wrote: post #108

Earlier quoted context omitted.

VFX engineer, I have suffered through: _early_ lustre (its much better now) GPFS Gluster (fuck that) clustered XFS (double fuck that) Isilon Nowadays, a single 2u server can realistically support 2x 100gig nics at full bore. So the biggest barrier is density. You can probably get 1pb in a rack now, and linking a bunch of jbods(well NVMEs) is probably easily to do now.

"1PB in a rack"? You can apparently already buy 2.5PB in a single 4U server: https://www.techradar.com/pro/seagate-has-launched-a-massive...

sorry I should have added a caveat of 1pb _at decent performance_

That seagate array will be fine for streaming (so long as you spread the data properly) as soon as you start mixing read/write loads on that, it'll start to chug. You can expect 70-150iops out of each drive, and thats a 60 drive array (from guess, you can get 72 drives in a 4u, but they are less maintainable, well used to be, thing might have improved recently)

When I was using luster with ultra scisi (yes, that long ago) we had good 10-20 racks to get to 100tb, that could sustain 1gigabyte a second.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#124

Post by nh2 »

KaiserPro wrote: post #123
nh2 wrote: post #116

Earlier quoted context omitted.

"1PB in a rack"? You can apparently already buy 2.5PB in a single 4U server: https://www.techradar.com/pro/seagate-has-launched-a-massive...

sorry I should have added a caveat of 1pb _at decent performance_ That seagate array will be fine for streaming (so long as you spread the data properly) as soon as you start mixing read/write loads on that, it'll start to chug. You can expect 70-150iops out of each drive, and thats a 60 drive array (from guess, you can get 72 drives in a 4u, but they are less maintainable, well used to be, thing might have improved…

Agreed, it depends on the use case. For some "more storage" is all that matters, for others you don't want to be bottlenecked on getting it into / out of the machine or through processing.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#125

Post by vdm »

pilgrim0 wrote: post #22
bnewbold wrote: post #12

SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…

I was quite surprised to discover that minio is one file per object. Having read some papers about object stores, this is definitely not what I expected.

This has not been true since 2021. https://blog.min.io/minio-optimizes-small-objects/

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#126

Post by cchance »

CyberDildonics wrote: post #117
chrislusf wrote: post #114

Earlier quoted context omitted.

> I really don't understand why you aren't eager to explain the differences and what problems are being solved. Sorry, everybody has different background of knowledge. Hard to understand where the question comes from. I think https://www.usenix.org/system/files/fast21-pan.pdf may be helpful here.

Sorry, everybody has different background of knowledge. Hard to understand where the question comes from. They were straightforward questions. The paper you linked talks about blobs as a term for appending to files. Mostly it seems to be about wrapping and replicating XFS. Is that why you are avoiding talking about specifics? Are you wrapping XFS?

I'm little confused why people are being so weird with the OP, asking what the difference between a blob and a file aren't something for seaweedfs lol, Blobs and Files, and other terms are terms used to describe different layers of data allocation in almost every modern object storage solution.

Blobs are what lie under files, you can have a file split into multiple blobs spread across different drives, or different servers etc, and then can put it back together into a file when requested, thats how i understand it at a basic level

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#127

Post by nh2 »

nh2 wrote: post #118
chrislusf wrote: post #40

Thanks for sharing! I work on SeaweedFS. SeaweedFS is built on top of a blob storage based on Facebook's Haystack paper. The features are not fully developed yet, but what makes it different is a new way of programming for the cloud era. When needing some storage, just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block. There will be more features built…

First, the feature set you have built is very impressive. I think SeaweedFS would really benefit from more documentation on what exactly it does. People who want to deploy production systems need that, and it would also help potential contributors. Some examples: * It says "optimised for small files", but it is not super clear from the whitepaper and other documentation what that means. It mostly talks about about ho…

I posted this on https://github.com/seaweedfs/seaweedfs/discussions/5290

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#128

Post by CyberDildonics »

cchance wrote: post #126
CyberDildonics wrote: post #117

Earlier quoted context omitted.

Sorry, everybody has different background of knowledge. Hard to understand where the question comes from. They were straightforward questions. The paper you linked talks about blobs as a term for appending to files. Mostly it seems to be about wrapping and replicating XFS. Is that why you are avoiding talking about specifics? Are you wrapping XFS?

I'm little confused why people are being so weird with the OP, asking what the difference between a blob and a file aren't something for seaweedfs lol, Blobs and Files, and other terms are terms used to describe different layers of data allocation in almost every modern object storage solution. Blobs are what lie under files, you can have a file split into multiple blobs spread across different drives, or different s…

I think they are being weird. According to the facebook pdf they linked I think that would fall under chunks, but either way, why would someone advertise a filesystem for 'blob' storage when users don't interact with that? According to the paper 'blobs' are sent to append to files, but that isn't really 'blob storage', it's just giving a different name for an operation that's been done since the 70s - appending to a networked file. No one would say 'this filesystem can store all your file appends' and no one is storing discreet serialized data without a name and once you do that, you have a file.

They also seem like they are being vague and patronizing to avoid admitting that their product is not a unique filesystem, but just something to distribute XFS.

Post reply on HN