Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SeaweedFS fast distributed storage system for blobs, objects, files and datalake

github.com

Post reply on HN
Print view View original

81–90 of 128 posts

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#81

Post by jamesblonde »

papaver-somnamb wrote: post #65

Tried and rejected SeaweedFS due to Postgres failing to even initialize itself on a POSIX FS volume mounted over SeaweedFS' CSI driver. And that's too bad, because SeaweedFS was otherwise working well! What we need and haven't identified yet is an SDS system that provides both fully-compliant POSIX FS and S3 volumes, is FOSS, a production story where individuals can do all tasks competently/quickly/effectively (manag…

You do know that you cannont implement a fully-compliant POSIX FS with only the S3 API? None of the scalalbe SDS' support random writes. Atomic rename (for building transactional systems like lakehouse table formats) is not there. Listing of files is often eventually consistent. The closest functional API to a posix-compliant one in scalable SDS' is the HDFS API. Only ADLS supports that. But then again, they are the only one who enable you to fuse mount a directory for local FS read/write access. All of the S3 fuse mount stuff is fundamentally limited by the S3 API.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#83

Post by jakjak123 »

hardwaresofton wrote: post #45
jakjak123 wrote: post #27

Have used SeaweedFS to store billions of thumbnails. The tooling is a bit clunky, but it mostly works. The performance is very good for small-ish objects (memory usage + latency), and latency remains consistently good into 99.9 percentiles. We had some issues with data loss and downtime, but that was mostly our own fault.

What issues did you run into? Not settling replication?

This was a couple of years ago now, but for example, some very minor amount of objects had not been replicated at all. This happened during heavy concurrent write traffic, and a couple of these race condition-ish bugs have been fixed over the years.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#84

Post by KaiserPro »

Things to make sure of when choosing your distributed storage:

1) are you _really_ sure you need it distributed, or can you shard it your self? (hint, distributed anything sucks at least one if not two innovation tokens, if you're using other innovation tokens as well. you're going to have a very bad time)

2) do you need to modify blobs, or can you get away with read/modify/replace? (s3 doesn't support partial writes, one bit change requires the whole file to be re-written)

3) whats your ratio of reads to writes (do you need local caches or local pools in gpfs parlance)

4) How much are you going to change the metadata (if theres posix somewhere, it'll be a lot)

5) Are you going to try and write to the same object at the same time in two different locations (how do you manage locking and concurrency?)

6) do you care about availability, consistency or speed? (pick one, maybe one and a half)

7) how are you going to recover from the distributed storage shitting it's self all at the same time

8) how are you going to control access?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#85

Post by sighansen »

I don't understand why you wouldn't just use plain s3. There is no comparison in the readme and I would love to understand what the benefits are. Also I would have expected a comparison to maybe Apache Iceberg, but this might be more specialized for relational data lake data?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#86

Post by SheddingPattern »

KaiserPro wrote: post #84

Things to make sure of when choosing your distributed storage: 1) are you _really_ sure you need it distributed, or can you shard it your self? (hint, distributed anything sucks at least one if not two innovation tokens, if you're using other innovation tokens as well. you're going to have a very bad time) 2) do you need to modify blobs, or can you get away with read/modify/replace? (s3 doesn't support partial writes…

Sounds like you are talking from experience. Are you storage specialist, how did you learn so much about this?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#87

Post by zeeZ »

stevekemp wrote: post #66
jdthedisciple wrote: post #62

Sounds great! Now I only need to wait 10 years until all the hidden but crucial bugs are found (at the massive loss of real data, ofc) before I'm ready to use it, like with every new piece of technology... Or what should give me the confidence that it isn't so?

This is an old project, I had a quick look and see that I submitted a pull-request back in 2015: https://github.com/seaweedfs/seaweedfs/pull/187

First commit in the Google code repo seems to be 2011-11-30

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#88

Post by ImPostingOnHN »

chrislusf wrote: post #49
CyberDildonics wrote: post #42

Earlier quoted context omitted.

what makes it different is a new way of programming for the cloud era. just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block. How is that not mmap? Also what is the difference between a file, an object, a blob, a filesystem and an object store? Is all this just files indexed with sql?

> How is that not mmap? The allocated storage is append only. For updates, just allocate another blob. The deleted blobs would be garbage collected later. So it is not really mmap. > Also what is the difference between a file, an object, a blob, a filesystem and an object store? The answer would be too long to fit here. Maybe chatgpt can help. :) > Is all this just files indexed with sql? Sort of yes.

I, too, am interested in your views on the last 2 questions, since your views, not chatGPT's, are what informed the design. Part of learning from others' designs [0] is understanding what the designers think about their own design, and how they came about it.

Would you mind elaborating on them? HN gives a lot of space, and I'm confident you can find a way to summarize without running out, or sounding dismissive (which is what the response kind of sounds like now).

0 – https://aosabook.org/en/

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#89

Post by thrusong »

I'm a small user— only about 250,000 objects in storage and a lot of those cold storage behind Cloudflare, but I've been using SeaweedFS for years.

I think since v0.7— I was always intrigued by Facebook's Haystack.

SeaweedFS been super reliable, efficient, and trouble free.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#90

Post by flemhans »

KaiserPro wrote: post #84

Things to make sure of when choosing your distributed storage: 1) are you _really_ sure you need it distributed, or can you shard it your self? (hint, distributed anything sucks at least one if not two innovation tokens, if you're using other innovation tokens as well. you're going to have a very bad time) 2) do you need to modify blobs, or can you get away with read/modify/replace? (s3 doesn't support partial writes…

1) only if it removes a "janitor" token of nannying the servers. Right now I just have one big server with a big 160TB ZFS pool, but it's running out.

2) No modifications, just new files and the occasional deletion request.

3) Almost just 1 write and 1 read per file, this is a backing storage for the source files, and they are cached in front.

4) Never

5) Files are written only by one other server, and there will be no parallel writes.

6) I pick consistency and as the half, availability.

7) This happened something like 15 years ago with MogileFS and thus scared us away. (Hence the single-server ZFS setup).

8) Reads are public, writes restricted to one other service that may write.

Post reply on HN