51–60 of 81 posts
A very good alternative is seaweedfs https://github.com/chrislusf/seaweedfs/ based on facebook haystack paper (efficient small files) & more.
Did you pull that URL out of some blog post or something? The accurate URL is https://github.com/seaweedfs/seaweedfs
Just finished installing it on my OpenIndiana NAS to replace Minio. Biggest difference so far is that Minio is just files on disk, Garage chunks all files and has a metadata db. Minios listing operations were horribly slow, still have to see if Garage resolves that.
> Biggest difference so far is that Minio is just files on disk, Garage chunks all files and has a metadata db. I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so. I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before I went over…
That's exactly what I did recently on new work: migrated blobs from DB to S3. It significantly reduced load from the servers (and will reduce more, right now the implementation is primitive - just proxying S3, using URL will allow other services to deal with S3 directly). It solved backup nightmare (those people couldn't do backup because their server run out of space every month). I'll admit that backup issue is more like admin incompetence but I work with what I get. Having database shrink from 200GB to 80MB now allows to backup/restore it in seconds rather than hours.
I didn't find any issues with S3 approach. Even transactions solved by a tiny possibility of leaving junk in S3 which is a non-issue. Just upload all data to S3 before commit and delete if commit fails (and if commit fails and delete fails, so be it).
Earlier quoted context omitted.
> I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so. Honestly, I'd expect the exact opposite. Filesystems are really good at storing files. Why not leverage all that work? > I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before…
> Filesystems are really good at storing files. Why not leverage all that work? As an asterisk, the S3 API is key-value pairs, not files; that distinction comes up a lot when interacting with Amazon S3, and I would expect the same with an S3 API clone. For example, ListObjects[1] has a "delimiter" that (AFAIK) defaults to / making it appear to be a filesystem but using "." or "!" would be a perfectly fine delimiter a…
It looks similar to minio, which as also an AGPL single binary that implements the S3 API. However Minio is written in Go and Garage is in Rust. I'd love to see a detailed comparison.
For this use-case, I like JuiceFS better. * https://juicefs.com/en/ * https://github.com/juicedata/juicefs I am not affiliated with them, just a regular user.
Earlier quoted context omitted.
> Filesystems are really good at storing files. Why not leverage all that work? As an asterisk, the S3 API is key-value pairs, not files; that distinction comes up a lot when interacting with Amazon S3, and I would expect the same with an S3 API clone. For example, ListObjects[1] has a "delimiter" that (AFAIK) defaults to / making it appear to be a filesystem but using "." or "!" would be a perfectly fine delimiter a…
Why is it useful?
Now you can `ListObjects(Prefix="/some-path/my.jar", Delimiter="!")` to get the "interior files" back.
I'm sure there are others, that's just one that I could think of off the top of my head. Mapping a URL and its interior resources would be another (`"https://example.com\t/script[1]", "console.log('hello, world')")`
Further fun fact that even I didn't know until searching for other examples: "delimiter" is a string and thus can be `Delimiter=unknown` or such: https://github.com/aws/aws-sdk-go/issues/2130
1: see the ListObject page under "encoding-type"
Kind of topic. But I’m wondering if it’s possible to get comparable pricing when self-hosting a solution similar to this in the cloud vs say AWS S3.
Yes. Specially S3 which is way overpriced. I have setup already a multi-region minio cluster at the cost of $0.0039/GB. The only "cloud" solution that could get closer to that is Storj. [0] [0]: https://www.storj.io/
I remember working at a company that got started before cloud took off. They used mogile fs which I recently found at https://mogilefs.github.io/mogilefs-docs/HighLevelOverview.h... but I never hear about anyone else using it. It wasn't as stable as S3 but it was okay, I guess. Does anyone else here remember that distributed open source file system?
Brad Fitzpatrick was (still is?) living in the future.