Live data from Hacker News

Garage: Open-Source Distributed Object Storage

garagehq.deuxfleurs.fr

41–50 of 147 posts

Re: Garage: Open-Source Distributed Object Storage

#41

Earlier quoted context omitted.

This is not intended for commercial services. Realistically, this software was made for people who keep servers in their basement. The security profile of LAN users is very different than public AWS.

The site says it was made (initially) and used for a commercial French hoster.

They’re a self-described “non-profit experimental hosting group”. It’s used to host their website, chat server data, etc.

It’s great they made it (I use personally!) but that’s more akin to a home-lab than commercial vendor.

Re: Garage: Open-Source Distributed Object Storage

#42
post #39

Whats the motivation behind project like this one? We got ceph, minio, seaweedfs ... and a dozen of others. I am genuinly curious what is the goal here?

Minio assumes each node has identical hardware. Garage is designed for use-cases like self-hosting, where nodes are not expected to have identical hardware.

Re: Garage: Open-Source Distributed Object Storage

#43
post #37

What I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IP…

Sounds a little like Kademlia, the DHT implementation that BitTorrent uses.

It's a distributed hash table where the value mapped to a hash is immutable after it is STOREd (at least in the implementations that I know)

Re: Garage: Open-Source Distributed Object Storage

#44

What is the difference between a "distributed object storage" and a file system?

There are few.

From the perspective of consistency guarantees, object storage gives fewer of such guarantees (this is seen as allowing implementations to be faster than typical file-systems). For example, since there isn't a concept of directories in object store, the implementation doesn't need to deal with the problems that arise while copying or moving directories with files open in those directories.

There are some non-storage functions that are performed only by filesystems, but not object storage. For example, suid bits.

It's also much more common to use object stores for larger chunks of data s.a. whole disk snapshots, VM images etc. While filesystems aim for the middle-size (small being RDBMs) s.a. text files you'd open in a text editor. Subsequently, they are optimized for these objectives. Filesystems care a lot about what happens when random small incremental and possibly overlapping updates happen to the same file, while object stores care about performance of sequential reads and writes the most.

This excludes the notion of "distributed" as both can be distributed (and in different ways). I suppose you meant to ask about the difference between "distributed object storage" and "distributed filesystem".

Re: Garage: Open-Source Distributed Object Storage

#45

I don’t understand why everyone wants to replicate AWS APIs for things that are not AWS. S3 is a horrible interface with a terrible lack of features. It’s just file storage without any of the benefits of a file syste - no metadata, no directory structures, no ability to search, sort, or filter. Combine that with high latency network file access and an overly verbose API. You literally have a bucket for storing files,…

It's a legitimate question and I'm glad you asked! (I'm not the author of Garage and have no affiliation).

Filesystems impose a lot of constraints on data-consistency that make things go slow. In particular, when it comes to mutating directory structure. There's also another set of consistency constraints when it comes to dealing with file's contents. Object stores relax or remove these constraints, which allows them to "go faster". You should, however, carefully consider if the constraints are really unnecessary for your case. The typical use-case for object stores is something like storing volume snapshots, VM images, layers of layered filesystems etc. They would perform poorly if you wanted to use them to store the files of your programming project, for example.

Re: Garage: Open-Source Distributed Object Storage

#46

I don’t understand why everyone wants to replicate AWS APIs for things that are not AWS. S3 is a horrible interface with a terrible lack of features. It’s just file storage without any of the benefits of a file syste - no metadata, no directory structures, no ability to search, sort, or filter. Combine that with high latency network file access and an overly verbose API. You literally have a bucket for storing files,…

Does your file system have search? Mine doesn’t. Instead I have software that implements search on top of it. Does it support filtering? Mine uses software on top again. Which an S3 api totally supports. Does your remote file server magically avoid network latency? Mine doesn’t. In case you didn’t know, inside the bucket you can use a full path for S3 files. So you can have directories or folders or whatever. Some be…

Be OS FS at least has this

https://en.m.wikipedia.org/wiki/Be_File_System

Re: Garage: Open-Source Distributed Object Storage

#47
post #37

What I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IP…

Have you seen https://github.com/willbryant/verm?

Re: Garage: Open-Source Distributed Object Storage

#48
We moved over to garage after running minio in production with about ~2PB after about 2 years of headache. Minio does not deal with small files very well, rightfully so, since they don't keep a separate index of the files other than straight on disk. While ssd's can mask this issue to some extent, spinning rust, not so much. And speaking of replication, this just works... Minio's approach even with synchronous mode turned on, tends to fall behind, and again small files will pretty much break it all together.

We saw about 20-30x performance gain overall after moving to garage for our specific use case.

Re: Garage: Open-Source Distributed Object Storage

#50
post #31
post #29

Earlier quoted context omitted.

Tried this for my own homelab, either I misconfigured it or it consumes x2(linearly) memory(working) of the stored data. So, for example, if I put 1GB of data, seaweed would immediately consume 2GB of memory constantly! Edit: memory = RAM

Are you claiming that SeaweedFS requires twice as much RAM as the sum of the sizes of the stored objects?

Correct. I experimented by varying the data volume, it was linearly correlated by x2 of data volume.
Post reply on HN