Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Garage: Open-Source Distributed Object Storage

garagehq.deuxfleurs.fr

Post reply on HN
Print view View original

101–110 of 147 posts

Re: Garage: Open-Source Distributed Object Storage

#101

Post by computerfan494 »

ianopolous wrote: post #97
computerfan494 wrote: post #5

I have used Garage for a long time. It's great, but the AWS sigv4 protocol for accessing it is just frustrating. Why can't I just send my API key as a header? I don't need the full AWS SDK to get and put files, and the AWS sigv4 is a ton of extra complexity to add to my projects. I don't care about the "security benefits" of AWS sigv4. I hope the authors consider a different authentication scheme so I can recommend G…

You can implement S3 V4 signatures in a few hundred lines of code. https://github.com/Peergos/Peergos/blob/master/src/peergos/s...

I have done this for my purposes, but it's slow and unnecessary bloat I wish I didn't have to have.

Re: Garage: Open-Source Distributed Object Storage

#102

Post by mdaniel »

skinkestek wrote: post #60
khimaros wrote: post #53

Earlier quoted context omitted.

you might be interested in https://github.com/perkeep/perkeep

Perkeep has (at least until last I checked it) the very interesting property of being completely impossible for me to make heads or tails of while also looking extremely interesting and useful. So in the hope of triggering someone to give me the missing link (maybe even a hyperlink) for me to understand it, here is a the situation: I'm a SW dev that also have done a lot of sysadmin work. Yes, I have managed to instal…

I was curious to see if I could help, and I wondered if you saw their mailing list? It seems to have some folks complaining about things they wish it did, which strangely enough is often a good indication of what it currently does

There's also "Show Parkeep"-ish posts like this one https://groups.google.com/g/perkeep/c/mHoUUcBz2Yw> where the user made their own Pocket implementation complete with original page snapshotting

The thing that most stood out to me was the number of folks who wanted to use Parkeep to manage its own content AND serve as the metadata system of record for external content (think: an existing MP3 library owned by an inflexible media player such as iTunes). So between that and your "import Twitter" comment, it seems one of its current hurdles is that the use case one might have for a system like this needs to be "all in" otherwise it becomes the same problem as a removable USB drive for storing stuff: "oh, damn, is that on my computer or on the external drive?"

Re: Garage: Open-Source Distributed Object Storage

#104

Post by rkunnamp »

j-pb wrote: post #37

What I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IP…

IPFS like "coordination free" local S3 replacement! Yes. That is badly needed.

Re: Garage: Open-Source Distributed Object Storage

#105

Post by ianopolous »

computerfan494 wrote: post #101
ianopolous wrote: post #97

Earlier quoted context omitted.

You can implement S3 V4 signatures in a few hundred lines of code. https://github.com/Peergos/Peergos/blob/master/src/peergos/s...

I have done this for my purposes, but it's slow and unnecessary bloat I wish I didn't have to have.

5 hmac-sha256's per signature are slow?

Re: Garage: Open-Source Distributed Object Storage

#107

Post by iscoelho »

vineyardmike wrote: post #20
klysm wrote: post #8

Earlier quoted context omitted.

Sending your api key in the header is equivalent to basic auth.

This is not intended for commercial services. Realistically, this software was made for people who keep servers in their basement. The security profile of LAN users is very different than public AWS.

You know FOSS software runs most of the internet right? (and, if you'll believe it, AWS internally)

I would find it completely unsurprising to see Garage used in some capacity by a Fortune 500 by the end of the year (not that they'd publicly say it).

Re: Garage: Open-Source Distributed Object Storage

#108

Post by iscoelho »

WhereIsTheTruth wrote: post #94
neon_me wrote: post #39

Whats the motivation behind project like this one? We got ceph, minio, seaweedfs ... and a dozen of others. I am genuinly curious what is the goal here?

performance, therefore cheaper

not just about cost! improved performance/latency can make workloads that previously required a local SSD/NVME to be actually able run to run on distributed storage or an object store.

it can not be understated how slow Ceph/Minio/etc can be compared to local NVME. there is plenty of room for improvement.

Re: Garage: Open-Source Distributed Object Storage

#109

Post by iscoelho »

icy wrote: post #35

I've been running this on K3s at home (for my website and file server) and it's been very well behaved: https://git.icyphox.sh/infra/tree/master/apps/garage I find it interesting that they chose CRDTs over Raft for distributed consensus.

from an operations point of view, I am surprised anyone likes Raft. I have yet to see any application implement Raft in a way that does not spectacularly fail in production and require manual intervention to resolve.

CRDTs do not have the same failure scenarios and favor uptime over consistency.

Re: Garage: Open-Source Distributed Object Storage

#110

Post by zimbatm »

makkesk8 wrote: post #48

We moved over to garage after running minio in production with about ~2PB after about 2 years of headache. Minio does not deal with small files very well, rightfully so, since they don't keep a separate index of the files other than straight on disk. While ssd's can mask this issue to some extent, spinning rust, not so much. And speaking of replication, this just works... Minio's approach even with synchronous mode t…

That's very cool; I didn't expect Garage to scale that well while being so young.

Are there other details you are willing/allowed to share, like the number of objects in the store and the number of servers you are balancing them on?

Post reply on HN