Garage: Open-Source Distributed Object Storage • Hacker News Board

Re: Garage: Open-Source Distributed Object Storage

#71

Post by nh2 » Sat, Jul 20, 2024, 12:13 PM UTC

Scaevolus wrote: post #28

fijiaarone wrote: post #3

I don’t understand why everyone wants to replicate AWS APIs for things that are not AWS. S3 is a horrible interface with a terrible lack of features. It’s just file storage without any of the benefits of a file syste - no metadata, no directory structures, no ability to search, sort, or filter. Combine that with high latency network file access and an overly verbose API. You literally have a bucket for storing files,…

S3 exposes effectively all the metadata that POSIX APIs do, in addition to all the custom metadata headers you can add. Implementing a filesystem versus an object store involves severe tradeoffs in scalability and complexity that are rarely worth it for users that just want a giant bucket to dump things in. The API doesn't matter that much, but everything already supports S3, so why not save time on client libraries…

There are many POSIX APIs that S3 does not cover. For example directories, and thus efficient renames and atomic moves of sub hierarchies.

Re: Garage: Open-Source Distributed Object Storage

#72

Post by seaghost » Sat, Jul 20, 2024, 12:15 PM UTC

I want something very simple to run locally that has s3 compatibility just for the dev work and testing. Any recommendations?

Re: Garage: Open-Source Distributed Object Storage

#73

Post by amluto » Sat, Jul 20, 2024, 12:19 PM UTC

j-pb wrote: post #37

What I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IP…

I would settle for first-class support for object hashes. Let an object have metadata, available in the inventory, that gives zero or more hashes of the data. SHA256, some Blake family hash, and at least one decent tree hash should be supported. There should be a way to ask the store to add a hash to an existing object, and it should work on multipart objects.

IOW I would settle for content verification even without content addressing.

S3 has an extremely half-hearted implementation of this for “integrity”.

Re: Garage: Open-Source Distributed Object Storage

#74

Post by zmj » Sat, Jul 20, 2024, 12:20 PM UTC

seaghost wrote: post #72

I want something very simple to run locally that has s3 compatibility just for the dev work and testing. Any recommendations?

https://hub.docker.com/r/localstack/localstack

Re: Garage: Open-Source Distributed Object Storage

#75

Post by rlonstein » Sat, Jul 20, 2024, 12:21 PM UTC

seaghost wrote: post #72

I want something very simple to run locally that has s3 compatibility just for the dev work and testing. Any recommendations?

https://min.io/

Re: Garage: Open-Source Distributed Object Storage

#76

Post by withinboredom » Sat, Jul 20, 2024, 12:25 PM UTC

sandGorgon wrote: post #54

makkesk8 wrote: post #48

We moved over to garage after running minio in production with about ~2PB after about 2 years of headache. Minio does not deal with small files very well, rightfully so, since they don't keep a separate index of the files other than straight on disk. While ssd's can mask this issue to some extent, spinning rust, not so much. And speaking of replication, this just works... Minio's approach even with synchronous mode t…

quick question for advice - we have been evaluating minio for a in-house deployed storage for ML data. this is financial data which we have to comply on a crap ton of regulations. so we wanted lots of compliance features - like access logs, access approvals, short lived (time bound) accesses, etc etc. how would you compare garage vs minio on that front ?

You will probably put a proxy in front of it, so do your audit logging there (nginx ingress mirror mode works pretty good for that)

Re: Garage: Open-Source Distributed Object Storage

#77

Post by j-pb » Sat, Jul 20, 2024, 12:27 PM UTC

singinwhale wrote: post #43

j-pb wrote: post #37

What I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IP…

Sounds a little like Kademlia, the DHT implementation that BitTorrent uses. It's a distributed hash table where the value mapped to a hash is immutable after it is STOREd (at least in the implementations that I know)

Kademlia could certainly be a part of a solution to this, but it's a long road from the algorithm to the binary that you can start on a bunch of machines to get the service, e.g. something like SeaweedFS. BitTorrent might actually be the closest thing we have to this, but it is at the opposite spectrum of the latency -distributed axis.

Re: Garage: Open-Source Distributed Object Storage

#78

Post by j-pb » Sat, Jul 20, 2024, 12:29 PM UTC

BageDevimo wrote: post #47

j-pb wrote: post #37

What I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IP…

Have you seen https://github.com/willbryant/verm ?

Yeah, the subdirectories and mime-type seemed like an unnecessary complication. Also looks pretty dead.

Re: Garage: Open-Source Distributed Object Storage

#79

Post by compressedgas » Sat, Jul 20, 2024, 12:36 PM UTC

j-pb wrote: post #37

What I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IP…

You might also be interested in Tahoe-LAFS https://www.tahoe-lafs.org/

Re: Garage: Open-Source Distributed Object Storage

#80

Post by jiggawatts » Sat, Jul 20, 2024, 12:46 PM UTC

j-pb wrote: post #37

What I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IP…

Something related that I've been thinking about is that there aren't many popular data storage systems out there that use HTTP/3 and/or gRPC for the lower latency. I don't just mean object storage, but database servers too.

Recently I benchmarked the latency to some popular RPC, cache, and DB platforms and was shocked at how high the latency was. Every still talks about 1 ms as the latency floor, when it should be the ceiling.