Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Garage: Open-Source Distributed Object Storage

garagehq.deuxfleurs.fr

Post reply on HN
Print view View original

131–140 of 147 posts

Re: Garage: Open-Source Distributed Object Storage

#131

Post by dtag00 »

A bit of an off-topic question: I would like to programmatically generate S3 credentials that allow only read access or r/w access to only a certain set of prefixes. Imagine something like "Dropbox": You have a set of users, each user has his own prefix, but also users want to be able to share certain prefixes with other users. (Users are managed externally in a Postgres DB - MinIO does currently not know about them).

I found this really difficult to achieve with MinIO, since this appears to require an AssumeRole request, which is almost not documented in any way and I did not find a Typescript example. Additionally, there's a weird set of restrictions in place for MinIO (and also AWS) that makes this really difficult to do, e.g. the size of policies is limited, which effectively limits the number of prefixes a user can share. I found this really difficult to work around.

Can anyone suggest a way to do this? Can garage do this? Am I just approaching this from the wrong side?

Thanks

Re: Garage: Open-Source Distributed Object Storage

#132

Post by withinboredom »

Too wrote: post #126
withinboredom wrote: post #99

Earlier quoted context omitted.

Hmm... maybe??? If you have a central audit log, what is the probability that whatever gets implemented in all the open (and closed) source projects will be compatible?

Log scrapers are decoupled from applications. Just log to disk and let the agent of your logging stack pick it up and send to the central location.

That isn't an audit log.

Re: Garage: Open-Source Distributed Object Storage

#133

Post by psychoslave »

dddw wrote: post #130
psychoslave wrote: post #46

Earlier quoted context omitted.

Be OS FS at least has this https://en.m.wikipedia.org/wiki/Be_File_System

So haiku has it?

Certainly, though to be transparent I never used it. That’s great that it exits, and kudo to their team anyway.

Re: Garage: Open-Source Distributed Object Storage

#134

Post by snthpy »

j-pb wrote: post #37

What I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IP…

Have a look at LakeFS (https://docs.lakefs.io/understand/architecture.html).

Files are stored by hash on S3. Metadata is stored in a database. I run it locally and access it just like an S3 store. Metadata is in a Postgres DB.

Re: Garage: Open-Source Distributed Object Storage

#137

Post by acdha »

vlovich123 wrote: post #127
acdha wrote: post #70

Earlier quoted context omitted.

Oh, I’m definitely not saying integration or compatibility have nothing to do with it - only that “horrible interface with a terrible lack of features” seems impossible to reconcile with its immense popularity.

First mover + ecosystem wins over interface. And also, I really don’t have so many issues with the interface as others seem to and we had to implement it from scratch for R2. There’s features we extended on top of S3 so it’s not really the interface so much but rather that Amazon doesn’t really invest in adding features so much. I suspect it’s really hard for them to do so. We also exposed a native JS API that was mo…

They weren’t the first mover on either object storage or storage over HTTP.

Re: Garage: Open-Source Distributed Object Storage

#138

Post by vlovich123 »

acdha wrote: post #137
vlovich123 wrote: post #127

Earlier quoted context omitted.

First mover + ecosystem wins over interface. And also, I really don’t have so many issues with the interface as others seem to and we had to implement it from scratch for R2. There’s features we extended on top of S3 so it’s not really the interface so much but rather that Amazon doesn’t really invest in adding features so much. I suspect it’s really hard for them to do so. We also exposed a native JS API that was mo…

They weren’t the first mover on either object storage or storage over HTTP.

They were first mover for cloud storage and combining object storage with HTTP. Previous attempts were WebDAV (not object storage and very complicated to the point that kk one implemented) and Hadoop which didn’t have HTTP and couldn’t scale like S3’s design.

Re: Garage: Open-Source Distributed Object Storage

#139

Post by SOLAR_FIELDS »

yencabulator wrote: post #125
SOLAR_FIELDS wrote: post #121

Earlier quoted context omitted.

Can you point me towards resources that help me understand the trade offs being implied here? I feel like there is a ton of knowledge behind your statement that flies right past me because I don’t know the background behind why the things you are saying are important.

It's a huge field, basically distributed computing, burdened here with the glorious purpose of durable data storage. Any introductory text long enough becomes essentially a university-level computer science course. RADOS is the underlying storage protocol used by Ceph ( https://ceph.com/ ). Ceph is a distributed POSIX-compliant (very few exceptions) filesystem project that along the way implemented simpler things suc…

I upvoted this but I also wanted to say as well that this summary is valuable for me to gain a better groundwork for an undoubtedly complex topic. Thank you for the additional context.

Re: Garage: Open-Source Distributed Object Storage

#140

Post by bluepuma77 »

TristanBall wrote: post #119
bluepuma77 wrote: post #112

Can it be easily deployed with old-school Docker Swarm?

I don't think I have ever personally felt older than having someone describe anything docker related as "old-school"

That’s was not my intention!

Docker is young and fashionable, every windows script kiddy uses it nowadays!

And then comes to the Docker forum complaining about strange issues, not realizing Docker Desktop is a different product, it uses a Linux VM to run the Docker engine, which was build for Linux ;-)

I explicitly wrote "old-school Docker Swarm", as that is missing love for years and everyone with 2 IT FTEs seems to be moving to k8s.

Post reply on HN