Live data from Hacker News

Alternatives to MinIO for single-node local S3

rmoff.net

91–100 of 112 posts

Re: Alternatives to MinIO for single-node local S3

#93

Does anyone here have experience using some of the newer S3 family API features like locking with any of the self-hostable options? After reading about celld's Litestream + locking on S3 approach ( https://github.com/denoland/celld ) I got curious about that style of deployment as a way to consolidate small tools so their state is just centralize S3 buckets, but from a bit of reading support for the features like loc…

Its not locking, it needs consistent conditional PUT. Minio did have that, Rustfs does, Garage is eventually consistent so explicitly does not support it correctly.

Ah! Thanks. For anyone else following this is documented along side the replication and failover behavior here:

https://celld.dev/docs/guarantees/

Re: Alternatives to MinIO for single-node local S3

#94
post #11

I did try seaweedfs ~half a year ago and I had issues with setting up users through terraform module (if I recall correctly, users endpoint were not correctly responding on delete), and lack of S3 expiry rules (you needed to use seaweedfs configuration or API for that). Other than that, I was pretty happy with it.

Thanks! I am sure most of the bugs are fixed now. SeaweedFS now should have the most complete S3 API support now.

(I work on SeaweedFS)

Re: Alternatives to MinIO for single-node local S3

#95

Running Sentry on prem, I had to swap out their default SeaweedFS setup for Garage because the former kept failing under concurrent writes. I tried digging around for a bit, but found that the swap was easier and faster, Garage has also worked great for single node use cases (e.g. tested up to around 10 TB of data). I still think that SeaweedFS is a cool project, might have been a config issue or something, wasn’t wo…

SeaweedFS performance should be the fastest. Please create an issue if there are any performance issues with SeaweedFS, or you see it is slower than any alternatives.

(I work on SeaweedFS)

Re: Alternatives to MinIO for single-node local S3

#96

I had previously used seaweedfs, which has more moving parts but seems more feature-rich. It was overkill for a homelab, so I went to garage. Garage is very stable, but the metadata engine / file MUST live on NVMe. Otherwise it gets slow on small files. It also lacks features like per bucket retention settings and stuff. It really is "just" a S3 bucket.

The trouble I had with SeaweedFS was that the docs I found were rather sparse. Garage, on the other hand seemed pretty well documented.

Did you miss the wiki or was the wiki not enough? Please let me know. I work on SeaweedFS.

Re: Alternatives to MinIO for single-node local S3

#97
post #30

I heard good news about SeaweedFS

It's nice, I am just afraid of the pace Chris Lu can maintain with LLM assistance. Too many features are added in each new release, no stabilization time at all.

Right. It takes full time to work on this. But I got help from a co-founder now.

Actually, AI does help a lot to setup all variations of testing scenarios and uncover edge case bugs. It's more helpful than vibes.

And you do need to be systematic to examine AI generated code. There are no shortcut for it.

Re: Alternatives to MinIO for single-node local S3

#98
I've dabbled in both Garage and RustFS. Both are good but IMO flawed.

Garage is good but has weird tooling and a terrible searchable term. You'll receive lots of garage doors and electrical gate results when searching for issues with it.

The biggest problem with Garage is the metadata on an sqlite. If you have a single file and that sqlite corrupts then all your data is unaddressable and treated as pile of anonymous blocks. If you have multiple servers you can rebuild the metadata but if you have only one server you REALLY need to make sure that file is backed up.

So if you use S3 as a backup you have to keep a backup of your backup.

RustFS is closer to Minio, but I hate the way they log stuff. They basically use journald as a metric database dumping json formatted lines about library calls and how long they took. Its hard to make sense of those logs. Its the kind of log not meant to be human readable but to render on a dashboard.

Why not just publish these as a prometheus endpoint?

Also high CPU usage for no reason. I also noticed RustFS constantly trashing disk and keeping 10% CPU usage while looping some health check I have no idea how to disable (I've tried RUSTFS_SCANNER_ENABLED=false and others with no results). And of course lots of json metrics on journald.

I personally prefer garage because my backup box sit at my office and I like not needing to listen to HDD sounds all day every day while Rustfs loops its health check.

Re: Alternatives to MinIO for single-node local S3

#99
Start a ready-to-use SeaweedFS S3 object store:

AWS_ACCESS_KEY_ID=admin AWS_SECRET_ACCESS_KEY=secret S3_BUCKET=my-bucket ./weed mini -dir=./data

That's it.

The S3 endpoint is at http://localhost:8333, my-bucket exists, and admin/secret are valid credentials.

It also starts an Iceberg REST catalog and S3 Table Buckets for Iceberg and Lance tables, with table maintenance jobs.

Post reply on HN