Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I just want simple S3

blog.feld.me

Post reply on HN
Print view View original

101–110 of 130 posts

Re: I just want simple S3

#101

Post by bombcar »

convolvatron wrote: post #42
PunchyHamster wrote: post #38

S3 isn't "simple" tho. It doesn't need to care about POSIX mess but there is whole swathes of features many implementations miss or are incomplete, both on frontend side (serving files with right headers, or with right authentication) and backend (user/policy management, legal hold, versioning etc.) It gets even more messy when migrating, for example migrating your backups to garagefs will lose you versioning, which…

I just spent some time with the s3 protocol and I agree completely. What should have been able to leverage the simplifying assumptions turned into another hodgepodge. It’s not like nfs is a real shining example of simplicity either. I’ve never worked with p9, but potentially that aside I think we really failed to come up with a decent distributed file model,

If all the "popular" solutions are complex, it means the problem domain is complex.

You either are doomed to reimplement and rediscover the complexity on your own, or you change your requirements to fit a narrower problem domain to avoid (some of) the complexity.

Re: I just want simple S3

#102

Post by PxldLtd »

Oxodao wrote: post #99

I don't cet how people are developing app using s3 without actually having a local s3 server ? Everyone is like "just use aws / whatever trendy host atm" but while developping ? I use garage but the main issue i have is the mandatory random keys which means i cannot hardcode them in my compose file. For prod, sure you want that. In dev ? Absolutely not I want to be able to clone my repo, dc up -d and lets go. I also…

You can hard-code them. Just write a setup script that imports your desired keys from a .env file or something:

``` docker exec $CONTAINER $BIN key import --yes "$S3_ACCESS_KEY_ID" "$S3_SECRET_ACCESS_KEY" ```

Re: I just want simple S3

#103

Post by bombcar »

jeroenhd wrote: post #35
jerf wrote: post #15

I think we get a "S3 clone" about once every week or two on the Golang reddit. It strikes me as a classic case of "we need all the interested people to pull in one project, not each start their own". AI may have made this worse then ever.

I'm pretty sure I set up most of what "Simple S3" using with Apache2 and WebDAV at least fifteen years ago. Every month there's a post of "I just want a simple S3 server" and every single one of them has a different definition of "simple". The moment any project overlaps between the use cases of two "simple S3" projects, they're no longer "simple" enough. That's probably why hosted S3-like services will exist even if…

It's like "Word/Excel is too bloated, I just need a simple subset!" and each simple subset is subtly different.

Re: I just want simple S3

#104

Post by Havoc »

I was thinking similar to OP and decided to vibe code my own…which was quite a journey in itself.

Can’t say I super trust it so will probably roll it out only for low stakes stuff. It does pass the ceph S3 test suite though and is rust so in theory somewhat safe-ish…maybe.

Re: I just want simple S3

#105

Post by Havoc »

0xbadcafebee wrote: post #54

Call me crazy, but wouldn't 15 minutes on GLM 5.1 produce a working implementation? I haven't looked at the code, but a non-production-grade Go implementation can't be that complicated. Edit: Minio is written in Go, and is AGPL3... fork it (publicly), strip out the parts you don't want, run it locally.

I did exactly that. 15 mins for initial implementation was about right. And it seemed fine at first glance.

Then I decided to run the Ceph S3 test suite against it. So many issues. Think it passed 3 tests on first run out of I think about a hundred. Took another couple of hours to get it to a state that is even vaguely passable for non production use.

Got something vaguely workable but even after many hours I can’t say I super trust it

> implementation can't be that complicated.

S3 has a fair bit of bloat in spec that nobody seems to use and it’s not clearly delineated which is core and what’s ahem optional. I ended up relying on the LLM to figure out what’s core and that ended up missing stuff too so that needed a couple of feature iterations too

Re: I just want simple S3

#106

Post by ferdzo »

I wanted the same, so I made my own. Initially I started it without vibe coding, but as I wanted to add more features than the initial working S3 API, I decided it's not worth the time to not vibe simple API stuff. I use it for backups, for a Minio replacement in a Milvus database and other random stuff. Not bad for a week work. It's single node so far and not meant for distributed computing.

https://github.com/ferdzo/fs

Re: I just want simple S3

#110

Post by convolvatron »

bombcar wrote: post #101
convolvatron wrote: post #42

Earlier quoted context omitted.

I just spent some time with the s3 protocol and I agree completely. What should have been able to leverage the simplifying assumptions turned into another hodgepodge. It’s not like nfs is a real shining example of simplicity either. I’ve never worked with p9, but potentially that aside I think we really failed to come up with a decent distributed file model,

If all the "popular" solutions are complex, it means the problem domain is complex. You either are doomed to reimplement and rediscover the complexity on your own, or you change your requirements to fit a narrower problem domain to avoid (some of) the complexity.

I'm working on a distributed S3 cache that supports just two functions, pread style contents of a file and all the elements in a directory. I've worked on other systems that represents this entire structure as RDF triples, which leaves you with just query and insert. To come at it from another direction EFS only implements about 2/3 of NFSv4, and S3 was perfectly functional before they larded it up with all this policy stuff.

I'm not saying that there weren't reasons to add these functions to the protocol, but if your aim is minimalism, then you can do _much_ better, and I think there is a real benefit in having a bare bones protocol that anyone could implement in many contexts.

Post reply on HN