Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I just want simple S3

blog.feld.me

Post reply on HN
Print view View original

111–120 of 130 posts

Re: I just want simple S3

#111

Post by 0xbadcafebee »

Havoc wrote: post #105
0xbadcafebee wrote: post #54

Call me crazy, but wouldn't 15 minutes on GLM 5.1 produce a working implementation? I haven't looked at the code, but a non-production-grade Go implementation can't be that complicated. Edit: Minio is written in Go, and is AGPL3... fork it (publicly), strip out the parts you don't want, run it locally.

I did exactly that. 15 mins for initial implementation was about right. And it seemed fine at first glance. Then I decided to run the Ceph S3 test suite against it. So many issues. Think it passed 3 tests on first run out of I think about a hundred. Took another couple of hours to get it to a state that is even vaguely passable for non production use. Got something vaguely workable but even after many hours I can’t s…

Would you be willing to share the result on the web somewhere? Would save a few tokens (and hours) for others who end up doing the same

Re: I just want simple S3

#112

Post by crabique »

I, too, can vouch for ZFS+VersityGW being a great solution, we were able to scale it vertically on a single-node deployment to a pretty high throughput, both reads and writes.

Most notably, the PutObject operation (which had always been a pain in the ass on HDD with MinIO) is performing well now, even with many small objects.

There is a natural synergy in the gateway storing its metadata in xattrs and using ZFS special VDEV with dnode_size=auto to store the entire ZFS+S3 metadata on fast media.

The latency impacts of the gateway itself can be further minimized by running multiple instances of the gateway pinned to CPU cores, all behind a HAProxy load-balancer communicating with them over UDS.

Re: I just want simple S3

#113

Post by dddw »

runlevel1 wrote: post #91

`rclone serve s3` is actually a thin wrapper around this: https://github.com/rclone/gofakes3 That repo is a fork of this project: https://github.com/johannesboyne/gofakes3 They bill it as being for testing, but it works great if all you want is a no-fuss S3-compatible API on top of a filesystem. I've run it on my NAS for a few years now to provide a much faster transfer protocol compared to SMB.

Rclone is amazing for data-management. S3 indeed just works fine with it.

Re: I just want simple S3

#115

Post by jkrejcha »

crote wrote: post #76
thayne wrote: post #75

Earlier quoted context omitted.

Simpler than it is now, but the authentication system was never simple. You can't just put a bearer token in the authorization header, you have to follow a complicated algorithm to sign the request. That made some sense 20 years ago when s3 didn't use tls to protect against a mitm that changed the changed the request. It is less valuable now when you use tls.

Isn't the whole signed request part a hard requirement for 3rd party access? S3 was originally designed for web use, which means giving not-fully-trusted browsers some access to your storage bucket. You can't exactly do "this client is allowed to download this one specific file for the next 24 hours (but not use it like their personal CDN and share the link with everyone)" or "this client is allowed to upload a singl…

Aside from the fact that you can do this statelessly (you could stuff a JWT or moral equivalent in the header as a Bearer token), if you were okay with server-side state, you can have the token be related to a principal or have permissions in of itself.

Re: I just want simple S3

#116

Post by bonesss »

thayne wrote: post #73
keyle wrote: post #45

I just need something that can do S3 and is reliable and not slow. Oh, simply that. I'm a simple man, I just need edge delivered cdn content that never fails and responds within 20ms.

I don't think that is what they are looking for. They just want something with an s3 compatible API they can run on their local network or maybe even on the same host.

So, why not write to a shared wrapper/facade?

If you split the interaction API out to an interface detailing actual program interaction with the service, then write an s3 backend and an FS backend. Then you could plug in any backend as desired and write agnostic application code.

Personally I end up there anyways testing and specifying the third party failure modes.

Re: I just want simple S3

#117

Post by jen20 »

thayne wrote: post #75
jen20 wrote: post #70

Earlier quoted context omitted.

It was simple(ish) 20 years ago, to be fair.

Simpler than it is now, but the authentication system was never simple. You can't just put a bearer token in the authorization header, you have to follow a complicated algorithm to sign the request. That made some sense 20 years ago when s3 didn't use tls to protect against a mitm that changed the changed the request. It is less valuable now when you use tls.

> a complicated algorithm to sign the request.

There are lots of benefits to this over the bearer token approach that many take. Binding the authentication to the specific request is valuable regardless of TLS.

Re: I just want simple S3

#118

Post by thayne »

bonesss wrote: post #116
thayne wrote: post #73

Earlier quoted context omitted.

I don't think that is what they are looking for. They just want something with an s3 compatible API they can run on their local network or maybe even on the same host.

So, why not write to a shared wrapper/facade? If you split the interaction API out to an interface detailing actual program interaction with the service, then write an s3 backend and an FS backend. Then you could plug in any backend as desired and write agnostic application code. Personally I end up there anyways testing and specifying the third party failure modes.

What if you need it because you are using a third party application that requires an s3 api? Or you want to test your code that interacts with an s3 API?

Re: I just want simple S3

#120

Post by skrtskrt »

syabro wrote: post #87
thayne wrote: post #73

Earlier quoted context omitted.

I don't think that is what they are looking for. They just want something with an s3 compatible API they can run on their local network or maybe even on the same host.

what's the point then? Just api around FS?

At this point S3 is an API spec more than a particular system. Plenty of things only work against the S3 API spec since the implementations have become such popular and relatively cheap and performant storage systems. It gives a nice limited surface area that doesn't allow you to do things that can get too complex or can vary too much across filesystems, etc.
Post reply on HN