Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Storage FUSE

cloud.google.com

Post reply on HN
Print view View original

71–80 of 112 posts

Re: Google Cloud Storage FUSE

#71

Post by qwertox »

crazygringo wrote: post #35
MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

This seems overly pessimistic to me. Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app. But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend fo…

I agree. For example if you want to use Google's ASR (Automated Speech Recognition), if your file is longer than 1 minute in duration, you first need to upload it to a bucket, which is a lot of added complexity compared to a regular HTTP POST.

Just copying the file to a mounted bucket would make this a lot easier.

Then again, how does one get the metadata of the uploaded file?

Re: Google Cloud Storage FUSE

#72

Post by ashishbijlani »

vamega wrote: post #70
ashishbijlani wrote: post #46

Earlier quoted context omitted.

I'm working on optimizing FUSE using eBPF (ExtFUSE [1]) and adding a caching layer exactly as you mentioned. Will post publicly when ready. 1. https://github.com/extfuse/extfuse

Is work on this continuing (or restarting)? I had heard of this a few years ago, but thought the project was shelved.

The project is active (just not merged in the kernel yet). Please DM me for questions.

Re: Google Cloud Storage FUSE

#73

Post by milesward »

Please, listen to me: use this only in extremely limited cases where performance, stability, and cost efficiency are not paramount. An object store is not a file system no matter how hard you bludgeon it.

Re: Google Cloud Storage FUSE

#74

Post by retrocryptid »

isanjay wrote: post #68
retrocryptid wrote: post #67

Earlier quoted context omitted.

In the old days, we had a system called NFS (Network File System) where, yes, you may decide to use only remote disks. There were several advantages apart from lowering the cost of disks, mainly that you could centrally manage boot images for a fleet of machines. Then we got the web and everyone seemed to assume you could do the same thing over the internet. I agree with you, I would prefer a local disk to one with 1…

I still use NFS in my home.

I do too. It just works. Though I boot off a local drive.

Re: Google Cloud Storage FUSE

#75

Post by throwawaaarrgh »

jefftk wrote: post #9
throwawaaarrgh wrote: post #6

Earlier quoted context omitted.

pretty standard limitation of object storage services iirc

Doesn't this mean that most programs you might want to use with the FUSE API won't actually work? They'll do fine for a while, until they try to seek, and then they'll get an error? Or is there a large group of programs that only ever write sequentially?

well yeah, but there's a lot of things FUSE makes easier. no need to implement a client library, no need to write some custom wrapper or rsync thing to sync files to the bucket or bucket to local system, etc. it won't work for every app but for the ones it does support it saves a ton of extra work and maintenance.

Re: Google Cloud Storage FUSE

#77

Post by ofek »

I do appreciate that Google is now officially supporting gcsfuse because it genuinely is a great project. However, their Kubernetes CSI driver seems to have in large part copied code from the one I and a co-maintainer have been working on for years:

- https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver

- https://github.com/ofek/csi-gcs

Here is the initial commit: https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/c...

Notice for example not just the code but also the associated files. In the Dockerfile it blatantly copied the one from my repo, even the dual license I chose because I was very into Rust at the time. Or take a look at the deployment examples which use Kustomize which I like but is very uncommon and most Kubernetes projects provide Helm charts instead.

They were most certainly aware of the project because Google reached out to discuss potential collaboration but never responded back: https://imgur.com/a/KDuf9mj

Re: Google Cloud Storage FUSE

#78

Post by SamuelAdams »

ofek wrote: post #77

I do appreciate that Google is now officially supporting gcsfuse because it genuinely is a great project. However, their Kubernetes CSI driver seems to have in large part copied code from the one I and a co-maintainer have been working on for years: - https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver - https://github.com/ofek/csi-gcs Here is the initial commit: https://github.com/GoogleCloudPlatform/gcs-fuse…

Your repository seems to have both an Apache and MIT license. What license are you distributing your code under?

Edit: I see you said it’s dual licensed. From the look of it both allow Google or any other company to copy and reuse code, so what are you upset about?

Re: Google Cloud Storage FUSE

#79

Post by mox1 »

ofek wrote: post #77

I do appreciate that Google is now officially supporting gcsfuse because it genuinely is a great project. However, their Kubernetes CSI driver seems to have in large part copied code from the one I and a co-maintainer have been working on for years: - https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver - https://github.com/ofek/csi-gcs Here is the initial commit: https://github.com/GoogleCloudPlatform/gcs-fuse…

You licensed the code as MIT - https://github.com/ofek/csi-gcs/blob/master/LICENSE-MIT

Are you saying you have an issue with them copying your MIT licensed code?

Re: Google Cloud Storage FUSE

#80

Post by ofek »

SamuelAdams wrote: post #78
ofek wrote: post #77

I do appreciate that Google is now officially supporting gcsfuse because it genuinely is a great project. However, their Kubernetes CSI driver seems to have in large part copied code from the one I and a co-maintainer have been working on for years: - https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver - https://github.com/ofek/csi-gcs Here is the initial commit: https://github.com/GoogleCloudPlatform/gcs-fuse…

Your repository seems to have both an Apache and MIT license. What license are you distributing your code under? Edit: I see you said it’s dual licensed. From the look of it both allow Google or any other company to copy and reuse code, so what are you upset about?

Either, the choice is up to you.

edit: as I express in a sibling comment this act is legally allowed of course, but is bad practice

Post reply on HN