Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Storage FUSE

cloud.google.com

Post reply on HN
Print view View original

41–50 of 112 posts

Re: Google Cloud Storage FUSE

#41

Post by MontyCarloHall »

VikingCoder wrote: post #36
MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

Uh, how does it perform from a Google Compute Engine Virtual Machine? If it performs well there, I could imagine that being pretty useful.

That is exactly where I tested it, and the latency was still abysmally poor (~1 second per file operation).

I don’t even want to know how bad the latency would be outside of a cloud VM.

Re: Google Cloud Storage FUSE

#43

Post by surajrmal »

curt15 wrote: post #19
askvictor wrote: post #12

Now for official Google Drive support on Linux...

How do Googlers access Google Drive from their Linux workstations? Do they have an internal GDrive client?

Through the web frontend. I'm not aware of any special fuse clients, nor is it particularly appealing. All files I store in it are for web based applications (primarily gsuite). We have alternative collosus based file share mounts which we can use for "native" files. I personally use git and/or rsync to share files between my various corp devices (laptop, cloud vm, desktop) in addition to those other options.

Re: Google Cloud Storage FUSE

#45

Post by ninkendo »

crazygringo wrote: post #35
MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

This seems overly pessimistic to me. Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app. But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend fo…

The problem is that such systems have a habit of growing in scope until they reach a point where you really do need the more optimal access patterns of using the real HTTP APIs, and the inefficiencies of emulating the full filesystem API will gradually start to bite you. Maybe you’re lucky enough that that won’t happen, but it’s important to understand it for the quick hack job it is, IMO.

Re: Google Cloud Storage FUSE

#46

Post by ashishbijlani »

MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

I'm working on optimizing FUSE using eBPF (ExtFUSE [1]) and adding a caching layer exactly as you mentioned. Will post publicly when ready.

1. https://github.com/extfuse/extfuse

Re: Google Cloud Storage FUSE

#47

Post by KingOfCoders »

plaidfuji wrote: post #27
curt15 wrote: post #19

Earlier quoted context omitted.

How do Googlers access Google Drive from their Linux workstations? Do they have an internal GDrive client?

I wonder the same, but I also wonder what the actual use case is for the Drive app on Linux. For me, Drive is mostly for syncing office docs (namely MS-office docs), PDFs and images among teams. That type of work doesn’t lend itself well to a Linux env anyway. And for programming-heavy sync tasks, a user will more likely use a remote Git repo for code and GCS for data. Does google even use MS office internally?

I write my articles in Markdown and would want to switch to a terminal based Linux (Pi Zero, low power, e-paper, distraction free) to do this instead of a GUI.

(I currently use Goland and Scrivener to write articles and books)

Re: Google Cloud Storage FUSE

#48

Post by e1g »

remram wrote: post #31

See also: JuiceFS: https://juicefs.com/ Adds a DBMS or key-value store for metadata, making the filesystem much faster (POSIX, small overwrites don't have to replace a full object in the GCS/S3 backend). Almost certainly a better solution if you want to turn your object storage into a mountable filesystem, with the (big) caveat that you can't access the files directly in the bucket (they are not stored transparently)…

JuiceFS is mostly POSIX compatible, but there are important caveats such as no extended ACL, copying files changes their mtime (impacts backup tools), it offers "close-to-open" consistency (dangerous for log appenders), etc.

Choosing an appropriate solution in this space still depends on what you need to do with the storage, and a few other options are MooseFS (https://github.com/moosefs/moosefs) SeaweedFS (https://github.com/seaweedfs/seaweedfs) Curve (https://github.com/opencurve/curve) GeeseFS (https://github.com/yandex-cloud/geesefs)

Re: Google Cloud Storage FUSE

#49

Post by lazide »

ninkendo wrote: post #45
crazygringo wrote: post #35

Earlier quoted context omitted.

This seems overly pessimistic to me. Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app. But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend fo…

The problem is that such systems have a habit of growing in scope until they reach a point where you really do need the more optimal access patterns of using the real HTTP APIs, and the inefficiencies of emulating the full filesystem API will gradually start to bite you. Maybe you’re lucky enough that that won’t happen, but it’s important to understand it for the quick hack job it is, IMO.

In most situations that time is years, decades, or ‘never’. Which is fine.

Not everyone or everything scales faster than bandwidth and/or CPU is.

Re: Google Cloud Storage FUSE

#50

Post by lazide »

markstos wrote: post #39
MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

I once evaluated using s3fuse for managing about 36 million images. The old storage model was on a filesystem so it was supposed to make a smooth transition to the cloud. AWS Premium Support wisely advised me against it, not just because of latency but also because the abstraction makes /far/ more API calls then a native solution would. After a bit of testing to confirm, I switched to using native API calls. That cod…

FUSE adapters in general are not for a product/production use in my experience. They’re great for one off convenience use, or basic admin scripts.
Post reply on HN