Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Storage FUSE

cloud.google.com

Post reply on HN
Print view View original

61–70 of 112 posts

Re: Google Cloud Storage FUSE

#61

Post by vrosas »

MontyCarloHall wrote: post #53
crazygringo wrote: post #35

Earlier quoted context omitted.

This seems overly pessimistic to me. Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app. But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend fo…

Applications for which filesystem-like access is important (i.e. requiring lots of POSIX file I/O system calls, e.g. read(2)/write(2)/lseek(2)) but latency is unimportant seem pretty niche to me. If you don't need any of the POSIX syscalls, it's not that much more difficult to work with bucket URLs vs. file paths — the general format is the same, i.e. slash-delimited file/directory hierarchies.

Not everything is a webserver. There's a lot of software out there that wouldn't expect files to exist anywhere else besides on disk, and it's not worth fetching them all from cloud storage before you begin working on the data. It's easier just to GCSFuse a bucket to a VM and let the user do what they will. Works great for ad-hoc analysis of poorly or unstructured data.

Re: Google Cloud Storage FUSE

#62

Post by remram »

e1g wrote: post #48
remram wrote: post #31

See also: JuiceFS: https://juicefs.com/ Adds a DBMS or key-value store for metadata, making the filesystem much faster (POSIX, small overwrites don't have to replace a full object in the GCS/S3 backend). Almost certainly a better solution if you want to turn your object storage into a mountable filesystem, with the (big) caveat that you can't access the files directly in the bucket (they are not stored transparently)…

JuiceFS is mostly POSIX compatible, but there are important caveats such as no extended ACL, copying files changes their mtime (impacts backup tools), it offers "close-to-open" consistency (dangerous for log appenders), etc. Choosing an appropriate solution in this space still depends on what you need to do with the storage, and a few other options are MooseFS ( https://github.com/moosefs/moosefs ) SeaweedFS ( https:…

Those other ones can't use GCS though, which is why I pointed to JuiceFS (at least the ones I recognize).

Re: Google Cloud Storage FUSE

#63

Post by [deleted] »

ninkendo wrote: post #45
crazygringo wrote: post #35

Earlier quoted context omitted.

This seems overly pessimistic to me. Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app. But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend fo…

The problem is that such systems have a habit of growing in scope until they reach a point where you really do need the more optimal access patterns of using the real HTTP APIs, and the inefficiencies of emulating the full filesystem API will gradually start to bite you. Maybe you’re lucky enough that that won’t happen, but it’s important to understand it for the quick hack job it is, IMO.

[deleted]

Re: Google Cloud Storage FUSE

#64

Post by MontyCarloHall »

vrosas wrote: post #61
MontyCarloHall wrote: post #53

Earlier quoted context omitted.

Applications for which filesystem-like access is important (i.e. requiring lots of POSIX file I/O system calls, e.g. read(2)/write(2)/lseek(2)) but latency is unimportant seem pretty niche to me. If you don't need any of the POSIX syscalls, it's not that much more difficult to work with bucket URLs vs. file paths — the general format is the same, i.e. slash-delimited file/directory hierarchies.

Not everything is a webserver. There's a lot of software out there that wouldn't expect files to exist anywhere else besides on disk, and it's not worth fetching them all from cloud storage before you begin working on the data. It's easier just to GCSFuse a bucket to a VM and let the user do what they will. Works great for ad-hoc analysis of poorly or unstructured data.

And for your use case, the latency is not a concern? I suppose that would be true if you were mostly dealing with really big files and only cared about reading large contiguous chunks of them, but I would consider this a fairly niche application.

In my use case, taking ~1 second each time to `ls` a directory, `stat` a file, or `lseek` within a file was simply unacceptable. This was on a cloud VM, so the latency would be at its absolute minimum.

Re: Google Cloud Storage FUSE

#65

Post by retrocryptid »

This has been a thing for a while; I remember using it (or something like it) several years ago. While it's great for random files you might want to place in the G-Cloud, what I really wanted was to access my google docs content from the Linux command line. And you can do that, it's just that they're in non-obvious, non-documented, frequently changing formats that will only ever be usable with Google Docs.

But if you're using the google cloud like you might use Box.Net or DropBox, it seems fine for light usage.

Re: Google Cloud Storage FUSE

#66

Post by renewiltord »

MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

Catfs is not super production (there are some small changes you need to make in inode handling), but you can do this. We have it on top of goofys. They both need a few changes to work under load but what we do is quite standard:

1. Goofys for S3 FUSE

2. Catfs for local disk caching

3. Linux caches in memory

4. Mmap file means processes share it

5. One device then exports this over the network to other machines, each of which have an application layer disk cache.

6. Machines are linked via 10 GigE (we use SFP+).

Overall the goofys and catfs guy (kahing) wrote very performant software. Big fan.

Re: Google Cloud Storage FUSE

#67

Post by retrocryptid »

crazygringo wrote: post #35
MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

This seems overly pessimistic to me. Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app. But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend fo…

In the old days, we had a system called NFS (Network File System) where, yes, you may decide to use only remote disks. There were several advantages apart from lowering the cost of disks, mainly that you could centrally manage boot images for a fleet of machines. Then we got the web and everyone seemed to assume you could do the same thing over the internet.

I agree with you, I would prefer a local disk to one with 100+ msec of latency and local storage prices are at the point where the right answer is probably "just add local storage."

But I watch with some sympathy the small army of sys-admins (something like 15-20 people) responsible for managing the 3000+ Macs our company uses and remember the 2 person staff which supported the 1500+ diskless workstations from my years at a sadly defunct mini-super-computer manufacturer. It was quite nice... you could go to any machine and log in and your desktop would follow you. I'm told doing the same thing with MSFT requires 10-20 people just to manage the AD hardware (though as a unix-fan, I hang out with other unix-fans who are notoriously rude to MSFT, so maybe it's only 5-10 people needed to manage the AD instance.)

Re: Google Cloud Storage FUSE

#68

Post by isanjay »

retrocryptid wrote: post #67
crazygringo wrote: post #35

Earlier quoted context omitted.

This seems overly pessimistic to me. Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app. But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend fo…

In the old days, we had a system called NFS (Network File System) where, yes, you may decide to use only remote disks. There were several advantages apart from lowering the cost of disks, mainly that you could centrally manage boot images for a fleet of machines. Then we got the web and everyone seemed to assume you could do the same thing over the internet. I agree with you, I would prefer a local disk to one with 1…

I still use NFS in my home.

Re: Google Cloud Storage FUSE

#69

Post by folmar »

linsomniac wrote: post #21
MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

>What I would really like to see is a two-tier cache system Is there any sort of Linux HSM (Hieracrhical Storage Manager)? I haven't see any and have been a bit surprised nothing has really developed there. They can manage putting hot data in RAM, SSDs, colder or larger data on spinning rust, deep freezing onto a tape silo or a cloud storage... Some of the NAS devices and RAID cards can support a two-tier caching or…

CASTOR comes to mind for a start.

Take a look a the CERN paper https://iopscience.iop.org/article/10.1088/1742-6596/331/5/0... as they have a large use case.

Re: Google Cloud Storage FUSE

#70

Post by vamega »

ashishbijlani wrote: post #46
MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

I'm working on optimizing FUSE using eBPF (ExtFUSE [1]) and adding a caching layer exactly as you mentioned. Will post publicly when ready. 1. https://github.com/extfuse/extfuse

Is work on this continuing (or restarting)? I had heard of this a few years ago, but thought the project was shelved.
Post reply on HN