Google Cloud Storage FUSE
cloud.google.com
Google Cloud Storage FUSE
1–10 of 112 posts
Re: Google Cloud Storage FUSE
#2Is this the same gcsfuse that's been around for years, only now with official Google support?
Re: Google Cloud Storage FUSE
#3Be aware that this is not free:
"Cloud Storage FUSE is available free of charge, but the storage, metadata, and network I/O it generates to and from Cloud Storage are charged like any other Cloud Storage interface. In other words, all data transfer and operations performed by Cloud Storage FUSE map to Cloud Storage transfers and operations, and are charged accordingly."
Re: Google Cloud Storage FUSE
#4Is this the same gcsfuse that's been around for years, only now with official Google support? https://github.com/GoogleCloudPlatform/gcsfuse
yes
> export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
Re: Google Cloud Storage FUSE
#5Cloud Storage FUSE does not support overwriting in the middle of a file. Only sequential writes are supported.
This seems like a big limitation?
Re: Google Cloud Storage FUSE
#6Cloud Storage FUSE does not support overwriting in the middle of a file. Only sequential writes are supported. This seems like a big limitation?
pretty standard limitation of object storage services iirc
Re: Google Cloud Storage FUSE
#7I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation.
For certain applications that consistently read limited subsets of the filesystem, this can be mitigated somewhat by the disk cache, but for applications that would thrash the cache, cloud buckets are simply not a good storage backend if you desire disk-like access.
What I would really like to see is a two-tier cache system: most recently accessed files are cached to RAM, with less recently accessed files spilling over to a disk-backed cache. That would open up a world of additional applications whose useful cache size exceeds practical RAM amounts.
Re: Google Cloud Storage FUSE
#8Be aware that this is not free: "Cloud Storage FUSE is available free of charge, but the storage, metadata, and network I/O it generates to and from Cloud Storage are charged like any other Cloud Storage interface. In other words, all data transfer and operations performed by Cloud Storage FUSE map to Cloud Storage transfers and operations, and are charged accordingly."
Using FUSE doesn't cost you anything extra, but it doesn't make the underlying storage free.
Re: Google Cloud Storage FUSE
#9Cloud Storage FUSE does not support overwriting in the middle of a file. Only sequential writes are supported. This seems like a big limitation?
pretty standard limitation of object storage services iirc
Doesn't this mean that most programs you might want to use with the FUSE API won't actually work? They'll do fine for a while, until they try to seek, and then they'll get an error?
Or is there a large group of programs that only ever write sequentially?
Re: Google Cloud Storage FUSE
#10I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…
My personal conspiracy theory: most "cloud services" are just... bad.
VMs and disk space I understand completely, having machines on-prem is too much of an hassle and the price isn't that bad. But for stuff like this, managed services, databases especially, you're just getting scammed.