Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Storage FUSE

cloud.google.com

Post reply on HN
Print view View original

11–20 of 112 posts

Re: Google Cloud Storage FUSE

#11

Post by jsnell »

jefftk wrote: post #9
throwawaaarrgh wrote: post #6

Earlier quoted context omitted.

pretty standard limitation of object storage services iirc

Doesn't this mean that most programs you might want to use with the FUSE API won't actually work? They'll do fine for a while, until they try to seek, and then they'll get an error? Or is there a large group of programs that only ever write sequentially?

I'd think non-appending writes are quite rare in practice, other than databases. Even when the application is logically overwriting data, in other kinds of programs it's almost always implemented as writing to a new file + an atomic rename, not in-place modification.

Re: Google Cloud Storage FUSE

#13

Post by manigandham »

Object storage is a higher-level abstraction than block-storage. FUSE and similar tech can do the job for basic requirements like read-only access by legacy applications but rarely works well for other scenarios.

A more complex layer like https://objectivefs.com/ (based on the S3 API) would be more useful, although I would've expected the cloud providers to scale their own block-store/SANs backed with object-stores by now.

Re: Google Cloud Storage FUSE

#14

Post by dontlaugh »

jsnell wrote: post #11
jefftk wrote: post #9

Earlier quoted context omitted.

Doesn't this mean that most programs you might want to use with the FUSE API won't actually work? They'll do fine for a while, until they try to seek, and then they'll get an error? Or is there a large group of programs that only ever write sequentially?

I'd think non-appending writes are quite rare in practice, other than databases. Even when the application is logically overwriting data, in other kinds of programs it's almost always implemented as writing to a new file + an atomic rename, not in-place modification.

Even databases tend to do sequential writes, whether to a WAL or LSM tree.

Re: Google Cloud Storage FUSE

#15

Post by hawski »

jefftk wrote: post #9
throwawaaarrgh wrote: post #6

Earlier quoted context omitted.

pretty standard limitation of object storage services iirc

Doesn't this mean that most programs you might want to use with the FUSE API won't actually work? They'll do fine for a while, until they try to seek, and then they'll get an error? Or is there a large group of programs that only ever write sequentially?

Most programs either write a full file every time and replace the old file by a single move or append to an old file. Writting in the middle could happen in a program writting to some kind of archive or disk image. There is probably a whole group of programs that do this I'm not familiar with, but I'm pretty sure of my first sentence.

Re: Google Cloud Storage FUSE

#17

Post by tyingq »

MontyCarloHall wrote: post #7

I’ve experimented with using gcsfuse and its AWS equivalent, s3fs-fuse in production. At best, they are suited to niche applications; at worst, they are merely nice toys. The issue is that every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation. For certain applications that consistently read limited subsets of the filesyste…

Well, and there's no such thing as opening a file and modifying some small part of it. That's emulated with a full rewrite of the whole object.

Re: Google Cloud Storage FUSE

#18

Post by formerly_proven »

hawski wrote: post #15
jefftk wrote: post #9

Earlier quoted context omitted.

Doesn't this mean that most programs you might want to use with the FUSE API won't actually work? They'll do fine for a while, until they try to seek, and then they'll get an error? Or is there a large group of programs that only ever write sequentially?

Most programs either write a full file every time and replace the old file by a single move or append to an old file. Writting in the middle could happen in a program writting to some kind of archive or disk image. There is probably a whole group of programs that do this I'm not familiar with, but I'm pretty sure of my first sentence.

sqlite, dbm and similars, various zip libraries and the file formats build on that
Post reply on HN