Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The File Filesystem (2021)

mgree.github.io

Post reply on HN
Print view View original

71–80 of 105 posts

Re: The File Filesystem (2021)

#71

Post by PMunch »

Oh this is cool! I recently wrapped libfuse in Nim and after porting the 'hello' filesystem example I made one which is more or less exactly this. However my version you pipe data and have to provide a mountpoint, then when it's done it writes the result over stdout. That means you can inline it in a pipe chain but also that you have to make sure to grab the output.

At the moment I'm exploring other stuff which could be made into file systems. I've got a statusbar thing for the Nimdow window manager which allows you to write contents to individual files and it creates a bar with blocks on them as the output. It makes it super easy to swap out what is on your bar which is pretty neat.

Another tool I've made is a music player. It uses libvlc and when given a folder it reads all the media with ID3 tags and sets up folders like 'by-artist', 'by-album', etc. Each file is named as ' - ' and contains the full path to the actual file. To play a song you cat one of these files into 'control/current' and write the word play to 'control/command'. There's a bit more to it like that like a playlist feature and some more commands, but that's the basic idea. The goal is to have a super-scriptable music player.

Re: The File Filesystem (2021)

#72

Post by purple-leafy »

When I saw the title I thought it was a meme.

But wow what a clever idea. Not sure id ever need to reach for it personally as I do most data processing in a higher level language, but I can imagine people can find use cases.

Nice out of the box thinking

Re: The File Filesystem (2021)

#73

Post by stuaxo »

duped wrote: post #26
RetroTechie wrote: post #21

Useful enough that it should be an OS-level standard feature, imho. Unix-like OSes allow mounting disk images to explore their contents. But there's many more file formats where exploring files-inside-files is useful. Compressed archives, for one. Some file managers support those, but (imho) application-level is not the optimal layer to put this functionality. Could be implemented with a kind of driver-per-filetype.

Really what you'd like to see is a way to write the mount command for each file type (do one thing well) and another command to detect the file type and dispatch accordingly (probably similar to the `file` command), all in user space. The only thing standing in the way of this today is that MacOS doesn't expose a user space file system API. You can do this on Linux, Windows, and BSDs today. (No, file provider extensi…

I want stuff to work like ZipMagic did in the early 90s/ early 2000s.

You could cd into zip files, they would act as directories and files at the same time.

I seem to remember Linus saying a file could act like a directory in Linux a long time ago too.

Though I don't think Linux has filters for the filesystem like Windows does so implementation might be more tricky.

Re: The File Filesystem (2021)

#74

Post by stuaxo »

skissane wrote: post #51
lmm wrote: post #44

Earlier quoted context omitted.

This was a core design feature of reiserfsv4, but Linux ultimately refused to merge it, probably not helped by the whole murdering-his-wife thing.

> This was a core design feature of reiserfsv4, but Linux ultimately refused to merge it IIRC, because it contained these strange beasts which functioned as both files and directories - i.e. cat would return data, but then you could cd into them and run ls. Linus (among others) didn’t want to permit those violations of the file-directory dichotomy into the Linux kernel.

Oh that's funny- I remember a much much earlier Linus mentioning how this would he possible in Linux, I didn't know anyone actually did it.

I think you really should be able to "cd" into any kind of structured data.

Re: The File Filesystem (2021)

#76

Post by mike_hearn »

duped wrote: post #60
skissane wrote: post #50

Earlier quoted context omitted.

> fuse-t uses NFS which is extremely janky and unreliable on MacOS I was wondering what issues you were talking about, and then I found this - https://github.com/macos-fuse-t/fuse-t/issues/45 - data corruption > The fundamental issue is that macOS doesn't provide an API for this natively. The API is there, Apple just doesn’t want to give anyone outside of Apple the entitlement that lets them use it. I don’t understan…

> The API is there, Apple just doesn’t want to give anyone outside of Apple the entitlement that lets them use it. If no one can call it it's not an API, it's an implementation detail. And I don't even think its exposed by headers, just alluded to by people who claim APFS is implemented in user space. > I was wondering what issues you were talking about, and then I found this Worse than this, it's possible to DoS a M…

No, it's almost certainly not because they don't give a fuck about developers. They definitely do.

It's much more likely that they want to:

a. Dogfood the API using internal use cases first when they can still make changes to the API without breaking anything. Note that the latest MacOS releases moved some filesystems into userspace using this new API. They probably learned some stuff by doing that.

b. Work out how to protect system stability from crappy userland filesystems. As you point out, bugs in FUSE providers can hang apps.

c. Work out how such an API interacts with their sandboxing system and how to avoid FUSE-style filesystems being used to subvert the sandbox. This is a common source of exploits in FUSE-style systems and is one of the key learnings from GNU/Hurd: UNIX software is written on the assumption that filing systems aren't malicious and invalidating that assumption creates new bug classes.

d. Work out what the most important use cases are and try to ensure those use cases will have a good or at least uniform UX first.

Providing a FUSE-like API is presumably also just not a high priority. By far the most common use case in terms of number of users is the Dropbox use case. FUSE is mostly used for toys and experiments beyond that (like filefs). Those matter and I'm sure there are friendly geeks on the Darwin team who'd like to enable those, but Linux also works for exploration. Certainly Apple management would not be happy about an engineer who decided to enable nerd experimentation but undermined the security system whilst doing so.

And it's worth remembering that you can have root on macOS. It means disabling SIP and adding a kernel boot arg, but that only takes a few minutes and then you can grant apps any entitlements you like:

https://github.com/osy/AMFIExemption

That's no good for people who aren't developers, but most FUSE filesystems are designed for developers anyway.

Re: The File Filesystem (2021)

#77

Post by lambdaxyzw »

jraph wrote: post #25
RetroTechie wrote: post #21

Useful enough that it should be an OS-level standard feature, imho. Unix-like OSes allow mounting disk images to explore their contents. But there's many more file formats where exploring files-inside-files is useful. Compressed archives, for one. Some file managers support those, but (imho) application-level is not the optimal layer to put this functionality. Could be implemented with a kind of driver-per-filetype.

It exists :-) For zip archives, there are fuse-zip and mount-zip which are FUSE filesystem. As an intermediate between OS level and application-level, there are desktop environment level: gvfs for GNOME and KIO for KDE, but they are compatible only in their own ecosystems.

Would be nice to have something that integrates with 7z - it supports a lot of weird archive types, including "weird" ones I care about (for example PE files, better known as ".exe files").

Re: The File Filesystem (2021)

#78

Post by lambdaxyzw »

PMunch wrote: post #71

Oh this is cool! I recently wrapped libfuse in Nim and after porting the 'hello' filesystem example I made one which is more or less exactly this. However my version you pipe data and have to provide a mountpoint, then when it's done it writes the result over stdout. That means you can inline it in a pipe chain but also that you have to make sure to grab the output. At the moment I'm exploring other stuff which could…

This makes me think, it would be nice if there was an easy built-in way to expose information about a process using the filesystem. Something like "cat /proc/$pid/fs/current_track" to get a name of a current song from a music player, or "ls /proc/$pid/fs/tabs" to list open tabs in my browser (and maybe use this to grab the html or embedded images).

I mean right now it's possible to do this using FUSE, but that's convoluted and nobody does it.

Re: The File Filesystem (2021)

#79

Post by andrewflnr »

ranger_danger wrote: post #39
andrewflnr wrote: post #37

Earlier quoted context omitted.

Wouldn't a disk image have a fixed size? It could be a pain to resize.

qcow2 is auto-expanding

But not trivially loop-mountable. I guess it's possible, though. https://unix.stackexchange.com/questions/268460/how-to-mount...

Re: The File Filesystem (2021)

#80

Post by russfink »

frizlab wrote: post #24
RetroTechie wrote: post #21

Useful enough that it should be an OS-level standard feature, imho. Unix-like OSes allow mounting disk images to explore their contents. But there's many more file formats where exploring files-inside-files is useful. Compressed archives, for one. Some file managers support those, but (imho) application-level is not the optimal layer to put this functionality. Could be implemented with a kind of driver-per-filetype.

Honest question: How is this useful? I don’t see any use-case where this would come in handy.

Exporting data to some format would be easy.
Post reply on HN