Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The File Filesystem (2021)

mgree.github.io

Post reply on HN
Print view View original

61–70 of 105 posts

Re: The File Filesystem (2021)

#61

Post by skissane »

ranger_danger wrote: post #57
skissane wrote: post #33

Earlier quoted context omitted.

Recent macOS versions do have a general purpose built-in API for user mode filesystems. That API is incompatible with FUSE. The big problem is it is undocumented and you need an entitlement from Apple to use it, and Apple won’t give you that. Apple do have a publicly available API for cloud file systems (Dropbox-style products), but it makes a lot of assumptions which makes it effectively unusable for other use cases…

According to the MacFUSE author, their specific approach is not actually undocumented: > Apple has put it in an umbrella called "unsupported" (in the kernel interfaces section) ... either Apple will not take this interface away, and if they do, it will be to provide a better interface http://preserve.mactech.com/articles/mactech/Vol.23/23.03/Ma...

Yes, that’s not what I’m talking about though

MacFUSE uses the kernel mode VFS API

I’m talking about the undocumented user mode filesystem LiveFS/UserFS/com.apple.filesystems.lifs API which was added in Monterey (macOS 12), and since Ventura (macOS 13) is used to implement the OOTB FAT and exFAT filesystem support. Using that requires private entitlements (e.g. com.apple.private.LiveFS.connection) which Apple (thus far) won’t give to anyone else

Re: The File Filesystem (2021)

#62

Post by skissane »

duped wrote: post #60
skissane wrote: post #50

Earlier quoted context omitted.

> fuse-t uses NFS which is extremely janky and unreliable on MacOS I was wondering what issues you were talking about, and then I found this - https://github.com/macos-fuse-t/fuse-t/issues/45 - data corruption > The fundamental issue is that macOS doesn't provide an API for this natively. The API is there, Apple just doesn’t want to give anyone outside of Apple the entitlement that lets them use it. I don’t understan…

> The API is there, Apple just doesn’t want to give anyone outside of Apple the entitlement that lets them use it. If no one can call it it's not an API, it's an implementation detail. And I don't even think its exposed by headers, just alluded to by people who claim APFS is implemented in user space. > I was wondering what issues you were talking about, and then I found this Worse than this, it's possible to DoS a M…

> Worse than this, it's possible to DoS a Mac with an NFS server just by refusing to reply to a request.

I wonder if their SMB/CIFS client implementation has these kinds of issues? It probably gets used more heavily

> And I don't even think its exposed by headers

Apple (accidentally?) released some of the private headers for this feature in one of their open source releases: https://github.com/apple-oss-distributions/msdosfs/blob/rel/...

Re: The File Filesystem (2021)

#63

Post by ranger_danger »

skissane wrote: post #61
ranger_danger wrote: post #57

Earlier quoted context omitted.

According to the MacFUSE author, their specific approach is not actually undocumented: > Apple has put it in an umbrella called "unsupported" (in the kernel interfaces section) ... either Apple will not take this interface away, and if they do, it will be to provide a better interface http://preserve.mactech.com/articles/mactech/Vol.23/23.03/Ma...

Yes, that’s not what I’m talking about though MacFUSE uses the kernel mode VFS API I’m talking about the undocumented user mode filesystem LiveFS/UserFS/com.apple.filesystems.lifs API which was added in Monterey (macOS 12), and since Ventura (macOS 13) is used to implement the OOTB FAT and exFAT filesystem support. Using that requires private entitlements (e.g. com.apple.private.LiveFS.connection) which Apple (thus f…

What about the File Provider API?

Re: The File Filesystem (2021)

#65

Post by skissane »

ranger_danger wrote: post #63
skissane wrote: post #61

Earlier quoted context omitted.

Yes, that’s not what I’m talking about though MacFUSE uses the kernel mode VFS API I’m talking about the undocumented user mode filesystem LiveFS/UserFS/com.apple.filesystems.lifs API which was added in Monterey (macOS 12), and since Ventura (macOS 13) is used to implement the OOTB FAT and exFAT filesystem support. Using that requires private entitlements (e.g. com.apple.private.LiveFS.connection) which Apple (thus f…

What about the File Provider API?

It is designed for the cloud storage use case (Dropbox, Google Drive, etc) - it creates local copies of files and synchs them with remote ones. Not what you want to do in the general case.

Re: The File Filesystem (2021)

#66

Post by IlliOnato »

ThinkBeat wrote: post #18

No XML, Excel,PDF or CSV support yet.

I guess support for XML would be tricky, because XML is just way more complex format than the ones already supported. It is still essentially a tree, but with additional structure.

Representing elements and their contents is easy enough. But attributes, comments, processing instructions, entities... And remember, an XML document can include a DTD (it does not have to be in a separate file).

To present it as a file system in a useful, non-convoluted way? I will be very, very interested if it's possible, but not holding my breath.

Re: The File Filesystem (2021)

#67

Post by duped »

skissane wrote: post #62
duped wrote: post #60

Earlier quoted context omitted.

> The API is there, Apple just doesn’t want to give anyone outside of Apple the entitlement that lets them use it. If no one can call it it's not an API, it's an implementation detail. And I don't even think its exposed by headers, just alluded to by people who claim APFS is implemented in user space. > I was wondering what issues you were talking about, and then I found this Worse than this, it's possible to DoS a M…

> Worse than this, it's possible to DoS a Mac with an NFS server just by refusing to reply to a request. I wonder if their SMB/CIFS client implementation has these kinds of issues? It probably gets used more heavily > And I don't even think its exposed by headers Apple (accidentally?) released some of the private headers for this feature in one of their open source releases: https://github.com/apple-oss-distributions…

Maybe? It's kind of hard to tell. It's not exactly easy to write any of these servers from scratch to find out. But I wouldn't be surprised - they want app developers to be using the file provider extension API, which is unsuitable for everyone who isn't making a Dropbox clone.

That link is very interesting. It doesn't smell like any other Apple API as they're exposing a vtable with good documentation comments. It would be interesting to hack with this with SIP disabled to see how it works. I'm especially curious about how mount/unmount work and how the plugin registers itself with the OS, or what application is the client/host.

Re: The File Filesystem (2021)

#68

Post by eyelidlessness »

IlliOnato wrote: post #66
ThinkBeat wrote: post #18

No XML, Excel,PDF or CSV support yet.

I guess support for XML would be tricky, because XML is just way more complex format than the ones already supported. It is still essentially a tree, but with additional structure. Representing elements and their contents is easy enough. But attributes, comments, processing instructions, entities... And remember, an XML document can include a DTD (it does not have to be in a separate file). To present it as a file sy…

On the one hand, I can’t help but point out you forgot to mention the other big inherent complexity that would make XML-as-FS a uniquely complex beast: namespaces.

On the other hand, I can’t help but point out that a related technology comes very close to demonstrating how you might map XML to a file system: XPath. Probably the biggest issue would be syntax, and again largely due to namespaces.

Re: The File Filesystem (2021)

#69

Post by Sophira »

RetroTechie wrote: post #21

Useful enough that it should be an OS-level standard feature, imho. Unix-like OSes allow mounting disk images to explore their contents. But there's many more file formats where exploring files-inside-files is useful. Compressed archives, for one. Some file managers support those, but (imho) application-level is not the optimal layer to put this functionality. Could be implemented with a kind of driver-per-filetype.

This already exists - avfs[0] does this as a FUSE filesystem. It's not the most intuitive to use, but it works, and is extensible.

[0] https://avf.sourceforge.net/

Re: The File Filesystem (2021)

#70

Post by hnlmorg »

paulgb wrote: post #28

This is really neat, but when I saw the headline I got excited that it was something I have been looking for / considering writing, and I figure the comments here would be a good place to ask if something like this exists: Is there a FUSE filesystem that runs in-memory (like tmpfs) while mounted, and then when dismounted it serializes to a single file on disk? The closest I can find are FUSE drivers that mount archiv…

Why does it have to be in memory?

I’m sure you’re already aware of this, but there are all kinds of very real scenarios that could lead to corrupted data if you’re only flushing the buffer upon unmounting.

Sounds like you’ve got an interesting problem you’re trying to solve though.

Post reply on HN