Live data from Hacker News

Why are there no true cross-platform filesystems?

yulian.kuncheff.com

51–60 of 109 posts

Re: Why are there no true cross-platform filesystems?

#51
I'm surprised that more people haven't mentioned ZFS! It's a stable, cross-platform filesystem, but that's just the tip of the iceberg.

There are stable implementations for Linux, Mac OS X, Solaris, and BSD (at least). You can export a filesystem from any one of these, unplug the hard drive, plug it into another computer, import the filesystem, and go. Doesn't even have to have the same CPU endianness.

Besides being designed from the ground up to be operating system and CPU endianness independent, it does a ton of other stuff that no other filesystem does. It's a total rethink of the entire filesystem / volume manager / RAID pool concept. All of these things are merged into ZFS, with a user interface that's both radically simpler and far more powerful than managing all of those things seperately.

You can easily add capacity to existing datasets (up to 256 quadrillion zettabytes, it's 128 bit!), do mirroring and striping at the filesystem level, and add hot standby drives. You can combine drives of different sizes into storage pools. You can do rolling drive size upgrades in the future to expand the filesystem size when drives become cheaper, without taking the filesystem offline. It has built-in pervasive checksumming and self-healing data corruption detection, there's a strong focus on data integrity despite flaky hardware, writes are atomic, you can take snapshots and roll back to previous filesystem states, there's an advanced adaptive memory caching system that's far beyond simple least-recently-used caches..

Is that not enough? How about built-in encryption, data compression with your choice of algorithms, deduplication of file data... You can add a small SSD drive to your huge traditional hard drive based dataset to act as an L2 cache for fast, durable writes, fully online maintenance and upgrades, builtin SMB and NFS sharing, you can stream updates...

Re: Why are there no true cross-platform filesystems?

#52
post #39

A friend of mine uses ZFS on OS X. Apparently it works quite well. I imagine Linux support is good too, but I'm not sure about Windows. At any rate, I'm surprised the author didn't mention it.

I thought ZFS on OS X was read-only.

Linux ZFS is still quite young, only supports a very early version of the fs, and can't be updated to the newest without serious reverse-engineering.

As for ZFS on windows, I definitely would not hold my breath.

I ran into the same frustration as the author and gave up. My notebook triple-boots Linux, Solaris 11, and OS X (I have no use for windows) and would be beyond thrilled to have ZFS shared among them.

One other problem with the idea though... you'd have to export the shared fs each time you shut down and import it into the next OS you booted, which would be kind of a pain (of course you could force the imports, but that's ugly and it only saves one step).

Re: Why are there no true cross-platform filesystems?

#53

I'm surprised that more people haven't mentioned ZFS! It's a stable, cross-platform filesystem, but that's just the tip of the iceberg. There are stable implementations for Linux, Mac OS X, Solaris, and BSD (at least). You can export a filesystem from any one of these, unplug the hard drive, plug it into another computer, import the filesystem, and go. Doesn't even have to have the same CPU endianness. Besides being…

Do you have experience using this through Windows? I haven't heard of many successful stories of doing so.

Re: Why are there no true cross-platform filesystems?

#54

I'm more upset there are no filesystems where I can organize files by applying labels to them, similar to how one can apply labels to email in Gmail.

Well, it isn't really a matter of file system since it is possible to implement "tags" on any FS using links. (i.e. by creating directories named after tags and by populating these directories with links to the actual files.) Therefore what you need isn't a new file system but a file system browser that allows you to specify a (set of) file(s) using tags. Considering how easy it would be to do just this using fuse, I…

Not really the approach you describe but quite some time ago I came across an experimental project which used a C implementation of Apache Lucene (I think that was called Lucy) to implement a new approach to file management, based on full-text search, where you don't store the files in a hierarchy but use more a concept liket tags and what a file is about and stuff like that. I thought it was pretty cool and the demo I tried worked pretty well for me. Unfortunately I can't find it anymore and can't remember the name or anything. I think this is a very interesting approach.

Re: Why are there no true cross-platform filesystems?

#55

I'm surprised that more people haven't mentioned ZFS! It's a stable, cross-platform filesystem, but that's just the tip of the iceberg. There are stable implementations for Linux, Mac OS X, Solaris, and BSD (at least). You can export a filesystem from any one of these, unplug the hard drive, plug it into another computer, import the filesystem, and go. Doesn't even have to have the same CPU endianness. Besides being…

As someone having a little over 6 TB of ZFS storage in my Linux home server:

How does that fit the 'One file system on Linux, OS X and _Windows_' requirement? ZFS is useless for this scenario. No need to sell it if it cannot even start in this competition.

Edit: On top of that, some features that you're selling aren't completely available cross-platform (i.e. builtin NTS/SMB support on Linux, encryption is(?) Solaris only iirc).

Re: Why are there no true cross-platform filesystems?

#56
There is one use case for such a file-system - flash cards and sticks. The 64Gb version is a tricky case. I need to use it in Android (this is sdxc card), Linux desktop, Windows desktop, hey, may be Mac too! exFAT works for Android-Windows, but not for Linux. And Cyanogenmod does not seem to play good with exFAT yet, at least the release I got.

Re: Why are there no true cross-platform filesystems?

#57

UDF is pretty cool. It is an open standard supported by Windows and Mac OS X, but for some reason linux doesn't support versions higher than 2.01 Maybe somebody here has the skill and time to fix this. http://en.wikipedia.org/wiki/Universal_Disk_Format

To be fair, the only versions higher than 2.01 listed on that page are 2.50 (which adds the ability to store all metadata on a separate partition... not that useful) and 2.60 (which adds support for "partial overwrites" on sequentially-recorded media).

Neither of those seem terribly useful for the "UDF filesystem on a shared hard-disk partition" use-case we're talking about, so I don't think Linux's limit of UDF 2.01 is very important in practice.

Re: Why are there no true cross-platform filesystems?

#58
post #7

About HFS: You seem to dismiss it a bit quickly. Trial versions of 'MacDrive' on Windows have always worked very solidly for me - if I were still using bootcamp I would pay for it. However the bigger question IMO is: Do we really need native windows nowadays? If we ban windows into VMs and only use OSX + linux natively, HFS is indeed a solid option. The only thing hindering us so far from using windows VM-only was ga…

HFS hard links don't even work readonly on Linux but are used extensively in Time Machine backups, so there is at least one common use case of HFS partitions that isn't cross platform.

Re: Why are there no true cross-platform filesystems?

#59
I ran into yet another consequence of this situation today: In a Linux/Windows setup, I lost two hours of minor Photoshop work (i.e. stuff that was simple but annoying to recreate).

I have generally found it safe to read from my NTFS drives in Linux (I avoid writing if at all possible). This time, however, when I went to load the result of my work back in Linux, it could not read the folder (Thunar reported some generic IO error). Going back into Windows to investigate, Window's opinion was "It smells like Linux has been here! I can't STAND that guy or anything he touches, so I burned the folder just to be safe. You can thank me later."

As best I can tell, it had something to do with a combination of:

1. Windows Update needing to do things upon shutdown and startup, but then not booting back into it for a month.

2. Linux having been in hibernation rather than halted. Might have had some access bit set on the NTFS drive when it hibernated? I don't actually have the domain knowledge to claim that is what might happen, so it's a complete hunch.

This was really just a rant to get out the anger at lost work. This article was quite timely, and while I'm not glad to see people share my pain (because it's still pain in the end), I hope we one day see improvements.

Re: Why are there no true cross-platform filesystems?

#60
post #59

I ran into yet another consequence of this situation today: In a Linux/Windows setup, I lost two hours of minor Photoshop work (i.e. stuff that was simple but annoying to recreate). I have generally found it safe to read from my NTFS drives in Linux (I avoid writing if at all possible). This time, however, when I went to load the result of my work back in Linux, it could not read the folder (Thunar reported some gene…

While technically hibernate (to disk at least) should unmount all filesystems - I'd be very wary of using hibernate (on either os) in a dualboot setup.

Other than that - my experience of late has been that ntfs is safe both to read and write from linux -- but as I now encrypt all my filesystems, that doesn't really help (no bitlocker support). If I actually needed to dualboot -- I suppose a separate disk/partition with truecrypt might do the trick.

My solution for the past several years have been to have all "interesting" files on a separate linux file server -- and access it with cifs/samba and/or ssh.

Post reply on HN