BigFAT – Backward compatible FAT extension for unlimited file size • Hacker News Board

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#71

Post by Tijdreiziger » Wed, Sep 07, 2022, 5:02 PM UTC

selfhoster11 wrote: post #62

chasil wrote: post #17

Earlier quoted context omitted.

There are so many good filesystems out there. Is it really necessary to keep dragging FAT along? ReactOS is using btrfs, which has so many useful options that FAT will never see (zstd, xxhash, flash-aware options, snapshots, send/receive, etc.). This is positioned both for Linux and Windows. Microsoft itself restrains ReFS to enterprise use, and btrfs offers so much more functionality. We should stop using a file sys…

BtrFS is unsafe for production use unless it's coupled with really good backups. On the other hand, NTFS on Windows, Ext* on Linux, or ZFS on any supported OS, has not been known to eat data as frequently.

Synology uses Btrfs + mdadm RAID for their NAS boxes, which are regarded as rock-solid.

https://www.synology.com/en-global/dsm/Btrfs

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#72

Post by noAnswer » Wed, Sep 07, 2022, 5:05 PM UTC

jaclaz wrote: post #47

noAnswer wrote: post #31

Earlier quoted context omitted.

Using split, cat or rar doesn't allow random access.

But also this "extension" doesn't seem like allowing it. On the contrary (but it is a specific "niche" case) the mentioned vmdk split format allows to mount the vmdk same as monolithic, with full random access.

Of course it does. "BigFAT allows random read and write access to any file, even if larger than 4GB, as required by databases."

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#73

Post by Dylan16807 » Wed, Sep 07, 2022, 5:19 PM UTC

chasil wrote: post #48

guipsp wrote: post #45

Earlier quoted context omitted.

But I just want 5GB files

Microsoft just wants a check from you. We are all forced to pay for this ancient software every time we buy a device that uses it. Wouldn't this money be better used elsewhere? https://en.m.wikipedia.org/wiki/File_Allocation_Table#Patent...

You know patents have a limited life, right?

x86_64 with SSE2 is also patent-free right now, as an example.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#74

Post by londons_explore » Wed, Sep 07, 2022, 5:22 PM UTC

NelsonMinar wrote: post #43

mort96 wrote: post #10

I like the idea. Making it backwards compatible with FAT means that, in principle, regular FAT filesystem implementations could be transparently changed to support big fat files (hehe) transparently. However, reading the spec, it doesn't look fully backwards compatible? It seems like there are file structures which are possible to represent in FAT which aren't possible to represent in BigFAT. In FAT, I could have a 4…

What FAT32 filesystem in the real world has a file named "foo.000.BigFAT" on it?

I can imagine that if bigfat is successful, such files will start to exist.

Imagine someone takes a bigfat drive and puts it in a non-bigfat capable machine, then zips up a directory and publishes it.

When that directory is unzipped on a bigfat machine, should the bigfat files be re-joined, or should they show as separate files? One breaks the OS file API and the unzip program might crash/fail, while the other leads to the application trying to create filenames which can't exist in the filesystem.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#75

Post by jonny_eh » Wed, Sep 07, 2022, 5:41 PM UTC

londons_explore wrote: post #74

NelsonMinar wrote: post #43

Earlier quoted context omitted.

What FAT32 filesystem in the real world has a file named "foo.000.BigFAT" on it?

I can imagine that if bigfat is successful, such files will start to exist. Imagine someone takes a bigfat drive and puts it in a non-bigfat capable machine, then zips up a directory and publishes it. When that directory is unzipped on a bigfat machine, should the bigfat files be re-joined, or should they show as separate files? One breaks the OS file API and the unzip program might crash/fail, while the other leads…

> should the bigfat files be re-joined, or should they show as separate files

They're only "rejoined" by the BigFAT compatible filesystem driver on access. By running such a driver, you're agreeing that such files should "appear" as one.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#76

Post by jonny_eh » Wed, Sep 07, 2022, 5:42 PM UTC

kreco wrote: post #37

chasil wrote: post #17

Earlier quoted context omitted.

There are so many good filesystems out there. Is it really necessary to keep dragging FAT along? ReactOS is using btrfs, which has so many useful options that FAT will never see (zstd, xxhash, flash-aware options, snapshots, send/receive, etc.). This is positioned both for Linux and Windows. Microsoft itself restrains ReFS to enterprise use, and btrfs offers so much more functionality. We should stop using a file sys…

Nothing beats the simplicity of FAT. Btrfs has a lot of bugs while being active for a long time. This is mostly related to its complexity. If I'm going to implement a filesystem for a custom hardware I would definitely not chose btrfs.

> Nothing beats the simplicity of FAT.

Or the sheer ubiquity, and therefore cross-device compatibility.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#77

Post by weinzierl » Wed, Sep 07, 2022, 5:43 PM UTC

noAnswer wrote: post #68

weinzierl wrote: post #58

Earlier quoted context omitted.

They offer their own file system implementation (emFile) which supports either their own storage format (EFS) or FAT. The BigFAT article is posted in the emFile section of their website. My suspicion is that customers are bugging them to support large files in emFile and they don't want to pay the license fee for exFAT. I think they even can't do that with their current licensing model, which is one-time per product…

It is not true that is "free since 2019". Source: https://www.microsoft.com/en-us/legal/intellectualproperty/t... https://www.paragon-software.com/exfat-license/ https://en.wikipedia.org/wiki/ExFAT#Legal_status You maybe of the hook if you use Linux >= 5.7. And it seem that you are of the hook if you are a member of the Open Invention Network (OIN). But SEGGER's embOS is not based on Linux and their costumers a OEMs…

This is interesting, thanks. So the Linux kernel contains essentially non-free code that is, despite being under GPL, in effect not usable by others because of patents.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#78

Post by Dylan16807 » Wed, Sep 07, 2022, 5:43 PM UTC

bArray wrote: post #63

Is this only compatible with FAT32, or is it also compatible with FAT12/16? It would be very cool if this would support floppy disks. Regarding the format, once you convert it, does the target device need to have a driver to support the format? It mentions that this would allow for > 4GB files for TVs, but these are typically non-updated very out of date OSes. I think MS missed a trick by not making the boot sector a…

> Is this only compatible with FAT32, or is it also compatible with FAT12/16? It would be very cool if this would support floppy disks.

It's simple enough to work on basically anything, but for what purpose?

The max file size on FAT12/16 is the same as the max drive size.

And FAT32 is very easy to implement for any system dealing with multiple megabytes.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#79

Post by sampa » Wed, Sep 07, 2022, 5:50 PM UTC

If only they released it back when exFAT was released. Now it has no future.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#80

Post by jaclaz » Wed, Sep 07, 2022, 6:21 PM UTC

noAnswer wrote: post #72

jaclaz wrote: post #47

Earlier quoted context omitted.

But also this "extension" doesn't seem like allowing it. On the contrary (but it is a specific "niche" case) the mentioned vmdk split format allows to mount the vmdk same as monolithic, with full random access.

Of course it does. "BigFAT allows random read and write access to any file, even if larger than 4GB, as required by databases."

I think we are talking of two different things, I was talking of the extension document/specification, that one seems to bring no particular innovation.

The actual implementation (by SEGGER or by someone else):

>Q: Can I implement BigFAT myself?

>A: Absolutely. BigFAT is a specification made available by SEGGER. Anybody is free to write a piece of software implementing it. No fees, no royalties, no headaches. You do not even have to let SEGGER or anybody else know.

is what may allow that (random access), this implementation would be useful if - instead of a "feature" of a given app/program - it would be implemented as a filesystem driver of sorts.