The thinking behind the 32GB Windows Format limit on FAT32 • Hacker News Board

Re: The thinking behind the 32GB Windows Format limit on FAT32

#61

Post by adrian_b » Mon, Jan 04, 2021, 6:11 PM UTC

Abishek_Muthian wrote: post #58

m000 wrote: post #3

TIL: In 2019, Microsoft released the exFAT [1] specs to a large group of vendors [2], including prominent Linux vendors. I.e. exFAT should now be preferred over FAT/NTFS for media used across different operating systems. [1] https://en.wikipedia.org/wiki/ExFAT [2] https://en.wikipedia.org/wiki/Open_Invention_Network

Much before MS did that, macOS had builtin exFAT support presumably licensed from MS, Paragon had exFAT support for android for a one time fee which works with other file manager apps(also licensed from MS), On Linux Exfat-utils enabled support for exFAT(open-source implementation). Has there been any changes to these since MS has opened the license to exFAT?

The change is that now there is an exFAT driver included in the Linux kernel.

Therefore, if you have a recent Linux kernel, you no longer need to install a user-mode driver for exFAT and the kernel driver has better performance.

Re: The thinking behind the 32GB Windows Format limit on FAT32

#62

Post by m000 » Mon, Jan 04, 2021, 6:18 PM UTC

Abishek_Muthian wrote: post #58

m000 wrote: post #3

TIL: In 2019, Microsoft released the exFAT [1] specs to a large group of vendors [2], including prominent Linux vendors. I.e. exFAT should now be preferred over FAT/NTFS for media used across different operating systems. [1] https://en.wikipedia.org/wiki/ExFAT [2] https://en.wikipedia.org/wiki/Open_Invention_Network

Much before MS did that, macOS had builtin exFAT support presumably licensed from MS, Paragon had exFAT support for android for a one time fee which works with other file manager apps(also licensed from MS), On Linux Exfat-utils enabled support for exFAT(open-source implementation). Has there been any changes to these since MS has opened the license to exFAT?

Third-party exFAT support did exist before 2019, but vendors had to pay a licence fee to Microsoft. Apple apparently paid the fee.

Regarding Linux, exfat-utils only provided limited functionality. The kernel mainline lacked a native exFAT driver. Mounting exFAT volumes was possible only through FUSE (with all its shortcomings).

Re: The thinking behind the 32GB Windows Format limit on FAT32

#63

Post by kasabali » Mon, Jan 04, 2021, 6:44 PM UTC

MrDOS wrote: post #57

kasabali wrote: post #53

Earlier quoted context omitted.

Were you using the FUSE driver?

That's a great question. I kind of skirted around indicating one way or the other in my parent post because I've forgotten. My troubles were as recent as the beginning of 2020, but I encountered them on a Raspbian machine, so Lord knows what kernel version it was running at the time. Probably pre-5.4, in which case, yes, it would've been the FUSE driver.

Thanks it was probably the FUSE driver. I haven't had a problem with it but I'm not a heavy user.

On the other hand I've been a heavy user of exFat formatted sd cards on Samsung phones for long years and had no problems with it, so hopefully the new Linux driver (which Samsung upstreamed) will be solid.

Re: The thinking behind the 32GB Windows Format limit on FAT32

#64

Post by KronisLV » Mon, Jan 04, 2021, 7:22 PM UTC

adrian_b wrote: post #60

fuzzfactor wrote: post #45

Earlier quoted context omitted.

The patents on FAT32 are now fully expired, including the final lingering LongFileName one. Therefore FAT32 will remain preferred until the patents expire on exFAT, even though the specification for exFAT has been released, things like that have proven to be not good enough until after the patents expire. The purpose of exFAT appears mainly to be an attempt to foist a new encumbered filesystem on developers now that…

The main use for FAT32 and exFAT is to transfer files between different operating systems or between a computer and some dedicated device, e.g. a photographic camera. They are not useful for anything else except memory cards, USB sticks and external SSDs. Unfortunately, nowadays it is frequently necessary to copy files larger than 4 GB. When using FAT32, you cannot do that without splitting the file and concatenating…

> It does not matter any more if there are still valid patents for exFAT.

Would you care to elaborate for why that is? Would anyone else with knowledge about the patent system and any possible risks imposed by it also like to do so?

Re: The thinking behind the 32GB Windows Format limit on FAT32

#65

Post by igneo676 » Mon, Jan 04, 2021, 9:05 PM UTC

KronisLV wrote: post #64

adrian_b wrote: post #60

Earlier quoted context omitted.

The main use for FAT32 and exFAT is to transfer files between different operating systems or between a computer and some dedicated device, e.g. a photographic camera. They are not useful for anything else except memory cards, USB sticks and external SSDs. Unfortunately, nowadays it is frequently necessary to copy files larger than 4 GB. When using FAT32, you cannot do that without splitting the file and concatenating…

> It does not matter any more if there are still valid patents for exFAT. Would you care to elaborate for why that is? Would anyone else with knowledge about the patent system and any possible risks imposed by it also like to do so?

Because, effectively, the exFAT filesystem is OSS since August 2019 and Microsoft released all the specifications to it [0]. Microsoft has essentially taken public, legal steps to ensure that anyone else can use exFAT without legal encumbrances and without threat of lawsuit.

IANAL - so there's likely some caveats (you probably can't just use the exFAT name unless certain qualifications are met, just like you can't use Firefox's trademarks willy-nilly), but that should be good enough for just about anyone. Including paranoid OSS peeps like me :)

0: https://cloudblogs.microsoft.com/opensource/2019/08/28/exfat...

Re: The thinking behind the 32GB Windows Format limit on FAT32

#66

Post by chungus_khan » Mon, Jan 04, 2021, 9:45 PM UTC

avidphantasm wrote: post #41

Hopefully exFAT implementations will get more robust. At least with the macOS implementation, it has always been prone to reporting corruption when devices are not ejected properly. I can usually fix it by running check disk or whatever on Windows, but sometimes not.

One change from previous FAT filesystems to exFAT is the removal of the second, redundant file allocation table. Disks are less unreliable now so it is less necessary, but there are still some scenarios, especially when the filesystem is in a dirty state, where the table can be corrupted.

This can also really hurt in unexpected ways with otherwise badly written drivers. Nintendo's driver for the switch would constantly write to the table unnecessarily, and while this doesn't do all that much in normal operation (other than waste a small number of writes), it made it drastically more likely (in some cases for some games almost guaranteed) that the table would be partially written and corrupt if the system crashed or lost power unexpectedly. When some new Pokemon games came out and were initially a bit unstable this came up a lot.

Re: The thinking behind the 32GB Windows Format limit on FAT32

#67

Post by Delk » Mon, Jan 04, 2021, 10:14 PM UTC

rlv-dan wrote: post #17

He made a decision and assumed when making future NT versions this would be revised and updated with the times. I can only image that future teams looked at this and thought "someone chose 32GB for a reason, better not change it"...

Maybe, but it could also be about just not getting around to doing it.

Changes, especially in a large system with a large user base, require extensive testing, and even small changes may come with a lot of red tape. I don't know how things work at Microsoft, but I wouldn't be surprised if making changes to the UI for a technology that's being superseded anyway just didn't make anybody's list.

Re: The thinking behind the 32GB Windows Format limit on FAT32

#68

Post by brnt » Mon, Jan 04, 2021, 10:14 PM UTC

MrDOS wrote: post #46

avidphantasm wrote: post #41

Hopefully exFAT implementations will get more robust. At least with the macOS implementation, it has always been prone to reporting corruption when devices are not ejected properly. I can usually fix it by running check disk or whatever on Windows, but sometimes not.

I've had more filesystem corruption trouble with exFAT on Linux than with any other filesystem, ever, including Btrfs. Really crossing my fingers and hoping for the quality of the Samsung driver in Linux 5.7. It's going to be great to have an option for high-capacity, cross-platform read/write storage. The day I never have to use NTFS-3G again cannot come too soon.

So, since we're on kernel 5.10, has this new Samsung exFAT driver been tested? Is it at least as reliable as btrfs?

Re: The thinking behind the 32GB Windows Format limit on FAT32

#69

Post by Delk » Mon, Jan 04, 2021, 10:21 PM UTC

igneo676 wrote: post #65

KronisLV wrote: post #64

Earlier quoted context omitted.

> It does not matter any more if there are still valid patents for exFAT. Would you care to elaborate for why that is? Would anyone else with knowledge about the patent system and any possible risks imposed by it also like to do so?

Because, effectively, the exFAT filesystem is OSS since August 2019 and Microsoft released all the specifications to it [0]. Microsoft has essentially taken public, legal steps to ensure that anyone else can use exFAT without legal encumbrances and without threat of lawsuit. IANAL - so there's likely some caveats (you probably can't just use the exFAT name unless certain qualifications are met, just like you can't us…

I'm also not a lawyer, but AFAIK publishing a specification has nothing to do with whether the tech can be legally used without a patent license.

I don't know how the release to Open Invention Network affects things, though.

Re: The thinking behind the 32GB Windows Format limit on FAT32

#70

Post by m000 » Mon, Jan 04, 2021, 10:56 PM UTC

KronisLV wrote: post #64

adrian_b wrote: post #60

Earlier quoted context omitted.

The main use for FAT32 and exFAT is to transfer files between different operating systems or between a computer and some dedicated device, e.g. a photographic camera. They are not useful for anything else except memory cards, USB sticks and external SSDs. Unfortunately, nowadays it is frequently necessary to copy files larger than 4 GB. When using FAT32, you cannot do that without splitting the file and concatenating…

> It does not matter any more if there are still valid patents for exFAT. Would you care to elaborate for why that is? Would anyone else with knowledge about the patent system and any possible risks imposed by it also like to do so?

> Would you care to elaborate for why that is? Would you care to elaborate for why that is?

To put it bluntly, if Microsoft's guarantees are good enough for Linus so as to merge the driver in the Linux mainline, they're good enough for me too.