Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BigFAT – Backward compatible FAT extension for unlimited file size

segger.com

Post reply on HN
Print view View original

101–109 of 109 posts

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#101

Post by MisterTea »

chasil wrote: post #17
mort96 wrote: post #10

I like the idea. Making it backwards compatible with FAT means that, in principle, regular FAT filesystem implementations could be transparently changed to support big fat files (hehe) transparently. However, reading the spec, it doesn't look fully backwards compatible? It seems like there are file structures which are possible to represent in FAT which aren't possible to represent in BigFAT. In FAT, I could have a 4…

There are so many good filesystems out there. Is it really necessary to keep dragging FAT along? ReactOS is using btrfs, which has so many useful options that FAT will never see (zstd, xxhash, flash-aware options, snapshots, send/receive, etc.). This is positioned both for Linux and Windows. Microsoft itself restrains ReFS to enterprise use, and btrfs offers so much more functionality. We should stop using a file sys…

> ReactOS is using btrfs, which has so many useful options that FAT will never see (zstd, xxhash, flash-aware options, snapshots, send/receive, etc.). This is positioned both for Linux and Windows.

I just want to transfer files on USB sticks without worrying about file size or the OS accessing it. The infuriating part is that it is 2022 and if you want to reliably and easily move files larger than 3-4GB on removable media people tell you to use proprietary MS file systems like ExFAT And NTFS. That is unacceptable.

We NEED a simple, portable, and freely open file system spec for removable media that handles large drives and files.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#102

Post by Tijdreiziger »

selfhoster11 wrote: post #98
Tijdreiziger wrote: post #71

Earlier quoted context omitted.

Synology uses Btrfs + mdadm RAID for their NAS boxes, which are regarded as rock-solid. https://www.synology.com/en-global/dsm/Btrfs

Whatever voodoo they are pulling out of thin air, is clearly not present on commodity, non-tuned Btrfs filesystems. That, and most people have backups set up on their NAS.

Well, if you want, you can just pull the drives out of your Synology box and hook them up to any Linux system [1]. So whichever way they're tuning it, it can't be that voodoo-esque.

[1] https://kb.synology.com/en-global/DSM/tutorial/How_can_I_rec...

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#103

Post by bArray »

pantalaimon wrote: post #88
bArray wrote: post #63

Is this only compatible with FAT32, or is it also compatible with FAT12/16? It would be very cool if this would support floppy disks. Regarding the format, once you convert it, does the target device need to have a driver to support the format? It mentions that this would allow for > 4GB files for TVs, but these are typically non-updated very out of date OSes. I think MS missed a trick by not making the boot sector a…

How do you get files > 4 GiB on a floppy disk?

You don't of course, it's about having a filesystem that works from floppy disk all the way to hard drive.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#104

Post by bArray »

Dylan16807 wrote: post #78
bArray wrote: post #63

Is this only compatible with FAT32, or is it also compatible with FAT12/16? It would be very cool if this would support floppy disks. Regarding the format, once you convert it, does the target device need to have a driver to support the format? It mentions that this would allow for > 4GB files for TVs, but these are typically non-updated very out of date OSes. I think MS missed a trick by not making the boot sector a…

> Is this only compatible with FAT32, or is it also compatible with FAT12/16? It would be very cool if this would support floppy disks. It's simple enough to work on basically anything, but for what purpose? The max file size on FAT12/16 is the same as the max drive size. And FAT32 is very easy to implement for any system dealing with multiple megabytes.

As I replied to the comment, it's about having a filesystem that scales from floppy disks to hard drives. This is quite important for hobby OSes.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#105

Post by scohesc »

tenebrisalietum wrote: post #97
scohesc wrote: post #49

Would it not be possible to create a filesystem with modern capabilities but with backwards compatibility with FAT? Why can't we just have "legacy" commands built into the ReFS filesystem that process any FAT filesystem access? I'm very ignorant to this but I'd love some insight from someone vastly more knowledgeable than me.

A filesystem translates a filename-based streaming I/O API to the way a disk talks, which is "Read/write 512 times X bytes of data at disk block N." The I/O API or "commands" are the same; different filesystems will implement it differently.

Thanks!

I wasn't aware of the intricacies of this stuff - I'll have to do some more reading in my free time.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#106

Post by derefr »

chasil wrote: post #17
mort96 wrote: post #10

I like the idea. Making it backwards compatible with FAT means that, in principle, regular FAT filesystem implementations could be transparently changed to support big fat files (hehe) transparently. However, reading the spec, it doesn't look fully backwards compatible? It seems like there are file structures which are possible to represent in FAT which aren't possible to represent in BigFAT. In FAT, I could have a 4…

There are so many good filesystems out there. Is it really necessary to keep dragging FAT along? ReactOS is using btrfs, which has so many useful options that FAT will never see (zstd, xxhash, flash-aware options, snapshots, send/receive, etc.). This is positioned both for Linux and Windows. Microsoft itself restrains ReFS to enterprise use, and btrfs offers so much more functionality. We should stop using a file sys…

FAT is for weak little cost-optimized embedded-microcontroller devices that write one file at a time to an SD card — which is something we're still building to this day, in the form of IoT devices. We don't really have any better option for this use-case; every newer filesystem is either non-portable, or assumes stronger hardware such that the overhead of using it on these devices would be huge.

I would note that one way to work around the cost-incentives of IoT manufacturers, would be to encourage them to externalize the storage-layer costs from the device's BOM, by focusing on getting "object-storage oriented" NAND flash controllers pushed down from enterprise to regular retail availability. That way, all the filesystem-layer smarts end up living in the SD card itself — which is sold separately. (It'd be sort of a second coming of the ancient Commodore 1540/1541 paradigm, where the disk controller presented not as block storage, but as, essentially, a single-user serial-attached NAS.)

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#107

Post by leetnewb »

rleigh wrote: post #85
winter_blue wrote: post #65

Earlier quoted context omitted.

Is BtrFS without RAID safe? According to them ( https://btrfs.wiki.kernel.org/index.php/Status ) only RAID56 is unstable.

There are still major bugs in the rest of it. You can trivially corrupt a mirror. There are examples for how to reproduce it exactly in qemu with virtual drives. I caused me total dataloss on my first Btrfs filesystem at least 8 years back. That bug is apparently still there. The unbalancing issues are still there. I have zero trust of Btrfs in any form.

Can you link to that? I have been using btrfs in raid1 and single for ~3-4 years now without any data loss.

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#108

Post by michaelbrave »

chasil wrote: post #17
mort96 wrote: post #10

I like the idea. Making it backwards compatible with FAT means that, in principle, regular FAT filesystem implementations could be transparently changed to support big fat files (hehe) transparently. However, reading the spec, it doesn't look fully backwards compatible? It seems like there are file structures which are possible to represent in FAT which aren't possible to represent in BigFAT. In FAT, I could have a 4…

There are so many good filesystems out there. Is it really necessary to keep dragging FAT along? ReactOS is using btrfs, which has so many useful options that FAT will never see (zstd, xxhash, flash-aware options, snapshots, send/receive, etc.). This is positioned both for Linux and Windows. Microsoft itself restrains ReFS to enterprise use, and btrfs offers so much more functionality. We should stop using a file sys…

I mostly use fat as a go between for different operating systems, things could be installed to implement similar functionality around another file system, but it's nice to have a default built into everything format that works on every machine. It has its flaws, but the universality of it is a huge strength

Re: BigFAT – Backward compatible FAT extension for unlimited file size

#109

Post by rleigh »

leetnewb wrote: post #107
rleigh wrote: post #85

Earlier quoted context omitted.

There are still major bugs in the rest of it. You can trivially corrupt a mirror. There are examples for how to reproduce it exactly in qemu with virtual drives. I caused me total dataloss on my first Btrfs filesystem at least 8 years back. That bug is apparently still there. The unbalancing issues are still there. I have zero trust of Btrfs in any form.

Can you link to that? I have been using btrfs in raid1 and single for ~3-4 years now without any data loss.

You (and many others) might well be using RAID1 mirrors without problems. I did as well. But the problems here are not encountered during day-to-day usage. They are bugs in the recovery codepaths following hardware failures. I suffered this due to a transient SATA cable glitch, but the instructions let you exactly reproduce this with qemu with a recent kernel. I've not tried the qemu approach myself; I moved over to ZFS a good while back now.

I've had a hunt for the specific instructions but I'm afraid I can't find it again with a search. The gist of it was to:

- create Btrfs mirror using two qemu virtual disks

- pull the (virtual) plug on one of the pair to disconnect it, then later reconnect it

- Btrfs ends up hosing both the outdated and current copies of the mirror, leading to complete dataloss of the entire mirror

Post reply on HN