Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux NILFS file system: automatic continuous snapshots

dataswamp.org

Post reply on HN
Print view View original

81–90 of 158 posts

Re: Linux NILFS file system: automatic continuous snapshots

#81

Post by newcup »

ComputerGuru wrote: post #67
newcup wrote: post #12

I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014. Apart from one kernel regression bug related to NILFS2 it’s worked flawlessly (no data corruption even with the bug just no access to the file system; effectively it forced running older kernel while the bug was fixed). The continuous snapshotting has saved me a couple of times; I’ve just mounted a version…

> Apart from one kernel regression bug related to NILFS2 it’s worked flawlessly Maybe on x86? I’ve tried repeatedly to use it on ARM for RaspberryPi where it would have been perfect, but always ran into various kernel panics as soon as the file system is mounted or accessed.

True, I only have used it on x86 devices. Thanks for the heads up!

I’ve heard so many stories of SD card failures (against which snapshotting might be of no help) with RaspberryPi that I’ve decided to send any valuable data promptly to safety over a network. (Though, I personally haven’t had any problems with failing SD’s.)

Re: Linux NILFS file system: automatic continuous snapshots

#82

Post by nintendo1889 »

I remember DEC/HP releasing the source to the digital unix AdvFS filesystem on sourceforge with the intent of porting it over to linux, but it never materialized. AdvFS had many advanced features. The source is still available and within it are some PDF slides that explain a lot of it's features.

Re: Linux NILFS file system: automatic continuous snapshots

#83

Post by nintendo1889 »

didgetmaster wrote: post #65
spookthesunset wrote: post #60

Earlier quoted context omitted.

It might sound weird but the hard part of what you describe is not the technology but how to design the UX in a way that you aren’t babysitting everything. And doing that is not at all easy. For all anybody knows your cat video is “worth more” to you than your thesis paper. How can you get the system to determine the worth of each file without manually setting an attribute each time you create a file? And if you let…

You could always set the default behavior to be uniform for all files (e.g. protect everything or protect nothing) and just forget about it. But it would be nice to be able to manually set the protection level for specific files that are the exception. If I was copying an important file into an unprotected environment, I could change how it was handled (likewise if I was downloading some huge video I didn't care abou…

A directory basis, or even better, a numerical priority that could be manually set in the application that generated them, or automatically, based on the user or application or in a hypervisor, based on the VM. Then it could be an opportunistic setting.

I thought ZFS had some sort of unique settings like this.

Re: Linux NILFS file system: automatic continuous snapshots

#85

Post by derefr »

lazide wrote: post #56
didgetmaster wrote: post #39

Does NILFS do checksums and snapshotting for every single file in the system? One of my biggest complaints about file systems in general is that they are all designed to treat every file the exact same way. We now have storage systems (even SSDs) that are big enough to hold hundreds of millions of files. Those files can be a mix of small files, big files, temp files, personal files, and public files. Yet every file s…

Why is it ‘wasted’? Those things are mostly free on modern hardware. The challenge with your thesis here is that the only one who can know what is ‘that important’ is YOU , and your decision making and communication bandwidth is already the limiting factor. For many users, that cat video would be heartbreaking to lose, and they don’t have term papers to worry about. So having to decide or think what is or is not ‘imp…

> For many users, that cat video would be heartbreaking to lose, and they don’t have term papers to worry about.

Depends on where that cat video is / how it ended up on the disk.

The user explicitly saved it to their user-profile Downloads directory? Yeah, sure, the user might care a lot about preserving that data. There's intent there.

The user's web browser implicitly saved it into the browser's cache directory? No, the user absolutely doesn't care. That directory is a pure transparent optimization over just loading the resource from the URL again; and the browser makes no guarantees of anything in it surviving for even a few minutes. The user doesn't even know they have the data; only the browser does. As such, the browser should be able to tell the filesystem that this data is discardable cache data, and the filesystem should be able to apply different storage policies based on that.

This is already true of managed cache/spool/tmp directories vis-a-vis higher-level components of the OS. macOS, for example, knows that stuff that's under ~/Library/Caches can be purged when disk space is tight, so it counts it as "reclaimable space"; and in some cases (caches that use CoreData) the OS can even garbage-collect them itself.

So, why not also avoid making these files a part of backups? Why not avoid checksumming them? Etc.

Re: Linux NILFS file system: automatic continuous snapshots

#86

Post by usr1106 »

ggm wrote: post #84

Didn't VMS have this baked in? My memory is that all 8.3 file names had 8.3[;nnn] version tagging under the hood

That's what it looked like, but I doubt it was deep in the filesystem. It was basically just a naming convention. User had to purge old versions manually. This gets tedious if you have many files that change often. Snapshots are a safety net, not something you want to have in your way all day long.

Re: Linux NILFS file system: automatic continuous snapshots

#87

Post by Arnavion »

heavyset_go wrote: post #71
didgetmaster wrote: post #39

Does NILFS do checksums and snapshotting for every single file in the system? One of my biggest complaints about file systems in general is that they are all designed to treat every file the exact same way. We now have storage systems (even SSDs) that are big enough to hold hundreds of millions of files. Those files can be a mix of small files, big files, temp files, personal files, and public files. Yet every file s…

You can turn off CoW, checksumming, compression, etc at the file and directory levels using btrfs.

Indeed. You can also make a directory into a subvolume so that that directory is not included in snapshots of the parent volume.

Re: Linux NILFS file system: automatic continuous snapshots

#88

Post by Arnavion »

mustache_kimono wrote: post #55
Conan_Kudo wrote: post #51

Earlier quoted context omitted.

On Btrfs, you can mark a folder/file/subvolume to have nocow, which has the effect of only doing a COW operation when you are creating snapshots.

And that may work for btrfs, but again at some cost: "When you enable nocow on your files, Btrfs cannot compute checksums, meaning the integrity against bitrot and other corruptions cannot be guaranteed (i.e. in nocow mode, Btrfs drops to similar data consistency guarantees as other popular filesystems, like ext4, XFS, ...). In RAID modes, Btrfs cannot determine which mirror has the good copy if there is corruption o…

While filesystem-integrated RAID makes sense since the filesystem can do filesystem-specific RAID placements (eg zfs), for now the safest RAID experience seems to be filesystem on mdadm on dm-integrity on disk partition, so that the RAID and RAID errors are invisible to the filesystem.

Re: Linux NILFS file system: automatic continuous snapshots

#89

Post by guipsp »

conradev wrote: post #75
nix23 wrote: post #18

Earlier quoted context omitted.

>I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014 Yes it's really sad, there we have a native and stable check-summing fs, and nearly no one knows about it.

BTRFS is also a native copy on write filesystem that verifies a configurable checksum and supports snapshots. The snapshots are not automatic, but short of that it is pretty feature complete

BTRFS is pretty stable nowadays.

Re: Linux NILFS file system: automatic continuous snapshots

#90

Post by usr1106 »

fuckstick wrote: post #70
goodpoint wrote: post #20

Earlier quoted context omitted.

Compared to other logging filesystems obviously.

Nilfs baseline (write throughput especially) is slow as shit compared to other filesystems including f2fs. So just because you have this feature that doesn’t make it even slower isn’t that interesting - you pay for it one way or the other.

For many users filesystem speed of your home directory is completely irrelevant unless you run on a Raspberry Pi using SD cards. You just don't notice it.

Of course if you haver server handling let's say video files things will be very different. And there are some users who process huge amounts of data.

I run 2 lvm snapshots (daily and weekly) on my home partition for years. Write performance is abysmal if you measure it, but you don't note it in daily development work.

Post reply on HN