Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux NILFS file system: automatic continuous snapshots

dataswamp.org

Post reply on HN
Print view View original

111–120 of 158 posts

Re: Linux NILFS file system: automatic continuous snapshots

#111

Post by perrygeo »

rodgerd wrote: post #104
newcup wrote: post #12

I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014. Apart from one kernel regression bug related to NILFS2 it’s worked flawlessly (no data corruption even with the bug just no access to the file system; effectively it forced running older kernel while the bug was fixed). The continuous snapshotting has saved me a couple of times; I’ve just mounted a version…

NILFS is absolutely wonderful; it was very unfortunate that Linus chose to dub btrfs as the ext4 successor all those years ago, because it cut off a lot of interest in the plethora of interesting work that was going on at the time. A decade later and btrfs is still riddled with problems and incomplete, people are still using xfs and ext4 for lack of trust, one kernel dev has a side hobby trying to block openzfs, and…

> one kernel dev has a side hobby trying to block openzfs

Can you elaborate?

Re: Linux NILFS file system: automatic continuous snapshots

#113

Post by [deleted] »

Arnavion wrote: post #100
mustache_kimono wrote: post #93

Earlier quoted context omitted.

> the safest RAID experience seems to be filesystem on mdadm on dm-integrity on disk partition, so that the RAID and RAID errors are invisible to the filesystem. I suppose I don't understand this. Why would this be the case?

dm-integrity solves the problem of identifying which replica is good and which is bad. mdadm solves the problem of reading from the replica identified as good and fixing / reporting the replica identified as bad. The filesystem doesn't notice or care.

[deleted]

Re: Linux NILFS file system: automatic continuous snapshots

#114

Post by curt15 »

Arnavion wrote: post #88
mustache_kimono wrote: post #55

Earlier quoted context omitted.

And that may work for btrfs, but again at some cost: "When you enable nocow on your files, Btrfs cannot compute checksums, meaning the integrity against bitrot and other corruptions cannot be guaranteed (i.e. in nocow mode, Btrfs drops to similar data consistency guarantees as other popular filesystems, like ext4, XFS, ...). In RAID modes, Btrfs cannot determine which mirror has the good copy if there is corruption o…

While filesystem-integrated RAID makes sense since the filesystem can do filesystem-specific RAID placements (eg zfs), for now the safest RAID experience seems to be filesystem on mdadm on dm-integrity on disk partition, so that the RAID and RAID errors are invisible to the filesystem.

What makes dm-integrity "safer" than zfs or btrfs raid?

Re: Linux NILFS file system: automatic continuous snapshots

#115

Post by IAmPaigeAT »

nerpderp82 wrote: post #99
guipsp wrote: post #89

Earlier quoted context omitted.

BTRFS is pretty stable nowadays.

What does that mean quantifiably?

I’ve used it for about 10 years without consequence; except for problems I had once with enabling experimental options in the kernel module (custom build.) I have used BTRFS exclusively and extensively in terms of what you can do with it since realizing it that many years ago. The only thing I haven’t used is its native RAID support. I think 0 and 1 are fine but I’m not sure about 5/6; parity was still experimental last I checked in 2016 maybe it’s reliable now. Everything else though; compression, snapshots, copy on write, online defrag has all been fine. It is the default for SuSE Leap and Tumbleweed, they use it for snapper (OS snapshots) and BTRFS sub volumes are also supported by Docker for container images and containers. It saves a lot of space when images start to add up.

Re: Linux NILFS file system: automatic continuous snapshots

#116

Post by IAmPaigeAT »

77pt77 wrote: post #96
nix23 wrote: post #79

Earlier quoted context omitted.

That's why i specifically wrote -> stable...

BTRFS is not stable?

If it’s not perhaps it should be, I’ve been using it for about 10 years without any problems that I didn’t cause for myself.

Re: Linux NILFS file system: automatic continuous snapshots

#117

Post by IAmPaigeAT »

yjftsjthsd-h wrote: post #29
harvie wrote: post #14

I had issues with file locking when running some legacy database software on NILFS2. Probably caused data corruption in that database (not the FS itself). SF website of NILFS2 suggests that there are some unimplemented features, one of them being synchronous IO, which might have caused that issue? https://nilfs.sourceforge.io/en/current_status.html In some cases, the NILFS2 is safer storage for your data than ZFS. So…

> In some cases, the NILFS2 is safer storage for your data than ZFS. What cases? Do you just mean due to continuous snapshots protecting against accidental deletes or such, or are there more "under the covers" things it fixes?

If the problem is anything like this:

https://www.reddit.com/r/btrfs/comments/fcntc6/qcow2_on_btrf...

You can either make a sub volume and mount it with nocow or set the +C file attribute to disable CoW:

http://blog.jim.nz/2015/12/04/btrfs-subvolume-with-nocow.htm...

Doesn’t really look like nilfs2 has this kind of flexibility, though

Re: Linux NILFS file system: automatic continuous snapshots

#118

Post by bcjordan »

compsciphd wrote: post #59

we used NILFS 15 years ago in dejaview - https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf We combined nilfs + our process snapshotting tech (we tried to mainline it, but it didn't go, but many of the concepts ended up in CRIU though) + our remote display + screen reading tech (i.e. normal APIs) to create an environment that could record everything you ever saw visually and textually. enable you to search…

Super cool work. Are any tools like this available today? I know some VM tools have snapshotting but full history and high speed scrubbing sounds awesome.

Re: Linux NILFS file system: automatic continuous snapshots

#119

Post by kiririn »

nix23 wrote: post #18
newcup wrote: post #12

I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014. Apart from one kernel regression bug related to NILFS2 it’s worked flawlessly (no data corruption even with the bug just no access to the file system; effectively it forced running older kernel while the bug was fixed). The continuous snapshotting has saved me a couple of times; I’ve just mounted a version…

>I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014 Yes it's really sad, there we have a native and stable check-summing fs, and nearly no one knows about it.

The name isn’t great, as someone unfamiliar with it, it sounds like a synonym of ‘Null FS’ and elicits thoughts of a mock filesystem for testing. In a list of filesystems I would gloss straight over it
Post reply on HN