Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux NILFS file system: automatic continuous snapshots

dataswamp.org

Post reply on HN
Print view View original

61–70 of 158 posts

Re: Linux NILFS file system: automatic continuous snapshots

#61

Post by compsciphd »

sargun wrote: post #48

I've always wondered why NILFS (or similar) isn't used for cases where ransomware is a risk. I'm honestly surprised that it's not mandated to use a append-only / log-structured filesystem for some critical systems (think patient records), where the cost of losing data is so high, rarely mutated, and trading it off for wasting storage isn't that bad (after all, HDD storage is incredibly cheap, and nobody said you had…

you don't need a log structured fs to do this, you could just have regular zfs/btrfs snapshots too.

BUT

if an attack has the ability to delete an entire file system / encrypt it, they really have the ability to delete the snapshots as well, the only reason they might not is due to "security through obscurity".

now, what I have argued is that an append only file system which works in a SAN like environment (i.e. you have random reads, but only append writes properties that are enforced remotely) could give you that, but to an extent you'd still get a similar behavior by just exporting ZFS shares (or even as block devices) and snapshotting them regularly on the remote end.

Re: Linux NILFS file system: automatic continuous snapshots

#62

Post by didgetmaster »

lazide wrote: post #56
didgetmaster wrote: post #39

Does NILFS do checksums and snapshotting for every single file in the system? One of my biggest complaints about file systems in general is that they are all designed to treat every file the exact same way. We now have storage systems (even SSDs) that are big enough to hold hundreds of millions of files. Those files can be a mix of small files, big files, temp files, personal files, and public files. Yet every file s…

Why is it ‘wasted’? Those things are mostly free on modern hardware. The challenge with your thesis here is that the only one who can know what is ‘that important’ is YOU , and your decision making and communication bandwidth is already the limiting factor. For many users, that cat video would be heartbreaking to lose, and they don’t have term papers to worry about. So having to decide or think what is or is not ‘imp…

Nothing is free or even 'mostly free' when managing data. Data security (encryption), redundancy (backups), and integrity (checksums, etc.) all impose a cost on the system.

Getting each piece of data properly classified will always be a challenge (AI or other tools may help with that), but it would still be nice to be able to do it. If I have a 50GB video file that I could easily re-download off the Internet, it would be nice to be able to turn off any security, redundancy, or integrity features for it.

I wonder how many petabytes of storage space is being wasted by having multiple backups of all the operating system files that could be easily downloaded from multiple websites. Do I really need to encrypt that GB file that 10 million people also have a copy of? Am I worried if a single pixel in that high resolution photo has changed due to bit rot?

Re: Linux NILFS file system: automatic continuous snapshots

#65

Post by didgetmaster »

spookthesunset wrote: post #60
didgetmaster wrote: post #39

Does NILFS do checksums and snapshotting for every single file in the system? One of my biggest complaints about file systems in general is that they are all designed to treat every file the exact same way. We now have storage systems (even SSDs) that are big enough to hold hundreds of millions of files. Those files can be a mix of small files, big files, temp files, personal files, and public files. Yet every file s…

It might sound weird but the hard part of what you describe is not the technology but how to design the UX in a way that you aren’t babysitting everything. And doing that is not at all easy. For all anybody knows your cat video is “worth more” to you than your thesis paper. How can you get the system to determine the worth of each file without manually setting an attribute each time you create a file? And if you let…

You could always set the default behavior to be uniform for all files (e.g. protect everything or protect nothing) and just forget about it. But it would be nice to be able to manually set the protection level for specific files that are the exception.

If I was copying an important file into an unprotected environment, I could change how it was handled (likewise if I was downloading some huge video I didn't care about into a system where the default protection was set to high).

I agree that if you have 100 million files, then it could be nearly impossible to classify every single one of them correctly.

Re: Linux NILFS file system: automatic continuous snapshots

#66

Post by DannyBee »

Nifty3929 wrote: post #63

Do any file systems have good, native support for tagging and complex searched based on those tags?

BeFS was the last real one i'm aware of at the complexity you are talking about (plenty of FSen have some very basic indexed support for say file sizes , but not the kind of generic tagging you are talking about)

At this point, the view seems to be "attributes happen in the file system, indexing happens in user space".

Especially on linux.

Part of the reason is, as i understand it, the surface/complexity of including query languages in the kernel, which is not horribly unreasonable

So all the common FSen have reasonable xattr support, and inotify/etc that support notification of attribute changes.

The expectation seems to be that the fact that inotify might drop events now and then is not a dealbreaker. The modern queue length is usually 16384 anyway.

I'm not saying there aren't tradeoffs here, but this seems to be the direction taken overall.

I actually would love to have an FS with native indexed xattr and a way to get at them.

I just don't think we'll get back there again anytime soon.

Re: Linux NILFS file system: automatic continuous snapshots

#67

Post by ComputerGuru »

newcup wrote: post #12

I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014. Apart from one kernel regression bug related to NILFS2 it’s worked flawlessly (no data corruption even with the bug just no access to the file system; effectively it forced running older kernel while the bug was fixed). The continuous snapshotting has saved me a couple of times; I’ve just mounted a version…

> Apart from one kernel regression bug related to NILFS2 it’s worked flawlessly

Maybe on x86? I’ve tried repeatedly to use it on ARM for RaspberryPi where it would have been perfect, but always ran into various kernel panics as soon as the file system is mounted or accessed.

Re: Linux NILFS file system: automatic continuous snapshots

#68

Post by ComputerGuru »

yjftsjthsd-h wrote: post #29
harvie wrote: post #14

I had issues with file locking when running some legacy database software on NILFS2. Probably caused data corruption in that database (not the FS itself). SF website of NILFS2 suggests that there are some unimplemented features, one of them being synchronous IO, which might have caused that issue? https://nilfs.sourceforge.io/en/current_status.html In some cases, the NILFS2 is safer storage for your data than ZFS. So…

> In some cases, the NILFS2 is safer storage for your data than ZFS. What cases? Do you just mean due to continuous snapshots protecting against accidental deletes or such, or are there more "under the covers" things it fixes?

It’s basically append-only for recent things so you theoretically you can’t lose anything (within a reasonable timeframe). I don’t know if the porcelain exposes everything you need to avail yourself of that design functionality, though.

Re: Linux NILFS file system: automatic continuous snapshots

#69

Post by Nifty3929 »

DannyBee wrote: post #66
Nifty3929 wrote: post #63

Do any file systems have good, native support for tagging and complex searched based on those tags?

BeFS was the last real one i'm aware of at the complexity you are talking about (plenty of FSen have some very basic indexed support for say file sizes , but not the kind of generic tagging you are talking about) At this point, the view seems to be "attributes happen in the file system, indexing happens in user space". Especially on linux. Part of the reason is, as i understand it, the surface/complexity of including…

Okay - how about tagging and non-complex searches then. Beggars can't be choosers :-)

Really what I'd like is just to search for some specific tags, or maybe list a directory excluding some tag, or similar. For bonus points, maybe a virtual directory that represents a search like this, and which "contains" the results of that search. (A "Search Folder")

I'll check out BeFS. Thanks!

Re: Linux NILFS file system: automatic continuous snapshots

#70

Post by fuckstick »

goodpoint wrote: post #20
fuckstick wrote: post #8

Earlier quoted context omitted.

> It does without a performance penalty. What is the basis for comparison? Sounds like a pretty meaningless statement at its face.

Compared to other logging filesystems obviously.

Nilfs baseline (write throughput especially) is slow as shit compared to other filesystems including f2fs. So just because you have this feature that doesn’t make it even slower isn’t that interesting - you pay for it one way or the other.
Post reply on HN