I remember DEC/HP releasing the source to the digital unix AdvFS filesystem on sourceforge with the intent of porting it over to linux, but it never materialized. AdvFS had many advanced features. The source is still available and within it are some PDF slides that explain a lot of it's features.
Linux NILFS file system: automatic continuous snapshots
131–140 of 158 posts
Re: Linux NILFS file system: automatic continuous snapshots
#132Earlier quoted context omitted.
> A COW write only needs to checksum the newly written bytes. A non-cow filesystem needs to checksum all data contained in the block (unchanged prior with now new values). No, a COW will re-read the entire block. Make the change and update the checksum and then write it back (to a new location, obviously). Way more than the newly written bytes - but way less than the entire file of course. Just as a non-COW fs will.…
I think you don't understand COW or non-COW filesystems? They don't work the way you are asserting.
I could say the same. Be more specific please. If everything I've said is wrong it should be easy to point out something demonstrably false.
Re: Linux NILFS file system: automatic continuous snapshots
#133Earlier quoted context omitted.
That's what it looked like, but I doubt it was deep in the filesystem. It was basically just a naming convention. User had to purge old versions manually. This gets tedious if you have many files that change often. Snapshots are a safety net, not something you want to have in your way all day long.
Er.. my memory is that it did COW inside VMS fs semantics and was not manually achived. You did have to manually delete. So I don't think it was just a hack. It didn't do directories so was certainly not as good as snapshot but we're talking 40 years ago!
https://en.wikipedia.org/wiki/OpenVMS#File_system
>> DEC attempted to replace it with a log-structured file system file system named Spiralog first released in 1995. However, Spiralog was discontinued due to a variety of problems, including issues with handling full volumes.
Re: Linux NILFS file system: automatic continuous snapshots
#134Earlier quoted context omitted.
> There is no comparison. What if I compare it to BTRFS + Snapper? No performance penalty there, plus checksumming.
About 2 years ago I believed the same. Then I used BTRFS as a store for VM images (with periodoc snapshot) and performance went down to really really bad. After I deleted all snapshots performance was good again. There is a big performance penalty in btrfs with more than about 100 snapshots.
Did you disable CoW on the VM image files? This makes a significant difference to performance on BTRFS
Re: Linux NILFS file system: automatic continuous snapshots
#135Earlier quoted context omitted.
Er.. my memory is that it did COW inside VMS fs semantics and was not manually achived. You did have to manually delete. So I don't think it was just a hack. It didn't do directories so was certainly not as good as snapshot but we're talking 40 years ago!
https://en.wikipedia.org/wiki/OpenVMS#File_system >> DEC attempted to replace it with a log-structured file system file system named Spiralog first released in 1995. However, Spiralog was discontinued due to a variety of problems, including issues with handling full volumes.
The link you want is https://en.m.wikipedia.org/wiki/Files-11
Every file has a version number, which defaults to 1 if no other versions of the same filename are present (otherwise one higher than the greatest version). Every time a file is saved, rather than overwriting the existing version, a new file with the same name but an incremented version number is created. Old versions can be deleted explicitly, with the DELETE or the PURGE command, or optionally, older versions of a file can be deleted automatically when the file's version limit is reached (set by SET FILE/VERSION_LIMIT). Old versions are thus not overwritten, but are kept on disk and may be retrieved at any time. The architectural limit on version numbers is 32767. The versioning behavior is easily overridden if it is unwanted. In particular, files which are directly updated, such as databases, do not create new versions unless explicitly programmed.
Re: Linux NILFS file system: automatic continuous snapshots
#136Earlier quoted context omitted.
you don't need a log structured fs to do this, you could just have regular zfs/btrfs snapshots too. BUT if an attack has the ability to delete an entire file system / encrypt it, they really have the ability to delete the snapshots as well, the only reason they might not is due to "security through obscurity". now, what I have argued is that an append only file system which works in a SAN like environment (i.e. you h…
> if an attack has the ability to delete an entire file system / encrypt it, they really have the ability to delete the snapshots as well, .. How so? Let's say you have one machine holding the actual data for working on it. And some backup server. You could use btrfs send over ssh and regularly btrfs receive the data on the backup machine. Even it they got encrypted by ransomware they wouldn't be lost in the backups.…
I'm not anywhere near a user of btrfs send/recieve api, but I was under the impression that it essentially streams commands and hence could conceptually also be used to delete snapshots on the remote side?
Re: Linux NILFS file system: automatic continuous snapshots
#137we used NILFS 15 years ago in dejaview - https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf We combined nilfs + our process snapshotting tech (we tried to mainline it, but it didn't go, but many of the concepts ended up in CRIU though) + our remote display + screen reading tech (i.e. normal APIs) to create an environment that could record everything you ever saw visually and textually. enable you to search…
Super cool work. Are any tools like this available today? I know some VM tools have snapshotting but full history and high speed scrubbing sounds awesome.
sadly (as with much work form phd students like I was), the closest one could get to it today is trying to duplicate it.
i.e. combining criu with nilfs (but a lot of the work that we did to get process downtime to minimal numbers requires being in kernel, as described in paper) and unsure criu can do it.
In addition our screenrecording mechanism was our own "proprietary" (not really proprietary as fully described in research papers, but also not a standard) and something that was built as an X display driver 15 years ago (so not directly usable today even if code is available). Could probably duplicate it with vnc based screencasting. vnc didn't work for us as we needed better performance (i.e. it was built to demonstrate remote display of video and games and there was no real remote audio setup back then so we had to create our own).
the "text" search just used gnome's accessible API much like a screenreader would do (with a bit of per application optimizations as can filter out things like menus and the like, primarily was to dump text out of terminals, firefox and perhaps open office and maybe even a pdf reader if memory serves me correctly, but a long time ago).
Re: Linux NILFS file system: automatic continuous snapshots
#138Earlier quoted context omitted.
https://en.wikipedia.org/wiki/OpenVMS#File_system >> DEC attempted to replace it with a log-structured file system file system named Spiralog first released in 1995. However, Spiralog was discontinued due to a variety of problems, including issues with handling full volumes.
The link you want is https://en.m.wikipedia.org/wiki/Files-11 Every file has a version number, which defaults to 1 if no other versions of the same filename are present (otherwise one higher than the greatest version). Every time a file is saved, rather than overwriting the existing version, a new file with the same name but an incremented version number is created. Old versions can be deleted explicitly, with the DE…
[deleted]
Re: Linux NILFS file system: automatic continuous snapshots
#139Earlier quoted context omitted.
Super cool work. Are any tools like this available today? I know some VM tools have snapshotting but full history and high speed scrubbing sounds awesome.
sadly (as with much work form phd students like I was), the closest one could get to it today is trying to duplicate it. i.e. combining criu with nilfs (but a lot of the work that we did to get process downtime to minimal numbers requires being in kernel, as described in paper) and unsure criu can do it. In addition our screenrecording mechanism was our own "proprietary" (not really proprietary as fully described in…
Very cool. I could see a system like this:
1. being the layer upon which AI-assisted computing is trained and built
2. being used for video game QA. For certain heisenbugs it'd be invaluable to be able to restore point-in-time repro to resume just before it happened
Re: Linux NILFS file system: automatic continuous snapshots
#140Earlier quoted context omitted.
> if an attack has the ability to delete an entire file system / encrypt it, they really have the ability to delete the snapshots as well, .. How so? Let's say you have one machine holding the actual data for working on it. And some backup server. You could use btrfs send over ssh and regularly btrfs receive the data on the backup machine. Even it they got encrypted by ransomware they wouldn't be lost in the backups.…
I'm not anywhere near a user of btrfs send/recieve api, but I was under the impression that it essentially streams commands and hence could conceptually also be used to delete snapshots on the remote side?
I don't know about the API but as far as I understand the man page, there doesn't appear to be a way to delete a subvolume on the remote side by doing btrfs receive.
That btrfs receive may well be exploitable (because of some bug) if you feed it with malicious data is IMO a different topic.