Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tagsistant: semantic filesystem for Linux

tagsistant.net

Post reply on HN
Print view View original

61–70 of 71 posts

Re: Tagsistant: semantic filesystem for Linux

#61

Post by leoc »

leoc wrote: post #47
nayuki wrote: post #14

Hierarchical file systems lack expressiveness and are awkward in places. In my day-to-day computing this has become more apparent and problematic with each passing year. For example, I dislike that you are forced to give unique names for each file, that you can classify each file in only one way, and that you can't tell how many copies you own of a piece of data. The smallest step up from a hierarchical file system i…

Well, since you ask, here's Hans Reiser's old stuff: https://reiser4.wiki.kernel.org/index.php/Future_Vision https://reiser4.wiki.kernel.org/index.php/V4 (and http://lwn.net/2001/1108/a/reiser4-transaction.php3 ) . And here's some emails etc. I wrote in response: https://web.archive.org/web/20040728044342/http://www.st-and... https://www.mail-archive.com/reiserfs-list@namesys.com/msg09... https://www.mail-archive.com…

I forgot to include this email of mine https://marc.info/?l=linux-kernel&m=111624697710426 , probably the most important of mine.

Re: Tagsistant: semantic filesystem for Linux

#62

Post by wruza »

eriknstr wrote: post #44
nayuki wrote: post #14

Hierarchical file systems lack expressiveness and are awkward in places. In my day-to-day computing this has become more apparent and problematic with each passing year. For example, I dislike that you are forced to give unique names for each file, that you can classify each file in only one way, and that you can't tell how many copies you own of a piece of data. The smallest step up from a hierarchical file system i…

I too have been thinking about alternatives to hierarchial filesystems. Originally I thought that I wanted a tag based system, but currently I believe that tagging adds too much overhead and that it might not scale well either for the user since the more tags you have the harder they will be to remember and then there is the issue of relationships between tags. On macOS they have a view that shows you all your files…

I can't understand why msbob-like interfaces failed. The idea is simple and natural: you don't have hierarchical folders with names, you have visually recognizable places. Even very non-technical users can understand this idiom when you just put papers somewhere to get back to it later. People are very good at locations, very bad at names. One of the best remembering techniques is based on locations.

That's actually how I organize my physical desktop: there is a stake of blanks ("new text file..." menu item), operative drawings in the center, non-operative at the far right, and lot of drawers in my table, each with its own geometry, cabinet, shelves, working/entertainment rooms or areas. If I start to collect something new, like DVD discs or project documents, I simply "create" new shelf for archiving or empty old space by moving it to less useful location.

Given that it is electronic representation, you may even put document into additional location, like dragging it via alt key, so semi-transparent version of it is created. You can take documents to your hands, go to places, drop some here and some there, instead of that cut-paste and "clipboard" idiom. Why would one really move folders via cutting them to the clipboard? Did you ever see what it looks like?

Even if hierachical folder/file systems could be good, today these are shit. You can edit file, but you can't edit folder. Attach photo or text preview on file/folder via paperclip? Make it bigger, so that it stands out in other icons? Create in-folder heading like github's README.md? Colorize them quickly? Nope. Even when possible it is a real pain. All files and folders look the same (or random) and if it is not jpeg, you're on your own to search it by low-perceptible metadata like ctime or name/type.

Re: Tagsistant: semantic filesystem for Linux

#63

Post by dredmorbius »

Chris2048 wrote: post #60
dredmorbius wrote: post #59

Earlier quoted context omitted.

Good point on mount counts. Doesn't that problem already exist to an extent for remote FS mounts (NFS, etc), especially over automount?

Yep, if the external mount is large. However, it's a little different for an fs system that can quickly (w/o network-speed limits) generate recursive fs structures. for example, what if a created a vfs that created a 'foo' folder, with a 'foo' folder inside, and so on. The system would crawl an infinite descent of /foo/foo/foo/foo... and so on, which would eventually fill some cache or another.

Good thoughts, and I'm thinking of possible hazards and pitfalls of the approach.

There's already a concept of setting flags to avoid traversal of certain filesystems, and given the proliferation of virtual and networked filesystems, this seems useful: /proc, /sys, /dev, /udev, and a few others (it's getting to the point I don't recognise a full mounts listing readily anymore under Linux).

With the concepts I'm considering, in particular, of the tree as being essentially search traversal rather than a static filesystem (not even one with lots of symlinks all over the place), the potential to create some deep dives or recursive tangles is pretty high.

Another example: tools such as 'locate' or Spotlight shouldn't attempt to traverse this tree when generating indices. Instead, they should query it when requests are made.

Information and metadata leakage is another key consideration when moving data off local host and/or caching among hosts.

Re: Tagsistant: semantic filesystem for Linux

#64

Post by Chris2048 »

dredmorbius wrote: post #63
Chris2048 wrote: post #60

Earlier quoted context omitted.

Yep, if the external mount is large. However, it's a little different for an fs system that can quickly (w/o network-speed limits) generate recursive fs structures. for example, what if a created a vfs that created a 'foo' folder, with a 'foo' folder inside, and so on. The system would crawl an infinite descent of /foo/foo/foo/foo... and so on, which would eventually fill some cache or another.

Good thoughts, and I'm thinking of possible hazards and pitfalls of the approach. There's already a concept of setting flags to avoid traversal of certain filesystems, and given the proliferation of virtual and networked filesystems, this seems useful: /proc, /sys, /dev, /udev, and a few others (it's getting to the point I don't recognise a full mounts listing readily anymore under Linux). With the concepts I'm consi…

Yep, in fact I think a lot of vfs are like this - doesn't /proc/ query the kernel when returning parts of its tree (e.g. representing processes).

wrt the fs representing queries: one of the tag systems is like this, it represents tags as folders, and the search for term 'X' and 'Y' can be found under either folder ./X/Y/ , or ./Y/X/ ; obviously, every new tag combinatorically expands the virtual space.

Re: Tagsistant: semantic filesystem for Linux

#65

Post by Pxtl »

adamrezich wrote: post #58
nerdponx wrote: post #54

Earlier quoted context omitted.

As long as the tags themselves can be hierarchical, can't you recover all the benefits of a tree-like file structure?

This. I believe a hierarchy of tags is a great solution.

But then don't you have the same probelm of the limits of taxonomies and you start thinking "well is this tag a member of A or B? It should be a member of both" and then you need to tag the tags. Turtles all the way down.

Re: Tagsistant: semantic filesystem for Linux

#66

Post by dredmorbius »

Chris2048 wrote: post #64
dredmorbius wrote: post #63

Earlier quoted context omitted.

Good thoughts, and I'm thinking of possible hazards and pitfalls of the approach. There's already a concept of setting flags to avoid traversal of certain filesystems, and given the proliferation of virtual and networked filesystems, this seems useful: /proc, /sys, /dev, /udev, and a few others (it's getting to the point I don't recognise a full mounts listing readily anymore under Linux). With the concepts I'm consi…

Yep, in fact I think a lot of vfs are like this - doesn't /proc/ query the kernel when returning parts of its tree (e.g. representing processes). wrt the fs representing queries: one of the tag systems is like this, it represents tags as folders, and the search for term 'X' and 'Y' can be found under either folder ./X/Y/ , or ./Y/X/ ; obviously, every new tag combinatorically expands the virtual space.

Right. My vision is of a /docfs under which you might travers /au/stephenson/ti/snowcrash or /ti/snowcrash/au/stephenson, as an example. Either way works.

If a search terminates in multiple (or no) results, it's a directory, if in a single result, it's that file. Plus a few twists (virtual / dynamically generated file formats, summaries, synopses, metadata).

Re: Tagsistant: semantic filesystem for Linux

#67

Post by Chris2048 »

dredmorbius wrote: post #66
Chris2048 wrote: post #64

Earlier quoted context omitted.

Yep, in fact I think a lot of vfs are like this - doesn't /proc/ query the kernel when returning parts of its tree (e.g. representing processes). wrt the fs representing queries: one of the tag systems is like this, it represents tags as folders, and the search for term 'X' and 'Y' can be found under either folder ./X/Y/ , or ./Y/X/ ; obviously, every new tag combinatorically expands the virtual space.

Right. My vision is of a /docfs under which you might travers /au/stephenson/ti/snowcrash or /ti/snowcrash/au/stephenson, as an example. Either way works. If a search terminates in multiple (or no) results, it's a directory, if in a single result, it's that file. Plus a few twists (virtual / dynamically generated file formats, summaries, synopses, metadata).

Of possible Interest is the new article:

https://news.ycombinator.com/item?id=14550060

and http://www.sqlite.org/src/doc/trunk/src/test_onefile.c for an idea of turning SQlite into and actual vfs.

Re: Tagsistant: semantic filesystem for Linux

#68

Post by nayuki »

clumsysmurf wrote: post #22
nayuki wrote: post #14

Hierarchical file systems lack expressiveness and are awkward in places. In my day-to-day computing this has become more apparent and problematic with each passing year. For example, I dislike that you are forced to give unique names for each file, that you can classify each file in only one way, and that you can't tell how many copies you own of a piece of data. The smallest step up from a hierarchical file system i…

The article mentions a few technologies like IPFS, so I'm just mentioning Camlistore for completeness because its in this space too. https://camlistore.org/ I'm reading "The Science of Managing Our Digital Stuff (MIT Press)". They seem to prefer hierarchies (I'm not far through the book yet). https://www.amazon.com/Science-Managing-Digital-Stuff-Press/...

I took your suggestion and explored these two new references. They were quite relevant, thanks.

Thoughts on Camlistore: I agree with their high-level https://camlistore.org/doc/principles and https://camlistore.org/doc/uses . Their presentation slides and videos gave a helpful explanation and demonstration of their functionality. The query string format and showing of live search results were very cool. I have doubts about the rich JSON metadata format, their model of mutable files, and whether I can represent and query the my kind of metadata in their system.

Thoughts on "Science of Managing Our Digital Stuff": I sat down at the public library and read part of the book. Your early warning about hierarchies proved correct. The authors seem to be very focused on conducting user studies and timing people's time and recall performance. All their text point toward the superiority of hierarchical organization due to the efficiency of human spatial navigation / folder traversal. The book has some interesting perspectives to offer (e.g. human behavior, group information management, motivations behind keeping data), but I don't expect it to contribute to any of my technical design decisions.

Re: Tagsistant: semantic filesystem for Linux

#69

Post by nerdponx »

Pxtl wrote: post #65
adamrezich wrote: post #58

Earlier quoted context omitted.

This. I believe a hierarchy of tags is a great solution.

But then don't you have the same probelm of the limits of taxonomies and you start thinking "well is this tag a member of A or B? It should be a member of both" and then you need to tag the tags. Turtles all the way down.

That's the advantage of tags vs true directories. A tag could have mutiple parent tags.

Of course, that raises the question of what happens when you encounter a cycle in the tag graph.

Re: Tagsistant: semantic filesystem for Linux

#70

Post by adamkruszewski »

Great work OP! Had done something similar as my master thesis in 2005 basing on one of Hans Reiser's papers. It was more of a quick-hack having the code written in three days so not much have survived to this day, but you can see some of it in action on an adobe flash[1] video at https://adam.kruszewski.name/assets/static/mtfs-demo.htm

[1] don't judge me, mp4 in the browser wasn't main-stream back then ;)

edit: for those without flash like me -- even as a quick hack it had manual and automatic file tagging (based on file metadata) and you could query it using logical expressions. It also had pretty nasty memory leak I didn't care to find out :) Still, without a first-class, built-in support from file managers like BeOS had for its filesystem the idea is not fully realized I think.

Post reply on HN