TMSU: a tool born out of frustration with the hierarchical nature of filesystems • Hacker News Board

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#81

Post by leanderleeco » Mon, May 09, 2016, 9:05 PM UTC

seagreen wrote: post #77

leanderleeco wrote: post #49

Earlier quoted context omitted.

Haha, I don't want to go too much off topic of this thread, but essentially we can analyze actual file contents, source and metadata to find patterns between file structures and associate them to user and "common" labels. On top of that we try to get further accuracy by using user information to help us find the context. We call it a Personal System, a repo of knowledge and learned preferences heavily tailored around…

I put my email down to spy on you:) My personal feeling is that (A) you're totally right that we need better _personal_ organization systems, but (B) the bottom layer (tagging, schemaing, relationships) should not involve fuzzy processes but should be totally understandable by the average user. I know (B) is a weird opinion though so I look forward to seeing how far you can get with (A) plus machine learning & whatev…

Absolutely, I think (B) is quite a big concern for people. While it's nice to have a fuzzy AI match things _most_ of the time, it's absolutely critical the user can always go in and override and take control of the situation.

So for us, we very clearly distinguish "a user labelled this item" vs "we guessed it was this label".

I'll keep you posted!

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#82

Post by igravious » Mon, May 09, 2016, 9:14 PM UTC

tremon wrote: post #72

igravious wrote: post #59

Earlier quoted context omitted.

I actually wnt to believe you because it accords with my own intuition but I don't trust my intuition on this! Conceptual categorisation + language + ? is totally how we humans comprehend and analyze the world around us, granted. (The devil is in the details of course.) But I don't see how we can go from there to "it relies on a hierarchy of defining attributes" I don't buy your example because the concepts in our he…

Glad to be of (some) help :) I'm not a developmental psychologist btw, just have an interest in cognitive development. The proof you're looking for would be more the field of cognitive neuroscience. There definitely are graph-like connections between concepts all over the brain, I didn't mean to suggest that our brains work hierarchical only, to the exclusion of all other relations. As an example: if our brains were…

I wonder about this a lot.

File systems are arranged hierarchically. Fluke of development? Library catalogues are organised hierarchically. Lucky coincidence? Books themselves are laid out hierarchically. I see a pattern emerging, but is it a trick of the light?

Seems like we like to chunk and splay amorphous informational units into graph-like rooted structures. Maybe cuz of its flexibility?

For something we use all the time and something we do all the time language-use and conceptualisation are deeply mysterious.

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#83

Post by Gibbon1 » Mon, May 09, 2016, 9:36 PM UTC

tremon wrote: post #56

igravious wrote: post #41

Earlier quoted context omitted.

> Humans organise and memorise things in hierarchies Big claim. Proof?

Big proofs. Conceptual categorization is a big part of how we humans comprehend and analyze the world around us, and it relies on a hierarchy of defining attributes, from abstract model to concrete specimen: https://en.wikipedia.org/wiki/Prototype_theory#Basic_level_c... https://en.wikipedia.org/wiki/Stereotype#Cognitive_functions https://en.wikipedia.org/wiki/Concept_learning Maybe less abstract: if you try to remem…

I read this as, humans with 8-24 years of western education forced on them tend to think this way.

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#84

Post by zaphar » Mon, May 09, 2016, 9:58 PM UTC

andrewflnr wrote: post #57

stormbrew wrote: post #38

Earlier quoted context omitted.

That iTunes view is still a hierarchy, though. More accurately, it's still a DAG. The hierarchy is not the problem here, it's the rigidity of trying to put something in only one part of that hierarchy.

Tags are still a DAG, too. I don't see what you're trying to prove here.

Tags are unordered sets that can be viewed in a DAG format if desired. Tag's themselves do not specify a DAG though.

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#85

Post by zaphar » Mon, May 09, 2016, 10:04 PM UTC

ori_b wrote: post #79

alpyne wrote: post #32

Earlier quoted context omitted.

I'd argue that tags provide a more graph-like structure, of which a hierarchy (tree)[1] is just a subset. Symlinks provide an "escape-hatch" to hierarchical systems allowing objects to appear to be in more than one directory. [1] https://en.wikipedia.org/wiki/Tree_(graph_theory)

How so? Unless you can tag tags, you can't represent a tree with them. If you can tag tags, you get an arbitrary graph, with all the problems that represents. I don't think I've seen tagging systems that let me tag tags out in the wild.

You can create a DAG from Tagged objects if you use Set operations. The set of things with tag A includes things from other sets. Those sets form the next level in the hierarchy.

The tagged objects define the relationships between differing tags which allows you to define that arbitrary graph. The graph can be viewed as an effectively infinite depth tree if you wish to.

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#86

Post by zaphar » Mon, May 09, 2016, 10:06 PM UTC

kujenga wrote: post #69

zaphar wrote: post #4

I'm actually not frustrated with the hierarchical nature of filesystems. I'm most frustrated with the state of filesystem search these days. I don't want to tag and curate my files. I want to search them. I've strung together something using bleve full-text search and some OCR libs to scratch my particular itch but it still doesn't quite get all there.

I hear you there! If you're looking for a tool that scratches that itch, I've spent the last year and a half working on a could based search engine that spans all services and devices. We inherit all the information from your existing file system structure, as well as the people files are shared with, and provide a powerful content-based search on top of that. Check out our website if you'd like: https://www.meta.sc

If my files include a scanned in PDF with text that isn't stored as text per se will meta.sc index those? My homegrown solution does this for the most part and that feature was the motivation for creating it.

That and the fact that it's available offline on my machine.

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#87

Post by zaphar » Mon, May 09, 2016, 10:08 PM UTC

tremon wrote: post #73

zaphar wrote: post #4

I'm actually not frustrated with the hierarchical nature of filesystems. I'm most frustrated with the state of filesystem search these days. I don't want to tag and curate my files. I want to search them. I've strung together something using bleve full-text search and some OCR libs to scratch my particular itch but it still doesn't quite get all there.

I'm not a big fan of semantic search itself, nor do I enjoy meticulously tagging my files. But I would love to have different views of my files, based on different metadata. I know that I'm not always searching for files along the same hierarchy: sometimes I want to search along a timeline, because I know that I edited two files around the same time. Or I remember where I was when I wrote something, and I'd like to s…

I don't really need semantic search. Full text search covers 99.9% of my needs.

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#88

Post by Bjartr » Mon, May 09, 2016, 10:10 PM UTC

blakeyrat wrote: post #62

kpil wrote: post #18

Earlier quoted context omitted.

I agree. I actually like the hierarchical organisation, and I don't like the 10 year usability trend, particularly driven by Microsofts attempts to patch over their horrid structure with even worse workarounds. The solution to learning my mother were to find her documents is not hiding the place 15 levels deep and having 10 symlinks to it. Fast, unobtrusive indexing and a good structure is all that is needed. Humans…

> Humans organise and memorise things in hierarchies, Humans also track a lot of things based on spatial memory, which no OS since Mac Classic has even tried to make use of, which is a shame.

I wouldn't say no OS has tried, the windows 8 start screen was designed to exploit spatial memory for instance.

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#89

Post by tambourine_man » Mon, May 09, 2016, 11:05 PM UTC

blakeyrat wrote: post #62

kpil wrote: post #18

Earlier quoted context omitted.

I agree. I actually like the hierarchical organisation, and I don't like the 10 year usability trend, particularly driven by Microsofts attempts to patch over their horrid structure with even worse workarounds. The solution to learning my mother were to find her documents is not hiding the place 15 levels deep and having 10 symlinks to it. Fast, unobtrusive indexing and a good structure is all that is needed. Humans…

> Humans organise and memorise things in hierarchies, Humans also track a lot of things based on spatial memory, which no OS since Mac Classic has even tried to make use of, which is a shame.

True, with the exemption of iOS spring board. It's a great spatial memory comeback and a testament to the easy of use of the model, even if it doesn't scale well when you get into the 100s of items.

Re: TMSU: a tool born out of frustration with the hierarchical nature of filesystems

#90

Post by JoeAltmaier » Mon, May 09, 2016, 11:09 PM UTC

Its been a generation since the hierarchical file system was obsolete. Its lame to have to hang every file on the ceremonial file tree like some Christmas ornament. Hardly any app wants data organized like that.

In fact, nearly every large app does something to avoid it. They create their own representations of a log, or a mail folder, or a document, or an image (and on and on) and manage the details themselves. Because 'file systems' are so lame and underpowered.

This tool begins to help. Creating flexible groupings (tags) resembles a relational database. That's a start. I'd like to replace the OS file system with something like that.

Instead of renaming files when you bring another copy onto your persistent storage, you could just add a version tag to them. Leave the names alone! I can tell my build system (or document store, or mail tool) what version I want to deal with e.g. tag='version' value='2.5'. No collisions any more. No requirement by the 'file system' to mash them into some file tree so they can still be found, but don't collide.

In fact this system can do everything the hierarchical file system can do, and more. Just add the tag 'parent directory name' and voila! You have a file tree (if you want).