TMSU: a tool born out of frustration with the hierarchical nature of filesystems

ding ding ding! This is the monkey in the wrench as it were.

Tagging is a really useful idea, it is also a naming thing and as such either it lives in the naming infrastructure (aka dirents) or it rots over time. A simple example I used to use in the 'object naming' [1] days was, imagine that instead of house numbers on the street you wrote down last names. That works fine until somebody moves and now not only did you show up at the wrong house, you don't even have a chance of knowing what the correct house is. [2]

Microsoft's LongHorn project was way out there but took a swing at the actual problem. Just make the file system an actual relational database. Then your home directory is simply 'select * from files where (owner = chuck);' It really does solve the problem at a more fundamental level, using naming by attribute rather than mapping. I got to observe that effort from the outside (I was at NetApp at the time) but I believe it died due to really horrible performance issues.

I find it pretty awesome that people can lose files, back when a "big" hard drive was 100MB it really wasn't all that hard to just look through all the files on it, but when its a couple or three terabytes, all bets are off!

[1] Object File systems were all the rage in the early 2000's, files themselves were object ids and the naming was a database that connected object ids to user recognizable names. -- https://en.wikipedia.org/wiki/Object_storage

[2] The typical solution is to add "tombstones" or redirects at the previous address. That then is a layer of additional meta data to maintain, and sometimes the file doesn't move, it just changes value (trivial example you have a file 'my-favorite-song.mp3' which is tagged 'jazz mp3' and then you discover techno and make something from Tiesto your favorite song and while the name and type are still valid, the tag 'jazz' is now invalid.

The other problems I was alluding to weren't really with FUSE, but one that does pertain to FUSE is speed, since FUSE imposes overhead through a daemon running in user space, and associated context level switches because of that. From just looking into is again, this may have been mitigated to some larger or smaller degree with some FUSE performance enhancements in 2012.

Specifically, I was referring to the different off the shelf database systems which could be used. Each will have it's own benefits and drawbacks to storing large chunks of data per-record. Benefits might include (relatively) easy sharding or replication. Drawbacks might include not being space efficient for removed files, not being as resilient to corruption due to crashes or corruption affecting more than the files in use, or overly aggressive use of memory to function efficiently.

If a custom database was developed, you could tailor to your exact needs, but then you have much more work to do, and a period of immaturity.

Off the top of my head, if I were designing a general purpose system for tagging files where people were expected to use it as a regular file system and some overhead from FUSE was acceptable, I think I would leverage the file system but in a different way. I would set up a specialized directory for the files themselves, and store then hashed within it, and have a BerkelyDB database relate filename to hash and tags, and use FUSE to do direct file access. But that's my 5 minute assessment, so I reserve the right to change it completely given someone pointing out the obvious problems. :)

This supplies you with only one relation between the tags: "And Then". Ex. 'Path' (and then) 'to' (and then) 'summer.mp3'. Or 'tags' (and then) 'big-jazz'.

What you want is to also have AND, OR and NOT available. How would you find 'music' (and) 'mp3' (and not) 'big-jazz'?

This is an interesting idea, but it seems like an awful lot of work on my part to go through and organize and tag every file. While some of it could be automated (pulling ID3 tags out of MP3s), a lot of it seems to depend on me figuring out good names for everything I make.

The biggest problem is that I have to figure out which tags are going to be useful to me in the future and where to add them. This is relatively easy for music (but even there can explode in complexity depending on how granular you want to be), but more difficult for things like photos or papers.

IMHO, fully general tagging systems never work because the complexity explodes as the number of potential tags increases. You need to narrow the scope down to a specific domain so your tags can be limited to human scale.

This is really complicated