Dotfile madness

Apps use dotfiles or dotdirs for caching data. Think browser caches. That's the point of the standard... All the non essential caching data lives in one place and so is easy to prune when needed.

not sure if this is the exact solution to your problem, but rcm (https://github.com/thoughtbot/rcm) allows you to create dedicated dotfiles in a .dotfiles directory which are being symlinked back to files, and it has a -t option to create and restore dotfiles with tags. mkrc -t emacs .emacs.d would create a .dotfiles/tag-emacs/ folder and rcup -t emacs restores it.

Associating tags with files is straight forward.

What's not quite so straight forward is how you query the system. The worst implementations of file tagging only implement retrieving a list of all files that have a given tag. Slightly better than this are tagging systems that will return the intersection between two or more tags. Most tag systems never go beyond this.

Going slightly further, tag exclusions are powerful and are sometimes implemented (given a set of files from some subquery, exclude all files that have a given tag.) What you rarely see are systems that allow you to exclude one subquery from another.

However what you almost never see implemented is a system for preferential tags; given a subquery, reorder the results such that files with preferred tags are raised to the top, ordered by how many of the preferred tags they have. Once you implement this, the system's UX changes dramatically because the user no longer has to make strong assumptions about how well their files have been tagged. Many files might be missing relevant tags and the user may not be sure if the file they're after is one of these incompletely tagged files. When using a system with preferential tag querying, the user will receive files in their result list that don't match all the tags listed, but most of them. This is a bigger deal than it may sound, since the main drawback of using tags for file management is incomplete tagging. By addressing this drawback, you stand to bring file tagging to the next conceptual level, which is rendering hierarchical file management obsolete.

Consider that hierarchical file management is a strict subset of file tagging. You can model file hierarchies inside a file tagging system. To demonstrate this by example, consider /home/joesixpack/Documents/seventh-novel.pdf For each level of the hierarchy, we can create a new tag, such that this document has the tags: '/home/', '/home/joesixpack/', '/home/joesixpack/Documents/'. But because we're using file tagging, we can also automatically tag that file with things like 'pdf' or maybe even 'Documents'.

Now before I go on, there is something to be said about the number of tags in the system exploding when you model a tree as a tagset. In practice this probably isn't the approach any real system should take, if only because most of those tags will be useless and because there is a great deal of redundancy in trees that wouldn't exist in a native filetagging system. Consider /home/joesixpack/Documents/ and /home/johnsmith/Documents/. We have two different Documents directories because they exist in different parts of the hierarchy. However in a file tagging system we'd ideally only have a single Documents tag and one tag per user, such that querying the intersection between 'joesixpack' and 'Documents' returns the files that would otherwise be in /home/joesixpack/Documents. Some tags, such as the user tag, could be implicit; such that if joesixpack simply queries 'Documents' the tag 'joesixpack' is automatically intersected with the results presented to him.

With that out of the way, consider how we can go further if we apply straight forward statistics to the tag system. Suppose 99% of joesixpack's 'Documents' that are 'pdf' are also 'novel'. When joesixpack creates a new file that's in the tags 'Documents' and 'pdf', what are the chances that file should also be tagged with 'novel'? We could analyze the contents of the file if the system has a semantic understanding of what 'novel' means, but let's not go there. Tag systems created by the likes of Facebook and Google do this sort of analysis, but it's heavy, tricky, goes wrong in ways that create PR disasters, etc. We can get pretty good results by ignoring the contents of the file and looking instead at merely what that file is already tagged with. If a file is tagged with 'joesixpack', 'Documents', and 'pdf', it _probably_ should be tagged with 'novel' as well. Probably, but not necessarily. So the system can expose to joesixpack the suggestion that he tag that particular file with 'novel'. By presenting suggestions like that to the user, you greatly reduce the UX friction needed to tag files well and therefore increase the usability of the tagging system, while at the same time improving the quality of the tag suggestions in the future. What we create here is a 'virtuous cycle' of sorts that gives back to the user more as it's used more. Such a tag suggester can be implemented as multi-label classification using naive bayes; it's fairly straightforward.

Going further; if we have such a tag suggester, and an appropriate caching scheme, tag suggestions can be used by queries. If joesixpack queries for 'pdf','novel', the system could return to him the intersection of 'pdf','novel', but also return to him files that are tagged with 'pdf' but not tagged with 'novel', in cases where the tag suggester indicates there is a high likelihood that the file should be tagged with 'novel'.

That last paragraph may or may not play out well, I haven't experimented with it extensively yet. But getting back to my original point: if you're going to implement file tagging in a filesystem, you should think carefully about how the users will query that system, and whether the query system will be extensible in userspace to facilitate new powerful methods of querying it. It would be a total tragedy if the system only supported basic tag intersections and the only way to extend it was to implement a new kernel module.

This kind of just sounds like more standard crazy unix/linux "someone did the wrong thing and now everyone's doing the wrong thing and it's too late to realistically fix", for which it's easier to just come up with another subversion (waves at the three different 'user' solutions) like not just building a user home dir, but creating a dir inside that the user's usename as well, so that ~ is for all the bullshit dotfiles and ~/yourusername is actually your data. And then someone will no doubt come up with an alias that takes you to that dir instead of just the original homedir, like "cd `" instead of "cd ~" (and no, ~ doesn't make any more sense than ` would, it's just what we got used to)

Suggestion. Just do:

    mkdir ~/real_home

And use that as your real home :)