Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Designing better file organization around tags, not hierarchies (2017)

nayuki.io

Post reply on HN
Print view View original

151–160 of 169 posts

Re: Designing better file organization around tags, not hierarchies (2017)

#151

Post by jessmartin »

TeMPOraL wrote: post #99
jessmartin wrote: post #54

Have you considered a file system organized as a timeline that _also_ supports tagging? I find one of the key concepts that's not a first-class concept is _when_ the file was modified. Rather than a file-and-folder physical analogy for the file system UI, I think a timeline-oriented UI could present some advantages for the way that humans actually think and work. Tags would be a helpful orthogonal organization scheme…

Not sure if it generalizes for the entire filesystem - not all files are modified due to explicit user action. Software keeps logging all the time. Many applications update their config files when closing, or during runtime. Saving a document in a program may trigger saving another 3 files elsewhere. All in all, it seems like a recipe for seeing the least relevant data first.

You've highlighted one of the primary challenges with such a filesytem. Determining what files have been modified as a _meaningful user action by the user_ (determined by the user thinking it was meaningful) and _what changed_ would be a very hard, but incredibly valuable problem for the user.

I think fundamentally the success and adoption of a timeline-oriented filesystem leans on new user experiences that have yet to be designed.

Re: Designing better file organization around tags, not hierarchies (2017)

#152

Post by chadcmulligan »

ausjke wrote: post #128
chadcmulligan wrote: post #119

Earlier quoted context omitted.

make an enemies of ausjke list probably

nope, just ask why? you're what you think I guess, I have not made any lists in my whole life and surprised you immediately guessed that.

its a line from an old Steve Martin movie (Dead mean don't wear plaid) - thought it might be to obscure :-)

Re: Designing better file organization around tags, not hierarchies (2017)

#153

Post by catenate »

Since 1 June 2012, I've been taking notes in unicode text files, which contain (occasional or adjacent) lines starting with 'nb ' and then a list of tags. I wrote a simple tool ("nb") in Inferno's shell (thanks to Robert J. Ennis for the port to Plan 9's rc), to (1) search for given keywords in per-directory index files pointed to by the global index, (2) index all of the nb lines in files in the current directory, and (3) if necessary, append, to a global index file, a reference to the index file in the current directory.

https://github.com/catenate/notabene

I've found that I'm comfortable with the eventual consistency this offers, in exchange for fast lookups when I want something (as opposed to indexing first, and/or indexing globally, and so waiting for indexing to get a result). This distributed-file approach also allows me to add tags to a variety of files: local files, or networked file-system files, or sshfs-mounted files, or Dropboxed files, or files under version control, or files with varying text formats; and find tags across all of them and across all the time I've been indexing.

It runs in linear time with respect to the number of tags I've entered, plus the time to read and process the global index, so obviously there are many ways I could improve the time performance (as an easy example, I could permute the index to list all the tags in alphabetical order, and next to each tag list the files that contain that tag).

I also wrote other tools, since the layout is so simple: for example, "nbdoc", to catenate the actual contents of the references returned by the primary tool (nb); and "so" (second-order), to return all the tags which appear in any nb line with the given tag(s).

I've also found that it's not easy for me to remember what tags I might have used in the past, or how I was thinking about something, so I try to use the conjuction of several tags to narrow down search results, rather than try to remember one specific tag (this seems to correspond to the observation that it can be difficult to remember exactly where in a hierarchy you put something).

The modular approach, of per-directory indexes referenced in a global file, also makes it easy for me to combine work-specific notes, with public notes, with private notes, all in the same global index file, at work; but only have the same public and private notes at home.

Re: Designing better file organization around tags, not hierarchies (2017)

#154

Post by xcvbxzas »

lambda wrote: post #148
xcvbxzas wrote: post #61

Earlier quoted context omitted.

I think many of these issues can be addressed with mechanisms proposed by the author. Mainly, the more complex tags which can themselves refer to other tags. 1. This is probably the trickiest one. You may be able to do some sort of translation between a hierarchical system and the tag system using tags themselves. You could have a series of tags that refer to each other, such that the hierarchical location is essenti…

I think you're missing some of the subtleties of solving these problems using "just more tags." In a hierarchical system, a lot of these organizational issues are local. If I have one directory that consists of a project organized one way, and another directory that consists of a different project organized a different way, those different organizations don't really interact with each other in any way. If you are usi…

I'm sure you're right that there are a lot of overlooked subtleties. That said I'm not sure some of those problems you mentioned would exist, or at least I'm not sure they would be any worse with a tag system than a hierarchical one.

For example, how is that example query any worse than the current situation? Right now you'd navigate to the project directory (requires specifying more than your example already) and then use some search method depending on OS/WM/etc. And then you still end up with a big list of jpegs to look through. This is sort of a worst-case example for both systems, and still I think the tag system comes out ahead here - by a little - just because it would give you the ability to spread the project across multiple drives without requiring you to do two searches if you don't know which drive the desired image is on. You can improve the situation for either system by manually specifying more information. Put better tags on the images or put them in more specific directories or title them.

As for specific applications, it's not the metadata encoded into the files that I'm talking about. It's as simple as the directory structure itself that is used to store all of this. I can't have one application organize everything and then trivially point another application at the directory and have it work.

With a tag-based system this starts to change. I don't need to tell a new music player where my music is, and then go through whatever process is needed to let it properly work with the current directory organization. At worst I tell it which tags to include or perhaps exclude. From there many options exist. Maybe it pulls in metadata from the files themselves. Maybe I provide an external file in whatever format. Maybe I tell it which tags to associate with which fields. You could do a lot of things here.

I also won't end up telling the application to reorganize things as I did many years ago with iTunes, which promptly made it nearly impossible to wade through my music manually. I had it sort everything into directories based on the artist with subdirectories for albums. It sounded great, until I remembered just how much music I had off OCRemix, where an album is a large collaboration between many people. All of those albums were ripped apart. Ironically, I also had some standardization issues with things like artist names which caused more trouble. Once I stopped using iTunes I basically abandoned that collection because of the work required to fix it.

Yeah, standardization is going to be sort of a problem, but I don't think it's quite as big of a deal as you think. For one, the OS is going to ship with a bunch of standard tags just for itself to work. There will also just be a lot of really standard stuff people are interested in that can be shipped with them. You also have file extentions, for both specific extentions and also generally what kind of information they contain. And finally there is just good old translations. The hierarchical system basically utilizes all these methods and suffers from the same problem - namely you can put directories wherever you want and name them whatever you want. Same problem, different manifestation.

I think the biggest benefit would come from a system that can present itself either hierarchically or tag-based. They both have merits. I've already presented some ideas on how you could store the hierarchical structure in the tags. I'm not so sure how you store the tags in a hierarchical system directly. You could probably fake it with a separate datastore easily enough though.

Finally, when did this discussion of general design goals turn into one of a real-world implementation, much less widespread adoption? I'm not sure how this is relevant.

Re: Designing better file organization around tags, not hierarchies (2017)

#155

Post by fao_ »

kazinator wrote: post #146
fao_ wrote: post #11

Earlier quoted context omitted.

> country and folk music Folk music has nothing to do with country. Country was invented in the 1800s by a businessman who aimed it at primarily white Americans. Folk music has a rich history dating before the 1200s and before much of written language. Many songs of Irish and Welsh descent date before record, still being played today. Nevertheless, to answer your question: 1) metatags, parent tags, and the like provi…

"Folk music" possibly rides on the definition of "folk", which brings in a lot of diversity if interpreted broadly. The word "folk" also denotes recognizeable format in the context of commercial broadcasting and streaming of canned music. It basically refers to a locus roughly centered around someone crooning while strumming chords on an acoustic guitar.

> The word "folk" also denotes recognizeable format in the context of commercial broadcasting and streaming of canned music.

That's a very american-centric view you have there.

Re: Designing better file organization around tags, not hierarchies (2017)

#156

Post by kazinator »

fao_ wrote: post #155
kazinator wrote: post #146

Earlier quoted context omitted.

"Folk music" possibly rides on the definition of "folk", which brings in a lot of diversity if interpreted broadly. The word "folk" also denotes recognizeable format in the context of commercial broadcasting and streaming of canned music. It basically refers to a locus roughly centered around someone crooning while strumming chords on an acoustic guitar.

> The word "folk" also denotes recognizeable format in the context of commercial broadcasting and streaming of canned music. That's a very american-centric view you have there.

That isn't what you might call my "view"; I'm just remarking on how it seems that a word happens to be used in a certain culture and context.

Re: Designing better file organization around tags, not hierarchies (2017)

#157

Post by sturakov »

reacweb wrote: post #107
nayuki wrote: post #91

Hello everyone, thank you for all the comments. Seeing this on the HN front page caught me by surprise. In the past year I shared this article publicly (Reddit) and privately (with tech-savvy acquaintances) for comment, and the general sentiment I received was that these ideas were not ready to be read by a mass audience. The article is way too long and pulls in many disparate ideas; it explains both why traditional…

Hello, I am trying to make an application for my wife to manage embroideries. I encounter almost all your issues. My wife has thousands of embroideries downloaded from internet. There are many duplicates (filenames not unique because of internationalisation and special characters). She needs to add tags to help search. She also needs groups of tags (tiger belongs to animals, ...). She also has metadata (origin of the…

If she is using Windows 10 (or similar), I've found http://tabbles.net/ is a pretty cool solution.

Re: Designing better file organization around tags, not hierarchies (2017)

#158

Post by nayuki »

enobrev wrote: post #132
TeMPOraL wrote: post #126

Earlier quoted context omitted.

> And in the case of search, it absolutely sucks on today's filesystems. It does. You mention grep, I'd even mention find - half of the time I'm wondering whether it has searched everything I wanted, or I misspelled the command. Or file search in Windows (Vista+) - I just don't trust it; I'm pretty sure it missed some data in the past for one reason or another. Now with traditional file systems, I at least have the f…

> I'd only have search - so it better be trustworthy Absolutely agreed. It should be as reliable and as immediate as what we have now. > I at least have the file tree I don't know how this would work in practice, but I'm imagining something where, UX-wise, a tag-based FS could act very much like what we're already used to. Google was very much on this track in their early versions of "labels" in gmail and google driv…

I like the content of your answer: filtering by tags to narrow down the search results, only showing tags that belong to the current set of results, the benefits of order-insensitive path parts, and the ease of taking unions of tag results.

The path examples that you created are Boolean queries with different symbols: slash means AND (low precedence), plus means OR (medium precedence), and exclamation means NOT (high precedence). Your last example could be rendered as "2017 AND receipts AND (business OR NOT client_3)" and mean the same thing.

In any case, the illustration you made is indeed the sort of user interaction that I want to design into a future prototype.

Re: Designing better file organization around tags, not hierarchies (2017)

#159

Post by dTal »

TeMPOraL wrote: post #97
enobrev wrote: post #89

Earlier quoted context omitted.

I absolutely agree with this, except to add that it seems, in theory, in the case of a 100% tag-based file-system, there would never (*very rarely) be a flat list where you have to scroll through millions of files. The UX of a single flat list with millions of files named DCIMxxxx.jpg is a limitation of the current format and doesn't make sense when so much information can be generated about our files upon creation.…

Part of the adoption problem here would be trust. Hierarchical filesystems are something we're used to, and we can trust they're implemented correctly. That means, if I visit a folder and see some files, I know what I see is all of the files there (+/- hidden file settings); if something is missing, it's not there, period. Tag search is a search. Can be broken. Can be optimized in a way that causes it to lie. I look…

>if I visit a folder and see some files, I know what I see is all of the files there (+/- hidden file settings); if something is missing, it's not there, period.

Funny you should say that. Only a few weeks ago a colleague of mine was perplexed by a file which showed up in a 'save as' box, but not in Windows Explorer. It was an ordinary log file, same as a bunch of others in that folder, no reason for it to be different. Apparently he later discovered the file was visible if navigated to from C:, but not through the desktop shortcut he'd made to that folder. We could only conclude it was a Windows bug. Whatever the cause, it wasted a good deal of our time hunting for that file...

Re: Designing better file organization around tags, not hierarchies (2017)

#160

Post by xtiansimon »

reacweb wrote: post #107
nayuki wrote: post #91

Hello everyone, thank you for all the comments. Seeing this on the HN front page caught me by surprise. In the past year I shared this article publicly (Reddit) and privately (with tech-savvy acquaintances) for comment, and the general sentiment I received was that these ideas were not ready to be read by a mass audience. The article is way too long and pulls in many disparate ideas; it explains both why traditional…

Hello, I am trying to make an application for my wife to manage embroideries. I encounter almost all your issues. My wife has thousands of embroideries downloaded from internet. There are many duplicates (filenames not unique because of internationalisation and special characters). She needs to add tags to help search. She also needs groups of tags (tiger belongs to animals, ...). She also has metadata (origin of the…

Your description is very similar to a problem I have with academic papers as PDFs. I was toying with `rsync` solution and naming conventions for original file names (as they were named when acquired) and renaming after review (EG. using the file system). Organizing the files with tags, collections would be a great improvement.
Post reply on HN