The big central problem here is that non-trivial taxonomies aren't trees but graphs. Trying to get a tree-based filesystem to represent a taxonomy means you're forcing a graph into a tree. Symlinks help because they turn your tree into a graph (albeit one that breaks too easily; I think that could be fixed, though). But in the end, a traditional filesystem is a poor way to represent a taxonomy.
Taxonomy Is Hard
21–30 of 83 posts
Re: Taxonomy Is Hard
#22In my "personal digital garden evolution" I've reached the timeline-level taxonomy or, with Emacs/org-mode/org-roam/org-attach etc:
- new textual entries (headings) goes into monthly notes ($org-roam-directory/timeline/year/$month-name.org), they might attach files or not of course. Attached files are generally directly linking in the heading/inside the textual content of the heading for a single-click quick access and glance view. Doing so allow to have not too many too small files, not too big ones who operate slowly;
- another subdir of org-roam-directory is for "topics", a note per topic, linking or org-transcluding (slow and a bit limited but still useful) the collected entries in timeline style;
- another is workdir where I craft my catalogue (using org-mode drawers created with templates to allow easy org-ql queries) and queries to explore my notes in different view. It's not as easy as TiddlyWiki transparent transclusion but allow a certain degree of practical usability, fine grain selection and ease composition.
MOST of my files and config live or as org-attachments or tangled from org-mode. So yes, taxonomy is hard, but we have tools to master them IF we decide to discover them and invest time in improving our digital garden for real instead of leaving classic mess of files hoping for some miracle "application" that solve automagically all issues. Unfortunately due to the lack of interests by most leave such systems too little developed to be as effective as they can...
My personal experience is:
- we need taxonomy anyway, just mere full-text searching with extras à-la-google do suffice for a certain percentage but fails more than that;
- we need taxonomy that are a bit flexible in storage terms and can change at a slow peace;
- we need integration, witch is NOT possible in ALL modern software, we need for that classic desktops where the OS was a framework/live image and anything is just a module, a bit of code, of it. With end-user programming concepts because no UI can be effective enough in "no code" style and no "modern programming" styles are usable for user programming.
A bottomline: people should learn a bit about information management at school, from how a library or a pharmacy organize books/meds on their shelves to book's indices and personal information archives. Nothing exaggerated but the bare minimum to understand how to manage data, digital and physical in various forms for a lifetime...
Re: Taxonomy Is Hard
#23All true. Taxonomy is indeed hard. But, does it actually matter? It seems what matters is not how files are stored/organized, but how one can find the files one is looking for. Taxonomy is mostly a search problem. Yet, although one is usually capable of remembering specific or unique details about a file, it's still incredibly hard to search for a file or its contents effectively. Dropbox, to pick just one example, d…
I'd guess only a small portion of people actually care for that feature. People are maybe better at remembering what's in their files by name alone than we give them credit, or willing to just look through things to find them, or are better organized via folders than we might assume.
Important for technical users, not so much regular people.
Re: Taxonomy Is Hard
#24Tagging, seems like a solution. But isn't.
Specific problems with tagging: - Need to tag every file (whereas in folders, you just navigate to the folder and everything you store there is in that folder) - Takes too long - Too much thinking overhead (at the time of storing) - To be effective have to enter the name for all tag entries (e.g. project, type, etc.). If anything is missed for a file, that file will never be found. - You have to remember what tag categories (e.g. project, type, etc.) you have used. If you don't use in 3 months, now you have forgotten. - You have to remember the enumeration you are using for some tag categories (e.g. for type you might decide to use only photo, video and music. Now you have to remember that. You also have to remember its "photo" not "image") - If tags were the solution, they would have already been used everywhere. The tagging system SEEMS like a good solution, but once you go deeper, it just doesn't work.
Re: Taxonomy Is Hard
#25tl,dr:
* Partitioning files into folders is most likely wrong: Most things need to be in multiple folders.
* (Sidenote: Symlinks are not a good solution)
* Tagging would be best but no good support from the OS for metadata
* We're working on something
Re: Taxonomy Is Hard
#26Tagging, seems like a solution. But isn't. Specific problems with tagging: - Need to tag every file (whereas in folders, you just navigate to the folder and everything you store there is in that folder) - Takes too long - Too much thinking overhead (at the time of storing) - To be effective have to enter the name for all tag entries (e.g. project, type, etc.). If anything is missed for a file, that file will never be…
For one thing you can train a model that will tag the same way you do.
Re: Taxonomy Is Hard
#27The big central problem here is that non-trivial taxonomies aren't trees but graphs. Trying to get a tree-based filesystem to represent a taxonomy means you're forcing a graph into a tree. Symlinks help because they turn your tree into a graph (albeit one that breaks too easily; I think that could be fixed, though). But in the end, a traditional filesystem is a poor way to represent a taxonomy.
If you really want to organize files basic on semantics you might generate file names based on, say, a SHA-3 hash but keep looking the attributes of the files in a database.
The trouble with that though is that people have different perspectives on documents.
The librarian in me wants to ingest a document and never modify it, such that content addressable storage is what I want. I want to attach metadata in an external database.
There's another culture though where people edit documents, most notable in Adobe's tools which will try to save a JPEG even if all you did was print it! Adobe developed
https://en.wikipedia.org/wiki/Extensible_Metadata_Platform
which embeds metadata in the files which fits that point of view.
Re: Taxonomy Is Hard
#28Tagging, seems like a solution. But isn't. Specific problems with tagging: - Need to tag every file (whereas in folders, you just navigate to the folder and everything you store there is in that folder) - Takes too long - Too much thinking overhead (at the time of storing) - To be effective have to enter the name for all tag entries (e.g. project, type, etc.). If anything is missed for a file, that file will never be…
If you tag using RDF[1] and use OWL[2] to define both the tag name and values then it really works pretty well. And with some entailment rules[3] you can even define rules that tags on folders becomes tags on items inside them, or you can just solve this with a bit more complex SPARQL query [4][5].
For an example of where such a system is used, see:
- WikiData[6]: They use RDF but they don't use OWL, they have a similar though less formal way of defining types, and then entailment has to be encoded in SPARQL queries.
- schema.org[7]: Uses RDF and also provide OWL specifications.
[1]: https://www.w3.org/TR/rdf11-primer/
[2]: https://www.w3.org/TR/owl2-primer/
[3]: https://www.w3.org/TR/rdf11-mt/#entailment-rules-informative
[4]: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/...
[5]: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/...
[6]: https://www.wikidata.org/wiki/Wikidata:Main_Page
[7]: https://schema.org/
Re: Taxonomy Is Hard
#29I guess the lesson to be learned is that you will need to support multiple classification systems. For biological taxonomy both morphological as well as genetics classifcation makes sense.
In the health data exchange format FHIR you have identifiers and codings have a system and a value/code. Usually you can specify multiple of them.
Re: Taxonomy Is Hard
#30All true. Taxonomy is indeed hard. But, does it actually matter? It seems what matters is not how files are stored/organized, but how one can find the files one is looking for. Taxonomy is mostly a search problem. Yet, although one is usually capable of remembering specific or unique details about a file, it's still incredibly hard to search for a file or its contents effectively. Dropbox, to pick just one example, d…
> it's still incredibly hard to search for a file or its contents effectively.
Underappreciated aspect of this is trust. As much as "taxonomy is mostly a search problem", search is in large part a trust problem - trust that the search was done exhaustively, and if it returned no results, it means there aren't any.
This is a problem that IMO most offerings are blind to. One big offender for me is Windows Explorer. It has a search tool that can search both in names, metadata and file content. Yet in the past, it frequently failed to find files I knew were there. Truth is, it may not have been even looking in the right place - it depends on what's been indexed and how, which is information you can find somewhere in the system, but notably not in the search interface itself. I've stopped using it long ago, as I don't trust it at emotional level. I prefer to literally walk the filesystem structure by hand.
And, on Windows, I at least get that option. This is the fallback, the baseline: even if I don't trust a search engine to be exhaustive, I can cope with it as long as I can perform an exhaustive search manually. I.e. as long as I have a way to list everything. But this, then, is almost universally missing from cloud offerings.
(There's a whole rant to be written about the incredibly dumb idea of hiding the filesystem / database from the end-user, but I'll skip it today.)
> there were startups a decade ago trying to address the search problem on the desktop but they all got acquired or folded; I don't understand why. The problem is real and seems quite solvable; yet it seems there's no actual market for it. It's a bit of a mystery.
Nobody really wants to solve this problem anymore, because it conflicts with the major thing vendors want: for you to move your data into their cloud. The desktop, and the ability to own your data, survives only because most cloud systems are still shit. This might eventually change, and in the meantime, the market definitely isn't interested in helping you own your data.