Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Designing better file organization around tags, not hierarchies (2017)

nayuki.io

Post reply on HN
Print view View original

71–80 of 169 posts

Re: Designing better file organization around tags, not hierarchies (2017)

#71

Post by nayuki »

discreteevent wrote: post #31
wrs wrote: post #25

This is good thinking, and mostly overlaps with what I've tried to do a few times, so I would love to see it finally happen somehow. For example, the Newton storage system I worked on at Apple 1990-1996 was based on separating organization from storage so we could have multiple (tag and/or hierarchy) organization systems. That eventually became the "soup" system in the shipping Newton OS, where objects were retrieved…

Would you have time to say a bit more about why WinFS failed from your point of view?

There is a series of blog posts that gives a retrospective on WinFS:

* https://hal2020.com/2013/02/14/winfs-integratedunified-stora...

* https://hal2020.com/2013/02/14/winfs-integratedunified-stora...

* https://hal2020.com/2013/02/25/winfs-integratedunified-stora...

* https://hal2020.com/2013/03/10/winfs-integratedunified-stora...

Re: Designing better file organization around tags, not hierarchies (2017)

#72

Post by wrs »

nayuki wrote: post #71
discreteevent wrote: post #31

Earlier quoted context omitted.

Would you have time to say a bit more about why WinFS failed from your point of view?

There is a series of blog posts that gives a retrospective on WinFS: * https://hal2020.com/2013/02/14/winfs-integratedunified-stora... * https://hal2020.com/2013/02/14/winfs-integratedunified-stora... * https://hal2020.com/2013/02/25/winfs-integratedunified-stora... * https://hal2020.com/2013/03/10/winfs-integratedunified-stora...

Wow, I forgot Hal practically did wrote a book. :)

I was on the Windows Shell side, sitting in a lot of schema meetings where products couldn't figure out how to combine their incompatible complex existing schemas into one new WinFS schema that would be understandable to anyone, and in a lot of other meetings where I was sort of a negotiation translator, decoding user interface concepts for the SQL team and SQL concepts for the Shell team, though their priorities were wildly different.

The common theme throughout was that it was pretty clear none of the product teams saw any reason they should go through unknown years of effort and compatibility hell, delaying the feature roadmap for their product, just to fulfill Bill's desire for this abstract notion of Integrated Storage. So these meetings went on interminably to make Bill and David and Bob and Jim happy, while none of the participants outside the WinFS team itself saw any point to the exercise.

It was not a pleasant experience, but I learned a lot about how a large organization works! (A reason I'm no longer in one.)

Re: Designing better file organization around tags, not hierarchies (2017)

#73

Post by vbezhenar »

yiyus wrote: post #34
wrs wrote: post #25

This is good thinking, and mostly overlaps with what I've tried to do a few times, so I would love to see it finally happen somehow. For example, the Newton storage system I worked on at Apple 1990-1996 was based on separating organization from storage so we could have multiple (tag and/or hierarchy) organization systems. That eventually became the "soup" system in the shipping Newton OS, where objects were retrieved…

I think most people would agree a tag filesystem or a similar concept is a great idea. I have wanted one myself for a long time. Yet, for some reason, it doesn't take off. Do you think it is just a problem of implementation?

I'm not sure about that. I'm fine with hierarchical file system and links for my projects (desktop or explorer side bar). I tried few times to make use of macOS tags, but I didn't find them useful. I guess, for people who work with actual documents, it might be different. For me all those systems trying to be smart end up showing me billions of .class files from my build directories and wasting cycles trying to index them.

Re: Designing better file organization around tags, not hierarchies (2017)

#74

Post by foobarbecue »

jessmartin wrote: post #54

Have you considered a file system organized as a timeline that _also_ supports tagging? I find one of the key concepts that's not a first-class concept is _when_ the file was modified. Rather than a file-and-folder physical analogy for the file system UI, I think a timeline-oriented UI could present some advantages for the way that humans actually think and work. Tags would be a helpful orthogonal organization scheme…

This might be a good idea if clocks were reliable.

Re: Designing better file organization around tags, not hierarchies (2017)

#75

Post by [deleted] »

Benjammer wrote: post #15
on_and_off wrote: post #5

>But fundamentally, there is a mismatch between the narrowness of hierarchies and the rich structure of human knowledge, and the proposed system will not presuppose the features of HFSes. This hits the nail on the head ! All the fileSystems I had to work with are fine as engineering tools. By that I mean using them as an engineer works just fine, their own implementation is off topic. As a user though. What the hell…

How do you make sure things are tagged properly though? A file must reside within a folder, even if it's a default location, which forces a user to think about the folder where the file is stored. With tags, a user could very easily forget one tag on a file, and now any filtering on that tag is never going to be aware of the new file existing. What if you find another picture somewhere from your trip, but you forget…

[deleted]

Re: Designing better file organization around tags, not hierarchies (2017)

#76

Post by nayuki »

kijin wrote: post #67
Baeocystin wrote: post #41

Earlier quoted context omitted.

Hierarchical file systems allow you to say for sure where a file isn't. Every tagging system I've ever played with has turned out to be a mess in actual use compared to the simplicity a hierarchy provides. (I am just a user, not a developer of filesystems, so this may be a naive opinion.)

Tags are fun when you have a few thousand items to test your MVP with. It gets much less fun when you have millions of items with thousands of tags, all on a flat hierarchy. On the other hand, when you're stuck with a flat hierarchy anyway (e.g. thousands of pictures, all named DCIMxxxx.jpg), tags can be more useful. But only if they're automatic. I want the best of both worlds. I want to organize my stuff into folde…

I understand your concerns, and they are indeed valid. First off, I doubt that managing millions of files in a traditional hierarchical file system is fun either. You'd likely run into problems with making unique names, sharding folders, and categorizing files that logically belong in multiple places. I also have some worries that existing file systems (say NTFS or XFS) will behave or perform well with millions of files. I believe that implementing tags is a starting point for the problem of managing millions of files in a sensible way.

Speaking of thousand of pictures, what I really want is to dump all my photos into one folder. Right now, tools are ill-equipped to deal with large folders, so I am forced to manually create a new folder for every thousand or so items.

Re: Designing better file organization around tags, not hierarchies (2017)

#77

Post by pronoiac »

lkj wrote: post #4

As I recently inherited a huge, well-tagged music collection (tens of terabytes of files) I am very interested in this. Is there something like it, also supporting .cue files and also storing the original filenames and structure? A mediaplayer agnostic way to access this treasure trove would be the best.

I'm not speaking from experience, but git-annex comes to mind: https://git-annex.branchable.com/tips/metadata_driven_views/

Re: Designing better file organization around tags, not hierarchies (2017)

#78

Post by nayuki »

jessmartin wrote: post #54

Have you considered a file system organized as a timeline that _also_ supports tagging? I find one of the key concepts that's not a first-class concept is _when_ the file was modified. Rather than a file-and-folder physical analogy for the file system UI, I think a timeline-oriented UI could present some advantages for the way that humans actually think and work. Tags would be a helpful orthogonal organization scheme…

I have thought about file tagging for over a decade, before setting out to write the article. But a timeline-oriented file organization only came to my awareness near the end of writing the first draft.

A year has passed since I wrote the article, and the idea of timeline presentation has grown on me a lot. Especially because I use numerous data systems daily that are already time-oriented: Every chat program, email, Twitter, Facebook personal profile timeline, the "recent documents" view in major popular applications like Microsoft Office or Adobe Reader.

You should find this blurb in my article helpful:

> The Lifestreams Software Architecture http://www.cs.yale.edu/homes/freeman/dissertation/etf.pdf (185 pages)

> Comprehensively designs and tests a system for workflow and archival, based on chronological presentation plus keyword and attribute filtering.

Thanks for your compliments on the thoroughness of my exposition. I did a lot of research and thinking as preparation for writing. I wanted to see how other people viewed the problem of file organization and what kind of solutions they proposed. I wanted to find weaknesses in my arguments and to avoid repeating unnecessary work, and of course I wanted to move toward the best solution.

Re: Designing better file organization around tags, not hierarchies (2017)

#79

Post by nayuki »

flukus wrote: post #59

> When people realize they need to classify a file in more than one way, they will start to use shortcuts/links to try to solve the problem. (Windows has shortcuts, Unix has soft/symbolic links, and Mac has aliases. This is a ubiquitous feature, but transferring shortcuts across existing platforms is very hard.) This sounds like a reasonable solution, but will face trouble in all but the simplest use cases. They rule…

Hi. Have you tried managing a collection of shortcuts? How do you deal with recategorization, removing files, renaming files, moving files to different storage devices, etc.?

The article has a whole section illustrating why both hardlinks and softlinks won't work in the general case. https://www.nayuki.io/page/designing-better-file-organizatio...

Re: Designing better file organization around tags, not hierarchies (2017)

#80

Post by nayuki »

Slansitartop wrote: post #7

I never personally used it, but I've heard the BeFS was designed to have significant non-hierarchical use cases: https://en.wikipedia.org/wiki/Be_File_System : > [BeFS] includes support for extended file attributes (metadata), with indexing and querying characteristics to provide functionality similar to that of a relational database. IIRC, this was pretty hyped at the time, but they had to back away from it. I don't…

BeFS extended file attributes are a good example to point out. I watched this excellent talk which shows the power of live queries: https://systemswe.love/archive/minneapolis-2017/ivan-richwal...

Unfortunately, file attributes is not a feature I want to see. They don't solve the problems with naming files or deduplicating files. Metadata is attached to a file, so when the file is gone the metadata is gone. I instead proposed that metadata can be freestanding, and can exist even if the main file is missing. Relevant section to read: https://www.nayuki.io/page/designing-better-file-organizatio...

I did read the entire book "Practical File System Design with the Be File System", but didn't find it helpful for what I was working on. I can skip the low-level bits because I'll probably build on top of a NoSQL database or something.

Post reply on HN