Live data from Hacker News

Ask HN: How do you organize your files

news.ycombinator.com

21–30 of 44 posts

Re: Ask HN: How do you organize your files

#21
- ebooks: I don't love Calibre, but it's the only game in town.

- music: Musicbrainz Picard to get the metadata right. I've been favoring RPis running mpd as a front-end to my music lately.

- movies/TV: MediaElch + Kodi

I don't have a good solution for managing pictures and personal videos that doesn't involve handing all of it to some awful, spying "cloud" service. Frankly most of this stuff is sitting in Dropbox (last few years worth) or, for older files, in a bunch of scattered "files/old_desktop_hd_3_backup/desktop/photos"-type directories waiting for my wife and I to go through them and do something with them. Which is increasingly less likely to happen—sometimes I think the natural limitations of physical media were a kind of blessing, since one was liberated from the possibility of recording and retaining so much. Without some kind of automatic facial recognition and tagging—and saving of the results in some future-proof way, ideally in the photos/videos themselves—this project is likely doomed.

My primary unresolved problem is finding some sort of way to preserve integrity and provide multi-site backup that doesn't waste a ton of my time+money on set-up and maintenance. When private networks finally land in IPFS I might look at that, though I think I'll have to add a lot of tooling on top to make things automatic and allow additions/modifications without constant manual intervention, especially to collections (adding one thing at a time, all separately, comes with its own problems, like having to enumerate all of those hashes when you want something to access a category of things, like, say, all your pictures). Probably I'll have to add an out-of-band indexing system of some sort, likely over HTTP for simplicity/accessibility. For now I'm just embedding a hash (CRC32 for length reasons and because I mostly need to protect against bit-rot, not deliberate tampering) at the end of filenames, which is, shockingly, still the best cross-platform way to assert a content's identity, and synchronizing backups with rsync—ZFS is great and all but doesn't preserve useful hash info if a copy of a file is on a non-ZFS filesystem, plus I need basically zero of its features aside from periodically checking file integrity.

Re: Ask HN: How do you organize your files

#24
post #18

I try not to over think it, just: ~/$MAJOR_TOPIC | |--- ./$MORE_SPECIFIC | |--- ./$MORE_SPECIFIC | |--- ./general-file.type | | ./general-file.type | |--- ./$MORE_SPECIFIC | |--- ./general-file.type etc As you find yourself collecting more general files under a directory that can be logically grouped, create a new directory and move them to it. Also keep all your directories in the same naming convention (idk maybe I…

That's pretty much how I did it on my NAS. My top level is basically "fiction", "non-fiction", "music", "pictures", "software" - then it just goes from there.

But it has problems. For instance, I like to collect information about robotics and artificial intelligence. In many cases I have papers with titles like "Using Computer Vision to Control a Robot Arm via a CNN". Do I put it under "robotics/sensors/vision" or "ai-ml/anns/cnn" or "robotics/motion-control/platform/arm"...or...? It can technically fit into any and all of those categories!

That's a problem with hierarchical category structures; when something can fit into multiple categories, you can either duplicate the information (not good - unless your system has some way of using pointer refs or such to prevent data duplication - which most file systems don't), cross-link the information (put it in a canonical spot and symlink to it), or just say "f-it" and stick it someplace, and hope you can find it later (which sometimes you can't).

What I wish I had, instead, was a simple means to search my filesystem in a very quick fashion. Ideally, it would be something like the old Google Search Appliance, which could spider and index my filesystem, read each file (and any metadata stored in the file, such as in the case of videos and images) and build up an index that can be quickly and easily searched. It would also keep this index up-to-date as files are added, removed, or changed.

Unfortunately, I've yet to find a low-cost (ideally free) open-source solution to this problem, that was also easy to set up and maintain. I've found more than a few solutions (or partials) which given enough admin and configuration (plus maintenance and/or glue code) could potentially become the system I want, but none of them were "turnkey" - install, simple setup (nothing more complex than a NAS or WiFi Router, for instance), and "let it go". They were all very "enterprise-y" and required more than a bit of effort to install and maintain. It isn't that I couldn't do that, I just don't have the time to dedicate to such a task. But it might be something I just have to bite the bullet for.

Maybe what I need to do is research the latest offering of FreeNAS - maybe they've (since the last time I used it) implemented a decent search engine module (or some third party plugin has been created) to handle this issue?

Re: Ask HN: How do you organize your files

#25
symlinks for ~/Downloads and ~/Documents into ~/Dropbox is my only interesting upgrade. Across the varying different devices I have different things selectively synced. Large media files are the only things that don't live in dropbox in some way or another. It's pretty convenient for mobile access (everything accessible from web/mobile). I've done some worrying about sensitive documents and such, but most of it is also present in my email, so I think I lost that battle already. It also means there's very little downside to wiping my HD entirely if I want to try a different OS (which I used to do frequently, but ended up settling on vanilla ubuntu).

Re: Ask HN: How do you organize your files

#26
post #16

One thing I do that I've found to be pretty helpful is to prefix files/directories with a number or date, for sorting. Some things are naturally ordered by date, for example events. So I might have a directory "my-company/archive", where each item is named "20170621_some-event". Other things are better sorted by category or topic. For tools or programming languages I'm researching I might have a directory with items…

I did this (with dates) for all calculations I did during my PhD and it turned out very useful despite its simplicity. I guess it was simple enough that I could actually be consistent over years. I wish the rest of my projects were as orderly ;)

Re: Ask HN: How do you organize your files

#27
If you're particularly asking about reference material that you take notes about and would like to search and retrieve and produce reports on, Zotero might work for you. I have many years of research notes on it - it's a hyper-bookmarking tool that can keep snapshots of web pages, keep PDFs and other "attachments" within saved articles, lets you tag and organize them along with search capabilities.

Outside of that scope, my files reside randomly somewhere in the ~/Documents folder (I use a mac) and I rely on spotlight to find the item I need. It's not super great but is workable often enough.

It's not a silly question!

edit: I've been trying to find a multi-disk solution and haven't had much success with an easy enough to use tool. I use git-annex for this and it helps to some extent. I've also tried Camlistore, which is promising, but has a long way to go.

Re: Ask HN: How do you organize your files

#28

    documents/projects/projectname/whateverlayoutthetoolsdemand
with each project under Git. Layouts for Go, Rust, ROS, and KiCAD are forced by the tools. Python isn't as picky.

Web sites are

    sitename/
        info - login data for site, domains, etc.
        site - what gets pushed to the server
        work - other stuff not pushed to server
with each site under version control.

Re: Ask HN: How do you organize your files

#30
Another option is to have a look at a tag-based filesystem instead of hierarchical ones to organize everything semantically. I'm using Tagsistant (there're other options) for a couple of months now and I'm almost happy. More satisfied with the idea itself and the potentiality.
Post reply on HN