Live data from Hacker News

Small Archives

helloruby.substack.com

11–12 of 12 posts

Re: Small Archives

#11

Earlier quoted context omitted.

Modern hard drives are now above 20TB in capacity. If your average file size is 100KB, then they will hold 200 million+ files. File systems with their directory or folder hierarchy are poor organizers of files. Searching for files is a huge hassle. File systems, which were designed over 50 years ago when storage was measured in MB, need to be replaced by object stores that are much better at organizing and finding da…

In what way are object stores better than hierarchical filesystems? With links you can create arbitrary indices over the same data with different (or the same) names.

This is what I do. Some 5 million files (3 TB) with hard and soft links everywhere to make the files show up in places I hope to find them again.

Ripgrep Anything or whatever it is called is pretty good for finding things, at least as long as I can restrict the search to a not too huge subset of my archive.

I also have org-mode documents full of links to interesting files, and sometimes to locations within files.

A bit messy, but I do not want to have dependencies on any special software. My oldest files have been with me since I first started hoarding on 360 kB DOS floppies.

Re: Small Archives

#12

Earlier quoted context omitted.

Modern hard drives are now above 20TB in capacity. If your average file size is 100KB, then they will hold 200 million+ files. File systems with their directory or folder hierarchy are poor organizers of files. Searching for files is a huge hassle. File systems, which were designed over 50 years ago when storage was measured in MB, need to be replaced by object stores that are much better at organizing and finding da…

In what way are object stores better than hierarchical filesystems? With links you can create arbitrary indices over the same data with different (or the same) names.

With file systems, searching for files (e.g. find all documents) requires traversing the entire directory hierarchy. With hard drives this comes at a cost of many seeks.

The last time I traversed a folder tree with over 100 million files, it took more than 30 minutes to complete.

A properly designed object store can find everything in less than a minute.

Post reply on HN