Naming things (2015) [pdf]

All this ancillary stuff that we'd like attached to files, like dates, client names and projects, versions and so on, are metadata. Some systems keep metadata in files: EXIF, Word, PDF. Some systems have conventions for this instead: header blocks in source code. But if neither of those applies? Only place you can put it reliably is the filename :(

If you're getting into R or data analysis, check out http://stat545.com/topics.html. She has put a lot of thought into the project management aspects of carrying out a data science project that don't get discussed as often as other, sexier topics.

I've written several "disk and file catalog" utilities over the years so I inevitably spent a lot of time thinking about the "metadata" problem.

I think the issue is that it's impossible to solve metadata in a universal way that satisfies all scenarios. This is why metadata often ends up being inscribed into the filename. It's the "least worse" solution.

Let's take one example of the scientific data of csv files. Typical Comma-Separated-Value files do not have metadata fields such as author, measuring device, timestamp of readings, GPS coordinates. (Yes, csv files sometimes have a first line for "column names" which is arguably metadata but that's not the higher-level metadata I'm talking about.)

Exactly where does one put that high-level metadata?

1) If one makes a new pseudo-standard that signifies any lines at the top the csv beginning with "//" as metadata, that means that modifying any metadata of a 100GB csv file (e.g. change author from "John Doe" to "Jacob Doe" is rewriting the whole 100GB file to add 1 byte.) As a related issue, let's say you have hash of "e1bb76e7391b93eb12" for the csv file. You really want a stable hash that represents the actual "raw data" of the csv file. You don't necessarily want the hash to change just because the metadata changed. In this case, embedding metadata into the file itself makes certain operations worse since typical hash utilities don't have "intelligence" about which parts of the file is "important" for hashing. (A similar problem is scanning mp3 files for duplicates. If 2 mp3 files have bit-identical audio output but the metadata tags are different, are they the same or different?!? It depends.)

2) if you put metadata at the end, typical utilities won't know about about it. (UNIX has "tail" command but standard MS Windows does not. The tail command is also unstructured and read-only which makes it a non-solution for managing end-of-file metadata fields. Also, the "quick" view of GUI file managers show the top of the file and not the bottom of it.)

3) If you put metadata in a separate file, it easily gets lost. File managers like MacOS Finder and MS Windows Explorer don't know when 2 files are supposed to be "treated as one unit" vs separately.

4) If you try to put metadata in a separate special area using os file system features suchs MS "NTFS alternate data streams" or Mac OSX "resource forks", they will get lost when transferring across incompatible filesystems or uploading to Amazon S3.

If one is feeling uncharitabe, one could say the MS WinFS[1] was a spectacular failed attempt at unifying metadata. (A relational database that makes metadata more of a 1st class concept.) Nobody has tried it on that level since. Even Apple's new file APFS system didn't have the same metadata ambitions as WinFS.

The combination of tradeoffs leads everybody to re-invent the idea of embedding metadata (including namespacing hierarchies) into filenames. The article's suggestions for scientific data filenames looks very similar to filenames that companies end up using for ETL pipelines.[2]

[1] https://en.wikipedia.org/wiki/WinFS

[2] https://en.wikipedia.org/wiki/Extract,_transform,_load

Not sure if it's a good solution but if I were to put the meta data somewhere I would somehow try to put it in the identifier of the file (the name) as it is data that would help me identify the file AND it's content!