Show HN: Tired of the non-portability of my playlists, I wrote my own format • Hacker News Board

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#51

Post by jcoffland » Wed, Mar 01, 2017, 6:49 PM UTC

alnitak wrote: post #18

bb88 wrote: post #15

Earlier quoted context omitted.

I don't see why this is being downvoted. It's fairly apropos to the subject. And it was the first thing I thought of too when I read the title.

À format that is independent of a specific platform or player has its own merits, besides the scratch your itch part. With a good plugin support an ecosystem to allow interoperability of playlists becomes possible.

But such player independent formats already exist. This is not a new idea. It's just another format.

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#52

Post by jstimpfle » Wed, Mar 01, 2017, 7:24 PM UTC

If I understand correctly, you want to make playlists portable by associating each item with enough metadata to allow identification even when some metadata, such as filepath, changes.

Playlist and metadata are separate things, and coupling them is not good from an architectural perspective (redundancy/denormalization). The two can still be related, even in a single file: relational database. (plug: I'm working on http://jstimpfle.de/projects/wsl/main.html)

Sketch to make more clear what I mean:

    {
      "metadata": {
        "buss_und_reu": {
          "filepath": "Bach/Matthäuspassion/Buss_und_Reu.mp3"
          "composer": "Johann Sebastian Bach"
        },
        "highway_to_hell": {
          "filepath": "ACDC/Highway_to_Hell.mp3",
          "artist": "ACDC",
          "sha512": ...
        }
      },
      "playlists": {
        "mypl": [
          "buss_und_reu",
          "highway_to_hell"
        ],
        "mypl2": [
          "buss_und_reu"
        ]
      }
    }

This is more normalized, less redundant. Of course other schemes are thinkable, for example storing each playlist in its own file, but this way metadata and playlists will more quickly get out of sync.

The library I'm working on allows automated conversions of such hierarchical data to/from relational databases, so data can be viewed, and even stored, as JSON, while integrity is checked based on an relational schema - something like

    % TABLE Track trackid
    % TABLE Filepath trackid filepath
    % TABLE SHA512 trackid sha512
    % TABLE Title trackid title
    % TABLE Composer trackid composer
    % [Declare also domains, keys, and foreign keys here...]

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#53

Post by derefr » Wed, Mar 01, 2017, 7:24 PM UTC

lucasgonze wrote: post #46

StavrosK wrote: post #10

Earlier quoted context omitted.

Nothing's wrong per se (and it has roughly the same goals as mine), it's just that it's pretty much the XML equivalent of M3U, and doesn't have any provisions for stronger identification (it relies on titles), at least as far as I know. I would really like the MBID to be the main way of identifying songs throughout the industry, and possibly the AcoustID fingerprint, as that's more specific. I think it would be fanta…

XSPF is a format to enable sharing, AKA universality. It does this by defining a list of metadata fields to be used for resolving each track in the local context of the listener. The UPF "ids" field maps to the XSPF "identifier" field: http://xspf.org/xspf-v1.html#rfc.section.4.1.1.2.14.1.1.1.2 XSPF is not M3U in any way. From the spec: http://xspf.org/xspf-v1.html#rfc.section.3.4 3.4 Content resolver On a surface le…

That sounds like a noble goal, but in a "modern" format designed for the same thing, I'd expect the role of the "identification key" for the files to be played by a format-standardized-algorithm audio fingerprint. Audio fingerprints are the only thing you can really expect to be "portable" between music libraries, when people can put arbitrary things in the ID3, and both combined tracks and compilation albums exist. And then, as long as you have such a key, the format doesn't need to consist of much else. It's just a list of keys.

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#54

Post by TTPrograms » Wed, Mar 01, 2017, 7:26 PM UTC

I think a few people have open-sourced various audio fingerprinting algorithms. Perhaps we should standardize around one and enable real UUID for audio files? That seems to be the missing piece for a lot of the novel functionality you're looking to hit. Otherwise you're stuck doing fuzzy pattern matching on metadata, which seems tricky especially considering possible variation in live recording etc.

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#55

Post by zeveb » Wed, Mar 01, 2017, 7:29 PM UTC

hawski wrote: post #9

Are relative paths supported? Probably in this day and age it would have better chance at adoption if it would be a JSON file. It's just network effects.

> Probably in this day and age it would have better chance at adoption if it would be a JSON file. It's just network effects.

And here I was feeling badly for thinking that it'd be better as S-expressions! I wasn't going to post that, but since you already did, here's the same example in the original YAML:

    ---
    format: UPL1
    name: Favorites
    id: 2b43009f-d6a6-4f00-8533-09a9a73d8b54
    entries:
    - artist: Anciients
      title: Following the Voice
      duration: 408.764081632
      ids:
        sha2: e577cce68a69735acccd5d8603b3e663f6aa5bc9
        sha3: e577cce68a69735acccd5d8603b3e663f6aa5bc9
        mbtrackid: b00a2b97-53f1-485a-9121-1fe76b55e651
        filepath: Anciients/Following the Voice.mp3
        uri: nfs://example.com/music/ftv.mp3

as JSON:

    {
        "format": "UPL1",
        "name": "Favorites",
        "id": "2b43009f-d6a6-4f00-8533-09a9a73d8b54",
        "entries": [
            {
                "artist": "Anciients",
                "title": "Following the Voice",
                "duration": 408.764081632,
                "ids": {
                    "sha2": "e577cce68a69735acccd5d8603b3e663f6aa5bc9",
                    "sha3": "e577cce68a69735acccd5d8603b3e663f6aa5bc9",
                    "mbtrackid": "b00a2b97-53f1-485a-9121-1fe76b55e651",
                    "filepath": "Anciients/Following the Voice.mp3",
                    "uri": "nfs://example.com/music/ftv.mp3"
                }
            }
        ]
    }

And as S-expressions:

    (playlist
     (format UPL1)
     (name Favorites)
     (id 2b43009f-d6a6-4f00-8533-09a9a73d8b54)
     (entries
      (entry
       (artist Anciients)
       (title "Following the Voice")
       (duration 408.764081632)
       (ids
        (sha2 e577cce68a69735acccd5d8603b3e663f6aa5bc9)
        (sha3 e577cce68a69735acccd5d8603b3e663f6aa5bc9)
        (mbtrackid b00a2b97-53f1-485a-9121-1fe76b55e651)
        (filepath "Anciients/Following the Voice.mp3")
        (uri nfs://example.com/music/ftv.mp3)))))

I wonder which in general people think are prettier.

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#56

Post by zeveb » Wed, Mar 01, 2017, 7:37 PM UTC

I think rather than having a single URI attribute and other identifiers, you ought to allow multiple URIs, and make all other identifiers just be URIs (since, after all, a URI is a Uniform Resource Identifier, and music files are resources).

You can represent hashes using the named-information URI scheme (https://tools.ietf.org/html/rfc6920), e.g. ni:///sha-256;u88lYWn4xAlto-6Bs79KHHYDAu28US71ui5Be6C-ZVw.

Your filepath could be file:///Anciients/Following%20the%20Voice.mp3; your mbtrackid could be musicbrainz:b00a2b97-53f1-485a-9121-1fe76b55e651 (since I don't think an authority makes sense in this case).

This would also permit multiple entries for certain types of identifier, which doesn't make so much sense for hashes (although it's possible for the same recording to have two different MP3 encodings, so … maybe it's not crazy), but would be nice when e.g. a song is found in multiple places in the filesystem, or can be retrieved from multiple locations.

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#57

Post by nbadg » Wed, Mar 01, 2017, 7:49 PM UTC

StavrosK wrote: post #44

autocracy wrote: post #41

A side note: at least in python, I benchmarked MsgPack as 3 times faster than JSON, and a whopping 850 times faster to read than YAML. It seems unlikely that people will ever be editing this file by hand, so for unstructured data, especially where playlists can get very large, I suggest MsgPack over YAML. BUT... you have a defined schema, so it's probably to your advantage to use a storage format with a defined schem…

Thank you, I agree. If I'm going to use JSON, I might as well use ProtoBuf. About MD5, I was worried about a case where a service that serves user-submitted files would be exploited by MD5 collisions, leading users to open files that might exploit decoder bugs to execute code. Far-fetched, I know, but the tradeoff didn't seem worth it. I'm not married to that decision, though.

The question of hash usage made me think of an alternative approach -- what about some kind of audio perceptual hash? P-hash has support for audio hashes [1] (at least it claims to, but I've never used it). The metadata is useful, sure, but coupling it with the playlist seems like a bit of a strange design choice, if it could be avoided. In my mind, an ideal world would have two databases (or equivalent): one for metadata -> perceptual hash, and one for playlist -> List[perceptual hashes].

The downside of course is this requires pre-calculation of the p-hash for every track to use. But I can't think of a music application that doesn't require some kind of "library loading" step, so perhaps this could be accomplished then?

Of course none of this mitigates your concern with decoder bugs resulting in RCE, but I think that's probably best handled elsewhere (for example, sandboxed upload validation in your hypothetical user-uploaded-files service).

[1] http://www.phash.org/

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#58

Post by nosuchthing » Wed, Mar 01, 2017, 7:54 PM UTC

derefr wrote: post #53

lucasgonze wrote: post #46

Earlier quoted context omitted.

XSPF is a format to enable sharing, AKA universality. It does this by defining a list of metadata fields to be used for resolving each track in the local context of the listener. The UPF "ids" field maps to the XSPF "identifier" field: http://xspf.org/xspf-v1.html#rfc.section.4.1.1.2.14.1.1.1.2 XSPF is not M3U in any way. From the spec: http://xspf.org/xspf-v1.html#rfc.section.3.4 3.4 Content resolver On a surface le…

That sounds like a noble goal, but in a "modern" format designed for the same thing, I'd expect the role of the "identification key" for the files to be played by a format-standardized-algorithm audio fingerprint. Audio fingerprints are the only thing you can really expect to be "portable" between music libraries, when people can put arbitrary things in the ID3, and both combined tracks and compilation albums exist.…

  format-standardized-algorithm audio fingerprint

Assuming that's not available on stock Android/iOS/etc a quick hash like md5 should suffice.

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#59

Post by rout39574 » Wed, Mar 01, 2017, 8:25 PM UTC

StavrosK wrote: post #44

autocracy wrote: post #41

A side note: at least in python, I benchmarked MsgPack as 3 times faster than JSON, and a whopping 850 times faster to read than YAML. It seems unlikely that people will ever be editing this file by hand, so for unstructured data, especially where playlists can get very large, I suggest MsgPack over YAML. BUT... you have a defined schema, so it's probably to your advantage to use a storage format with a defined schem…

Thank you, I agree. If I'm going to use JSON, I might as well use ProtoBuf. About MD5, I was worried about a case where a service that serves user-submitted files would be exploited by MD5 collisions, leading users to open files that might exploit decoder bugs to execute code. Far-fetched, I know, but the tradeoff didn't seem worth it. I'm not married to that decision, though.

You can apply a schema to JSON, too.

Can you hear us underneath that pile of bike sheds?

Re: Show HN: Tired of the non-portability of my playlists, I wrote my own format

#60

Post by djrogers » Wed, Mar 01, 2017, 8:27 PM UTC

Is the lack of portability really due to the unavailability of a (or yet another) standard format? Seems it's far more related to the music player decisions about what it's playlist format should be...