Parsing an Undocumented File Format

oh god, that sounds like a painful thing to try to do. there's an Access redistributable at https://www.microsoft.com/en-us/download/details.aspx?id=549...

i've never attempted JS interop, so i don't know how much it sucks, but it definitely seems doable. it'll likely be an adventure on it's own, but it's gotta be 1000x less fuckery than trying to reverse the binary format of Access

the saner option would probably just be a small .Net program that creates an endpoint for your JS

unless i'm assuming too much, and you're just doing it for sheer masochistic pleasure, and in that case: i salute you

I'm reminded of the comment in XeePhotoshopLoader.m:

  // At this point, I'd like to take a moment to speak to you about the Adobe PSD format.
  // PSD is not a good format. PSD is not even a bad format. Calling it such would be an
  // insult to other bad formats, such as PCX or JPEG. No, PSD is an abysmal format. Having
  // worked on this code for several weeks now, my hate for PSD has grown to a raging fire
  // that burns with the fierce passion of a million suns.
  // If there are two different ways of doing something, PSD will do both, in different
  // places. It will then make up three more ways no sane human would think of, and do those
  // too. PSD makes inconsistency an art form. Why, for instance, did it suddenly decide
  // that *these* particular chunks should be aligned to four bytes, and that this alignement
  // should *not* be included in the size? Other chunks in other places are either unaligned,
  // or aligned with the alignment included in the size. Here, though, it is not included.
  // Either one of these three behaviours would be fine. A sane format would pick one. PSD,
  // of course, uses all three, and more.
  // Trying to get data out of a PSD file is like trying to find something in the attic of
  // your eccentric old uncle who died in a freak freshwater shark attack on his 58th
  // birthday. That last detail may not be important for the purposes of the simile, but
  // at this point I am spending a lot of time imagining amusing fates for the people
  // responsible for this Rube Goldberg of a file format.
  // Earlier, I tried to get a hold of the latest specs for the PSD file format. To do this,
  // I had to apply to them for permission to apply to them to have them consider sending
  // me this sacred tome. This would have involved faxing them a copy of some document or
  // other, probably signed in blood. I can only imagine that they make this process so
  // difficult because they are intensely ashamed of having created this abomination. I
  // was naturally not gullible enough to go through with this procedure, but if I had done
  // so, I would have printed out every single page of the spec, and set them all on fire.
  // Were it within my power, I would gather every single copy of those specs, and launch
  // them on a spaceship directly into the sun.
  //
  // PSD is not my favourite file format.

Whenever I see stories like that I always wonder if anyone has succeeded at parsing an undocumented file format that included custom compression scheme.

Parsing a binary file is tedious but you can progress steadily at least, whereas you would never be sure you even decompressed correctly, before even trying to decode the format.

Fortunately this is mostly a theoretical problem. There are very few cases where a custom compression would be more efficient than slapping a .zip/.zstd/.tar on it if it ever goes too big.

> Why, for instance, did it suddenly decide that these particular chunks should be aligned to four bytes, and that this alignement should not be included in the size?

Probably because, like many other ancient document formats (e.g. MS Office), it was a straight dump of memory structures into a file [1]. Obviously a very bad idea in hindsight (especially given the truckload of deserialization vulns resulting from it), but computers from that age were so memory-constrained that anything else wouldn't cut it, and by the time computers got more powerful the old formats were hopelessly entrenched.

[1] https://www.joelonsoftware.com/2008/02/19/why-are-the-micros...

I picked apart some of the Dark Forces files myself as a kid. The GOB file was pretty obvious - I was familiar with Doom WADs, it's basically the same. I figured out the graphics formats by setting up VGA 320x200 mode with the contents of a .PAL file, and dumping the graphics file into screen memory. Then looking at the noise around the familiar patterns to figure out all the run/skip length stuff that wasn't just a pixel value.