This is something I've been interested in for a while. I've collected a few links people have already posted to their own projects or write-ups here and elsewhere, but is there any single excellent resource for learning how to do this? I've a number of dead and/or proprietary formats that I've always wanted to crack open, but I'm totally overwhelmed with where to start.
I had reversed engineered some ASCII file formats. While probably overkill, my background parsing simple programming languages (for which there are many good educational resources) was really helpful (in the approach I use). I tokenize, and try figuring out syntax structures from the order of token types, then from there, extract the information I need into my program's data representation. I'm not sure if this is the approach used by everyone else, but it seems plausible for someone with a CS/PL implementation background.
But first, it helps to have sample files to see recurring structures. Ideally, you also have access to software that generates these files. This allows you to deal with simpler files containing less information to reason about, make small changes within the program and compare the corresponding change(s) in the file.