Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing an Undocumented File Format

blog.vivekpanyam.com

Post reply on HN
Print view View original

11–20 of 65 posts

Re: Parsing an Undocumented File Format

#11

Post by mungoman2 »

nemothekid wrote: post #7
malux85 wrote: post #2

Reverse engineering a file format or protocol is almost a rite of passage for programmers, it is incredibly fun and rewarding, something I'd recommend for all medium/senior programmers get into at least once. A few years ago I was using LiDAR scanners from a manufacturer that didn't provide a linux driver, only windows - the way it worked is that you programmed the firmware to fire UDP packets at a specified IP and p…

> is almost a rite of passage for programmers, it is incredibly fun and rewarding, Only when you are doing it for yourself or when it's a known undertaking. It can be very frustrating when you are integrating with some hardware and you are 99% complete and you've told everyone you are ready to ship and the last 1% is a surprise reverse protocol engineering project.

Well tbh, failing to deliver after over committing is also a good experience.

Re: Parsing an Undocumented File Format

#13

Post by LASR »

More than a decade ago, I wrote a parser for Google Sketchup object files. It was part of some computer graphics assignment. We were not supposed to write parsers, but instead manually write out a 3D model vertex by vertex and then texture map them. I instead decided to write a parser and build my model in Sketchup.

It was a TON of fun doing that. And I learned a lot from that exercise.

Fast forward to a few weeks ago I "wrote" a parser / serializer for handling knowledge-graphs as input/output between my app and LLMs.

I used ChatGPT to walk me through the whole thing. It did very good job of converting between mermaidjs and an object type I defined in typescript. It wrote the code, the unit tests - the whole thing.

I don't understand how it works. The code is great. But not as satisfying.

Re: Parsing an Undocumented File Format

#14

Post by SuperNinKenDo »

This is something I've been interested in for a while.

I've collected a few links people have already posted to their own projects or write-ups here and elsewhere, but is there any single excellent resource for learning how to do this?

I've a number of dead and/or proprietary formats that I've always wanted to crack open, but I'm totally overwhelmed with where to start.

Re: Parsing an Undocumented File Format

#16

Post by danielvaughn »

Reminds me of a situation I ran into years ago. I worked at a fintech startup where we were reverse engineering the mobile APIs of retail stock brokerages. Eventually we ran out of brokers in the US and began looking overseas. The first one we looked at was a large broker in Singapore.

Their API responses were in some absolutely insane markup language that I'd never seen before. I actually had to spend a good deal of time reading up on the history of markup languages, carefully going through each one to see if the syntax matched.

Eventually I gave up and just had to write a parser myself. The worst bit was that the attributes didn't use quotation marks around the values. So you'd literally have markup like:
It was...fun times.

Re: Parsing an Undocumented File Format

#17

Post by lifthrasiir »

SuperNinKenDo wrote: post #14

This is something I've been interested in for a while. I've collected a few links people have already posted to their own projects or write-ups here and elsewhere, but is there any single excellent resource for learning how to do this? I've a number of dead and/or proprietary formats that I've always wanted to crack open, but I'm totally overwhelmed with where to start.

While I don't have any handy link, I did reverse-engineered several file formats without any further information and I can give some points.

First, make sure that you know what the format is actually supposed to encode. For example, if some file weighs (say) 40 KB then it is unlikely to be a raster image. The file name, if any, helps a lot to narrow the scope.

Second, you should have some understanding of similar file formats. I generally recommend to study PNG first because it gives an example of typical structured file formats and raster image formats. (Don't delve into the compression though---bitwise analysis is much harder.) This is also why you needed to know what the format is for, many formats with the same goal tend to have similar structures.

Third, collect as many examples as possible. You can line them up to see commonalities and differences and spot patterns. Even better if you can actively generate different files. This is generally the last hope when you are run out of reasonable hypotheses.

Fourth, optimize the feedback loop. You will have to do a lot of hypothesization, validation and automation. You can't really optimize the number of iterations, but you can optimize the time for a single iteration. Use a comfortable scripting language with good binary operation. I tend to use a vanilla Python with struct and make everything else by my own, but there are several libraries that greatly help you if you don't feel like doing so.

Re: Parsing an Undocumented File Format

#18

Post by lifthrasiir »

Dwedit wrote: post #8

Just watch out for the encrypted file formats. Need a debugger to figure out what it's doing.

They are relatively easier to detect though. Compressed file formats or virtual machine codes in disguise tend to be more annoying, especially since they can look like somehow structured and waste your time.

Re: Parsing an Undocumented File Format

#19

Post by malux85 »

ikari_pl wrote: post #5
malux85 wrote: post #2

Reverse engineering a file format or protocol is almost a rite of passage for programmers, it is incredibly fun and rewarding, something I'd recommend for all medium/senior programmers get into at least once. A few years ago I was using LiDAR scanners from a manufacturer that didn't provide a linux driver, only windows - the way it worked is that you programmed the firmware to fire UDP packets at a specified IP and p…

Some of the worst code of my life, created 20 years ago when I was a teenager, today posted openly on my GitHub, was reverse engineered custom chat server protocol, as I wrote my own client to replace the Java applet. And to have logs. The catch is... I didn't have any Internet connection. I was going to an internet cafe, logging onto the chat server, and chatting, while recording the connection with Wireshark. At ho…

I did this too! Having no interenet at home I would take a box of floppy disks to the library every day and save articles on C programming, OpenGL programming, and the raw weather satellite images from the NOAA - I was trying to re-create the 3D weather fly-over I saw in Jurassic Park when I was 7. I agree - having that disconnection was ultimatly good because it made me think a lot for myself rather than just google a solution, it developed my fundimentals a lot.

Re: Parsing an Undocumented File Format

#20

Post by mabster »

Back in the day I had a dual boot machine and was using a corner skin of Winamp on the Windows side and missed this on the Linux side.

So I started reverse engineering the Winamp skinning engine with the intention of making an engine that can run it on one of the Linux media apps. I did this by writing short programs and looking at the generated binary.

I had about 95% of it figured out when we had a robbery and the thief took the laptop I was using. By the time I got a new machine I had completely lost interest!

I wouldn't be surprised if it was using a well known VM that I just didn't know about at the time!

Post reply on HN