Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing an Undocumented File Format

blog.vivekpanyam.com

Post reply on HN
Print view View original

1–10 of 65 posts

Re: Parsing an Undocumented File Format

#2

Post by malux85 »

Reverse engineering a file format or protocol is almost a rite of passage for programmers, it is incredibly fun and rewarding, something I'd recommend for all medium/senior programmers get into at least once.

A few years ago I was using LiDAR scanners from a manufacturer that didn't provide a linux driver, only windows - the way it worked is that you programmed the firmware to fire UDP packets at a specified IP and port and then when the device powered up it would push this continuous stream of data to you. 300,000 points a second.

So I started capturing these UDP packets and then decoding them with python, eventually I had to write a plugin in C to do the high performance parsing and bit packing, but nothing beats that feeling when you're stumped on what a bit of data means and then a eurika moment hits you in the shower, and the project advances!

Re: Parsing an Undocumented File Format

#3

Post by mattpallissard »

malux85 wrote: post #2

Reverse engineering a file format or protocol is almost a rite of passage for programmers, it is incredibly fun and rewarding, something I'd recommend for all medium/senior programmers get into at least once. A few years ago I was using LiDAR scanners from a manufacturer that didn't provide a linux driver, only windows - the way it worked is that you programmed the firmware to fire UDP packets at a specified IP and p…

Agreed, reverse engineering a job submission and validation protocol that ran over a Unix socket was some of the most fun I ever had on the clock.

I wound up basically re-implementing the software that was listening to create a test suite by the time it was all said and done.

Re: Parsing an Undocumented File Format

#4

Post by inferiorhuman »

mattpallissard wrote: post #3
malux85 wrote: post #2

Reverse engineering a file format or protocol is almost a rite of passage for programmers, it is incredibly fun and rewarding, something I'd recommend for all medium/senior programmers get into at least once. A few years ago I was using LiDAR scanners from a manufacturer that didn't provide a linux driver, only windows - the way it worked is that you programmed the firmware to fire UDP packets at a specified IP and p…

Agreed, reverse engineering a job submission and validation protocol that ran over a Unix socket was some of the most fun I ever had on the clock. I wound up basically re-implementing the software that was listening to create a test suite by the time it was all said and done.

Even the less challenging stuff can be fun. After seeing that SimCity themed Chicago zoning map I got to thinking about if it would be possible to do the same for the Bay Area. Turns out the cities I've looked at so far publish all sorts of machine readable information, the fun part is in finding and exfiltrating it.

Re: Parsing an Undocumented File Format

#5

Post by ikari_pl »

malux85 wrote: post #2

Reverse engineering a file format or protocol is almost a rite of passage for programmers, it is incredibly fun and rewarding, something I'd recommend for all medium/senior programmers get into at least once. A few years ago I was using LiDAR scanners from a manufacturer that didn't provide a linux driver, only windows - the way it worked is that you programmed the firmware to fire UDP packets at a specified IP and p…

Some of the worst code of my life, created 20 years ago when I was a teenager, today posted openly on my GitHub, was reverse engineered custom chat server protocol, as I wrote my own client to replace the Java applet. And to have logs.

The catch is... I didn't have any Internet connection. I was going to an internet cafe, logging onto the chat server, and chatting, while recording the connection with Wireshark.

At home, I'd print the hex + ASCII connection dump on my dot matrix printer, and used a highlighter and ballpoint pen to mark the fields of the message packet.

Then I'd code something around it, planned new tests, compiled a new version of the app and.... took it with me on a hard drive to the internet cafe to test next day, or next weekend.

I think I was way smarter and goal oriented than I am today.

Re: Parsing an Undocumented File Format

#6

Post by sedatk »

malux85 wrote: post #2

Reverse engineering a file format or protocol is almost a rite of passage for programmers, it is incredibly fun and rewarding, something I'd recommend for all medium/senior programmers get into at least once. A few years ago I was using LiDAR scanners from a manufacturer that didn't provide a linux driver, only windows - the way it worked is that you programmed the firmware to fire UDP packets at a specified IP and p…

That’s what I was gonna say. Reverse engineering was the only way in the 90’s as documentation was scarce. I had to reverse anything if I wanted to understand how it worked.

Here is an extractor I wrote for Westwood PAK and Lucasarts LFD files when I was 16: https://gist.github.com/ssg/e3e9654612be916336c01e104b10ddc7

Re: Parsing an Undocumented File Format

#7

Post by nemothekid »

malux85 wrote: post #2

Reverse engineering a file format or protocol is almost a rite of passage for programmers, it is incredibly fun and rewarding, something I'd recommend for all medium/senior programmers get into at least once. A few years ago I was using LiDAR scanners from a manufacturer that didn't provide a linux driver, only windows - the way it worked is that you programmed the firmware to fire UDP packets at a specified IP and p…

>is almost a rite of passage for programmers, it is incredibly fun and rewarding,

Only when you are doing it for yourself or when it's a known undertaking. It can be very frustrating when you are integrating with some hardware and you are 99% complete and you've told everyone you are ready to ship and the last 1% is a surprise reverse protocol engineering project.

Post reply on HN