Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing an Undocumented File Format

blog.vivekpanyam.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Parsing an Undocumented File Format

#41

Post by danielvaughn »

m463 wrote: post #34
danielvaughn wrote: post #16

Reminds me of a situation I ran into years ago. I worked at a fintech startup where we were reverse engineering the mobile APIs of retail stock brokerages. Eventually we ran out of brokers in the US and began looking overseas. The first one we looked at was a large broker in Singapore. Their API responses were in some absolutely insane markup language that I'd never seen before. I actually had to spend a good deal of…

I'm reminded of the comment in XeePhotoshopLoader.m: // At this point, I'd like to take a moment to speak to you about the Adobe PSD format. // PSD is not a good format. PSD is not even a bad format. Calling it such would be an // insult to other bad formats, such as PCX or JPEG. No, PSD is an abysmal format. Having // worked on this code for several weeks now, my hate for PSD has grown to a raging fire // that burns…

This is hilarious, and not at all surprising. The format has been around longer than most programmers today have been alive, with all the legacy cruft you could expect as it's changed over the years.

Re: Parsing an Undocumented File Format

#42

Post by danielvaughn »

cedws wrote: post #39
danielvaughn wrote: post #16

Reminds me of a situation I ran into years ago. I worked at a fintech startup where we were reverse engineering the mobile APIs of retail stock brokerages. Eventually we ran out of brokers in the US and began looking overseas. The first one we looked at was a large broker in Singapore. Their API responses were in some absolutely insane markup language that I'd never seen before. I actually had to spend a good deal of…

I have seen code that produces output like this first hand. Instead of doing proper serialization, they were using string templating to construct the response and never bothered to validate the output. Laziness and stupidity basically.

You're very likely correct, which is funny because they turned out to have incredibly security. We hacked the APIs of all the US brokers without an issue, but I didn't even make it past the auth stage with this Singapore broker.

One morning I was working on their login flow - not doing anything crazy, mind you. Just a bit weird; logging in and out, watching the req/res cycles with Charles Proxy. All of a sudden my boss comes over and tells me to stop immediately. Apparently I set off so many alarm bells at the broker that the CTO was woken up (it was 2am where they were). That was a fun gig lol.

Re: Parsing an Undocumented File Format

#43

Post by jvdvegt »

KolmogorovComp wrote: post #36

Whenever I see stories like that I always wonder if anyone has succeeded at parsing an undocumented file format that included custom compression scheme. Parsing a binary file is tedious but you can progress steadily at least, whereas you would never be sure you even decompressed correctly, before even trying to decode the format. Fortunately this is mostly a theoretical problem. There are very few cases where a custo…

There was a story about reverse engineering highspeed Broadcom networkcard firmware on HN last week. That included a custom compression if I remember correctly.

This is the post: https://news.ycombinator.com/item?id=38772862

Re: Parsing an Undocumented File Format

#44

Post by tacone »

cedws wrote: post #39
danielvaughn wrote: post #16

Reminds me of a situation I ran into years ago. I worked at a fintech startup where we were reverse engineering the mobile APIs of retail stock brokerages. Eventually we ran out of brokers in the US and began looking overseas. The first one we looked at was a large broker in Singapore. Their API responses were in some absolutely insane markup language that I'd never seen before. I actually had to spend a good deal of…

I have seen code that produces output like this first hand. Instead of doing proper serialization, they were using string templating to construct the response and never bothered to validate the output. Laziness and stupidity basically.

In my former PHP life I've seen people looping through objects and constructing a JSON string by hand instead of using a simpler single json_encode() call.

Re: Parsing an Undocumented File Format

#45

Post by disruptiveink »

m463 wrote: post #34
danielvaughn wrote: post #16

Reminds me of a situation I ran into years ago. I worked at a fintech startup where we were reverse engineering the mobile APIs of retail stock brokerages. Eventually we ran out of brokers in the US and began looking overseas. The first one we looked at was a large broker in Singapore. Their API responses were in some absolutely insane markup language that I'd never seen before. I actually had to spend a good deal of…

I'm reminded of the comment in XeePhotoshopLoader.m: // At this point, I'd like to take a moment to speak to you about the Adobe PSD format. // PSD is not a good format. PSD is not even a bad format. Calling it such would be an // insult to other bad formats, such as PCX or JPEG. No, PSD is an abysmal format. Having // worked on this code for several weeks now, my hate for PSD has grown to a raging fire // that burns…

As a side note, I really miss Xee. It was clearly a labour of love and it showed - MacPaw has been a terrible steward ever since Dag handed it over to them.

No updates other than straight up SDK bumps and recompiles, broken loading of random images in recent macOS/Apple Silicon, they somehow managed to break cropping in one of the two or three updates they did, still an Intel binary. Clearly they haven't tested it more than just checking if the app opens.

Really wish Dag had just open sourced Xee3 instead, my opinion of MacPaw plumetted after seeing how they massacred my boy Xee.

The Archive Browser was equally neglected. At least The Unarchiver still works, which in retrospect was clearly the only app MacPaw wanted to take off Dag's hands.

Re: Parsing an Undocumented File Format

#47

Post by KolmogorovComp »

jvdvegt wrote: post #43
KolmogorovComp wrote: post #36

Whenever I see stories like that I always wonder if anyone has succeeded at parsing an undocumented file format that included custom compression scheme. Parsing a binary file is tedious but you can progress steadily at least, whereas you would never be sure you even decompressed correctly, before even trying to decode the format. Fortunately this is mostly a theoretical problem. There are very few cases where a custo…

There was a story about reverse engineering highspeed Broadcom networkcard firmware on HN last week. That included a custom compression if I remember correctly. This is the post: https://news.ycombinator.com/item?id=38772862

Thanks for the link! Impressive work indeed. Relevant snippets

> but had no idea as to how the image was compressed. It clearly wasn't compressed with any common compression algorithm. Mercifully unlike the MIPS firmware, it had at least a few strings, which is how I was able to tell it was compressed; a hex dump showed chunks of human-readable text with garbage interrupting them.

> A hunch. After extensive amounts of time trying and failing to eyeball the compression algorithm from hexdumps of compressed code, and trying any decompression algorithm I could think of against it,

But they eventually could break through by reverse engineering the decompression code.

> Once I finally had a concise, sane description of the decompression algorithm in C, the algorithm turned out to be hilariously simple. I was also then able to figure out the origins of the compression algorithm; it's called LZSS

Re: Parsing an Undocumented File Format

#48

Post by samus »

m463 wrote: post #34
danielvaughn wrote: post #16

Reminds me of a situation I ran into years ago. I worked at a fintech startup where we were reverse engineering the mobile APIs of retail stock brokerages. Eventually we ran out of brokers in the US and began looking overseas. The first one we looked at was a large broker in Singapore. Their API responses were in some absolutely insane markup language that I'd never seen before. I actually had to spend a good deal of…

I'm reminded of the comment in XeePhotoshopLoader.m: // At this point, I'd like to take a moment to speak to you about the Adobe PSD format. // PSD is not a good format. PSD is not even a bad format. Calling it such would be an // insult to other bad formats, such as PCX or JPEG. No, PSD is an abysmal format. Having // worked on this code for several weeks now, my hate for PSD has grown to a raging fire // that burns…

> I can only imagine that they make this process so // difficult because they are intensely ashamed of having created this abomination.

They also don't want anybody building a dependency on that sh*t, which would prevent them from ever cleaning up the mess.

Re: Parsing an Undocumented File Format

#49

Post by samus »

mschuster91 wrote: post #38
m463 wrote: post #34

Earlier quoted context omitted.

I'm reminded of the comment in XeePhotoshopLoader.m: // At this point, I'd like to take a moment to speak to you about the Adobe PSD format. // PSD is not a good format. PSD is not even a bad format. Calling it such would be an // insult to other bad formats, such as PCX or JPEG. No, PSD is an abysmal format. Having // worked on this code for several weeks now, my hate for PSD has grown to a raging fire // that burns…

> Why, for instance, did it suddenly decide that these particular chunks should be aligned to four bytes, and that this alignement should not be included in the size? Probably because, like many other ancient document formats (e.g. MS Office), it was a straight dump of memory structures into a file [1]. Obviously a very bad idea in hindsight (especially given the truckload of deserialization vulns resulting from it),…

Flash and PDF have the same illnesses. Suspiciously many Adobe file formats are both overly complicated and contain lots of features that are way too powerful in combination and impossible to support properly.

Re: Parsing an Undocumented File Format

#50

Post by samus »

LASR wrote: post #13

More than a decade ago, I wrote a parser for Google Sketchup object files. It was part of some computer graphics assignment. We were not supposed to write parsers, but instead manually write out a 3D model vertex by vertex and then texture map them. I instead decided to write a parser and build my model in Sketchup. It was a TON of fun doing that. And I learned a lot from that exercise. Fast forward to a few weeks ag…

> We were not supposed to write parsers, but instead manually write out a 3D model vertex by vertex and then texture map them.

Good grief. It's like writing assembly: a good exercise, but only for trivial or particularly tricky parts of a program. For everything else, proper tooling (a compiler or a 3D modeler) is the way to go. I fully agree that the best learning experience is to build tools to automate away the annoying parts :-D

Post reply on HN