Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The long tail of LLM-assisted decompilation

blog.chrislewis.au

Post reply on HN
Print view View original

21–30 of 31 posts

Re: The long tail of LLM-assisted decompilation

#21

Post by measurablefunc »

seddonm1 wrote: post #20
measurablefunc wrote: post #19

Earlier quoted context omitted.

Decompilation does not preserve semantics. You generally do not know whether the code from the decompiler will be compiled to semantically equivalent binary that you initially decompiled.

My test harness loads up the original DLL then executes that in parallel against the converted code (differential testing). That closes the feedback loop the LLM needs to be able to find and fix discrepancies. I'm also doing this on an old Win32 DLL so the task is probably much easier than a lot of code bases.

What are you tracking during the runtime tracing? Or is that written up in your link?

Re: The long tail of LLM-assisted decompilation

#22

Post by seddonm1 »

measurablefunc wrote: post #21
seddonm1 wrote: post #20

Earlier quoted context omitted.

My test harness loads up the original DLL then executes that in parallel against the converted code (differential testing). That closes the feedback loop the LLM needs to be able to find and fix discrepancies. I'm also doing this on an old Win32 DLL so the task is probably much easier than a lot of code bases.

What are you tracking during the runtime tracing? Or is that written up in your link?

I am applying differential/property based testing to all the side effects of functions (mutations) and return values. The rust code coverage is also used to steer the LLM as it finds discrepancies in side effects.

It is written up in my link - please bear in mind it is really hard to find the right level to communicate this level of detail at - so I'm happy to answer questions.

Re: The long tail of LLM-assisted decompilation

#23

Post by measurablefunc »

seddonm1 wrote: post #22
measurablefunc wrote: post #21

Earlier quoted context omitted.

What are you tracking during the runtime tracing? Or is that written up in your link?

I am applying differential/property based testing to all the side effects of functions (mutations) and return values. The rust code coverage is also used to steer the LLM as it finds discrepancies in side effects. It is written up in my link - please bear in mind it is really hard to find the right level to communicate this level of detail at - so I'm happy to answer questions.

That's fine, that answers my question.

Re: The long tail of LLM-assisted decompilation

#24

Post by qingcharles »

lstodd wrote: post #16
qingcharles wrote: post #14

Earlier quoted context omitted.

Me too. I'm going to be reverse-engineering Elite PC (original version) and I can't help but think the source is lost. The developer seems to have totally dropped off the face of the Earth. I've contacted others who might know and nobody knows where they are. Even the game I was a developer on which was published by Eidos in ~1998 is probably lost source. I can't think that anyone has the Visual Source Safe database…

You mean 1991 Elite Plus? The whole series has been reverse-engineered to death and back. Maybe you mean some other game? Anyway, for those old titles I don't think not having source is that much of a problem. I participated in two reimplementations of 1994 XCOM : UFO2000 and OpenXcom, helped the 1oom project (first Master of Orion) and I don't think having original source would have helped much.

No, I'm doing the original 1987 PC Elite. The later one was written by Chris Sawyer. I asked him recently and he also has no idea about Andy who wrote the prior version (both for Realtime). [both versions I assume were written in 100% ASM] Surprisingly Gemini seems to be pretty good at writing 8088 CGA assembler, especially in Deep Think mode. It one-shot an entire filled poly renderer and 3D engine.

I worked with some of the original XCOM guys after a bunch of them left Microprose to set up on their own. I wrote a lot of the graphics engine for this, which was really a direct descendent of XCOM:

https://www.youtube.com/watch?v=9UOYps_3eM0

Re: The long tail of LLM-assisted decompilation

#25

Post by sureglymop »

Here's an interesting thing. I decided to do advent of code in assembly last year. What I noticed is that there must be a lot of code and binaries in AI training data but not a lot of intermediate representation. Be it LLVM IR, assembly or other forms of IR, it seems underrepresented. LLMs kept trying to give me code patterns that would make sense for high level code but not really for assembly because by hand one could find much more optimized solutions there.

But coincidentally this seems like an easy win for generated training data. Take all your code and have a compiler spit out assembly as well as binary. Now your LLM will not only be able to be a compiler but also make that useful and understandable by humans.

Re: The long tail of LLM-assisted decompilation

#26

Post by skerit »

titzer wrote: post #9
bri3d wrote: post #6

Claude is doing the decompilation here, right? Has this been compared against using a traditional decompiler with Claude in the loop to improve decompilation and ensure matched results? I would think that Claude’s training data would include a lot more pseudo-C C knowledge than MIPS assembler from GCC 2.7 and C pairs, and even if the traditional decompiler was kind of bad at N64 it would be more efficient to fix bad…

It's wild to me that they wouldn't try this first. Feeding the asm directly into the model seems like intentionally ignoring a huge amount of work that has gone in traditional decompilation. What LLMs excel at (names, context, searching in high-dimensional space, making shit up) is very different from, e.g. coming up with an actual AST with infix expressions that represents asm code.

I've been doing some decompilation with Ghidra. Unfortunately, it's of a C++ game, which Ghidra isn't really great at. And thus Claude gets a bit confused about it all too. But all in all: it does work, and I've been able to reconstruct a ton of things already.

Re: The long tail of LLM-assisted decompilation

#28

Post by easyThrowaway »

measurablefunc wrote: post #19
seddonm1 wrote: post #18

Earlier quoted context omitted.

I delivered a talk at Rust Sydney about this exact topic last week: https://reorchestrate.com/posts/your-binary-is-no-longer-saf... I am able to translate multi-thousand line c functions - and reproduce bug-for-bug implementation

Decompilation does not preserve semantics. You generally do not know whether the code from the decompiler will be compiled to semantically equivalent binary that you initially decompiled.

Many of the decompiled console games of the '90s were originally written in C89 using an ad-hoc compiler from Metrowerks or some off-branch release of gcc-2.95 plus console specific assemblers.

I willing to bet that the decompiled output is gonna be more readable than the original source code.

Re: The long tail of LLM-assisted decompilation

#29

Post by measurablefunc »

easyThrowaway wrote: post #28
measurablefunc wrote: post #19

Earlier quoted context omitted.

Decompilation does not preserve semantics. You generally do not know whether the code from the decompiler will be compiled to semantically equivalent binary that you initially decompiled.

Many of the decompiled console games of the '90s were originally written in C89 using an ad-hoc compiler from Metrowerks or some off-branch release of gcc-2.95 plus console specific assemblers. I willing to bet that the decompiled output is gonna be more readable than the original source code.

Not related to what I was saying. Compilation is a many-to-one transformation & although you can try to guess an inverse there is no way to guarantee you will recover the original source b/c at the assembly level you don't have any types & structs.

Re: The long tail of LLM-assisted decompilation

#30

Post by macabeus »

roelljr wrote: post #5

If you turn this into a benchmark, it will be solved in no time :)

I'm developing a pipeline runner for matching decompilation: https://github.com/macabeus/mizuchi

The initial motivation is to run benchmarks, though the foundation is flexible and can support many other use cases over time.

It's already proving useful. For example, I can run a benchmark, view the results in a dashboard, and even feed the report into Claude Code to answer questions like: "How did changing X affect the results?" or "What could be improved in the next run?"

Post reply on HN