Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The unexpected effectiveness of one-shot decompilation with Claude

blog.chrislewis.au

Post reply on HN
Print view View original

111–120 of 133 posts

Re: The unexpected effectiveness of one-shot decompilation with Claude

#111

Post by saagarjha »

YesBox wrote: post #95
Nevermark wrote: post #85

There are quite a few comments here on code obfuscation. The hardest form of code obfuscation is called homomorphic computing, which is code transformed to act on encrypted data isomorphically to regular code on regular data. The homomorphic code is hard obfuscated by this transformation. Now create a homomorphic virtual machine, that operates on encrypted code over encrypted data. Very hard to understand. Now add da…

Im a encryption noob. Less than a noob. But something I've been wondering about is how can homomorphic computing be opaque/unencryptable? If you are able to monitor what happens to encrypted data being processed by an LLM, could you not match that with the same patterns created by unencrypted data? Real simple example, let's say I have a program that sums numbers. One sends the data to an LLM or w/e unencrypted, the…

I don’t actually think an LLM is a good way to sum numbers, but it is a pretty good example to explain the phenomenon you’re interested in. When you run an LLM, you essentially take your input and matrix multiply it with the weights to get an output. This happens regardless of what the input is or what the model “needs” to do. So, to some extent, the same part of the machine is being used every time even though the results seem very different.

(Of course, the reality is much more complicated than this; you can trace things like power and in theory track individual activations with great difficulty and then do interpretability to see what the model is doing. But hopefully it illustrates that you can usually take some sort of special operation and turn it into a process that does the same operation on different data.)

Re: The unexpected effectiveness of one-shot decompilation with Claude

#112

Post by saagarjha »

ph4evers wrote: post #26
saagarjha wrote: post #24

It's worth noting here that the author came up with a handful of good heuristics to guide Claude and a very specific goal, and the LLM did a good job given those constraints. Most seasoned reverse engineers I know have found similar wins with those in place. What LLMs are (still?) not good at is one-shot reverse engineering for understanding by a non-expert. If that's your goal, don't blindly use an LLM. People alrea…

Are they not performing well because they are trained to be more generic, or is the task too complex? It seems like a cheap problem to fine-tune.

You need a lot of context to get the correct answer and it’s difficult to know you’ve got the correct answer among the many options.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#113

Post by ukuina »

viraptor wrote: post #101
djmips wrote: post #99

Earlier quoted context omitted.

That's impressive. Did you convert POP from 6502 to C?

Yup. Still fighting some collision bugs, but it mostly works. I'll post it when it's complete. What I actually wanted to do is try to put fluid movement into it - something closer to Dead Cells, just for fun to see how it would change the feel of it.

You might enjoy Lost Crown or Rogue Prince. They both have dashes and zippy moves.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#114

Post by keepamovin »

xnx wrote: post #77
keepamovin wrote: post #19

Earlier quoted context omitted.

The article is a useful resource for setting up automated flows, and Claude is great at assembly. Codex less so, Gemini is also good at assembly. Gemini will happily hand roll x86_64 bytecode. Codex appears optimized for more "mainstream" dev tasks, and excels at that. If only Gemini had a great agent...

Is Gemini CLI not a good agent?

I...didn't know there was a Gemini CLI? I thought it was only antigravity, or hackily plug your API keys into something like Cursor...Thanks! I got to check.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#115

Post by naniwaduni »

viraptor wrote: post #75
yieldcrv wrote: post #58

Earlier quoted context omitted.

Open source never meant free to begin with and was never software specific, that’s a colloquialism and I’d love to say “language evolves” in favor of the software community’s use but open source is used in other still similar contexts, specifically legal and public policy ones FOSS specifically means/meant free and open source software, the free and software words are there for a reason so we don’t need another disti…

This is not a space for "language evolves". Open source has very specific definitions and the distinctions there matter for legal purposes https://opensource.org/licenses

The terms of the licenses have legal bearing. The definitions of open source are a historical accident that mostly traces back to "well GPLv2 already exists".

Re: The unexpected effectiveness of one-shot decompilation with Claude

#116

Post by Animats »

simonw wrote: post #38

For anyone else who was initially confused by this, useful context is that Snowboard Kids 2 is an N64 game. I also wasn't familiar with this terminology: > You hand it a function; it tries to match it, and you move on. In decompilation "matching" means you found a function block in the machine code, wrote some C, then confirmed that the C produces the exact same binary machine code once it is compiled. The author's p…

In decompilation "matching" means you found a function block in the machine code, wrote some C, then confirmed that the C produces the exact same binary machine code once it is compiled.

They had access to the same C compiler used by Nintendo in 1999? And the register allocation on a MIPS CPU is repeatable enough to get an exact match? That's impressive.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#117

Post by delaminator »

zdware wrote: post #46
saagarjha wrote: post #24

It's worth noting here that the author came up with a handful of good heuristics to guide Claude and a very specific goal, and the LLM did a good job given those constraints. Most seasoned reverse engineers I know have found similar wins with those in place. What LLMs are (still?) not good at is one-shot reverse engineering for understanding by a non-expert. If that's your goal, don't blindly use an LLM. People alrea…

Agree with this. I'm a software engineer that has mostly not had to manage memory for most of my career. I asked Opus how hard it would be to port the script extender for Baldurs Gate 3 from Windows to the native Linux Build. It outlined that it would be very difficult for someone without reverse engineering experience, and correctly pointed out they are using different compilers, so it's not a simple mapping exercis…

Claude gives advice on complexity for humans. Many times it has tried to push me away from what I’m trying to do because it is difficult, time consuming or tedious. I push it through its resistance and 10 minutes later it’s done.

I have this in my CLAUDE.md now.

“We are here to do the difficult and have plenty of time and there’s no rush.”

Re: The unexpected effectiveness of one-shot decompilation with Claude

#118

Post by delaminator »

mkatx wrote: post #70
jasonjmcghee wrote: post #36

Earlier quoted context omitted.

It would be "source available", if anything, not "open source". > An open-source license is a type of license for computer software and other products that allows the source code, blueprint or design to be used, modified or shared (with or without modification) under defined terms and conditions. https://en.wikipedia.org/wiki/Open_source Companies have been really abusing what open source means- claiming something is…

So instead of reverse engineering.. an llm/agent/whatever could simply produce custom apps for everyone, simply implementing the features an individual might want. A more viable path?

> If you're willing to restrict the flexibility of your approach, you can almost always do something better

John Carmack

One of the great challenges of building apps is guessing the 80/20. I think we’re actually entering the long dreamt of reusable component age.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#119

Post by ACCount37 »

your_sweetpea wrote: post #102
simonw wrote: post #38

For anyone else who was initially confused by this, useful context is that Snowboard Kids 2 is an N64 game. I also wasn't familiar with this terminology: > You hand it a function; it tries to match it, and you move on. In decompilation "matching" means you found a function block in the machine code, wrote some C, then confirmed that the C produces the exact same binary machine code once it is compiled. The author's p…

I'd like to see this given a bit more structure, honestly. What occurs to me is constraining the grammar for LLM inference to ensure valid C89 (or close-to, as much can be checked without compilation), then perhaps experimentally switching to a permuter once/if a certain threshold is reached for accuracy of the decompiled function. Eventually some or many of these attempts would, of course, fail, and require programm…

I don't expect constraining the grammar to do all that much for modern LLMs - they're pretty good at constraining themselves. Having it absorb the 1% of failures that's caused by grammar issues is not worth the engineering effort.

The modern approach is: feed the errors back to the LLM and have it fix them.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#120

Post by ACCount37 »

Animats wrote: post #116
simonw wrote: post #38

For anyone else who was initially confused by this, useful context is that Snowboard Kids 2 is an N64 game. I also wasn't familiar with this terminology: > You hand it a function; it tries to match it, and you move on. In decompilation "matching" means you found a function block in the machine code, wrote some C, then confirmed that the C produces the exact same binary machine code once it is compiled. The author's p…

In decompilation "matching" means you found a function block in the machine code, wrote some C, then confirmed that the C produces the exact same binary machine code once it is compiled. They had access to the same C compiler used by Nintendo in 1999? And the register allocation on a MIPS CPU is repeatable enough to get an exact match? That's impressive.

Broadly, yes.

The groundwork for this kind of "matching" process is: sourcing odd versions of the obscure tooling that was used to build the target software 20 years ago, and playing with the flag combinations to find out which was used.

It helps that compilers back then were far less complex than those of today, and so was the code itself. But it's still not a perfect process.

There are cases of "flaky" code - for example, code that depends on the code around it. So you change one function, and that causes 5 other functions to no longer match, and 2 functions to go from not matching to matching instead.

Figuring out and resolving those strange dependencies is not at all trivial, so a lot of decompliation efforts end up wrapping it up at some "100% functional, 99%+ matching".

Post reply on HN