Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The unexpected effectiveness of one-shot decompilation with Claude

blog.chrislewis.au

Post reply on HN
Print view View original

81–90 of 133 posts

Re: The unexpected effectiveness of one-shot decompilation with Claude

#81

Post by baq »

simonw wrote: post #69
wavemode wrote: post #57

Earlier quoted context omitted.

"one-shot" usually just means, one example and its correct answer was provided in the prompt. See also, "zero-shot" / "few-shot" etc.

I've seen one-shot used to mean two different things in LLMs: 1. Getting an LLM to do something based on a single example 2. Getting an LLM to achieve a goal from a single prompt with no follow-ups I think both are equally valid.

One-shot as in ‘given one example’ is the ML term. One-shot as in ‘in a single prompt’ is the colloquial meaning. Both are useful, but it can be confusing when discussing LLMs in ML topics.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#84

Post by theturtle32 »

hobs wrote: post #73
dimitri-vs wrote: post #53

Earlier quoted context omitted.

FWIW most LLMs are pretty terrible at estimating complexity. If you've used Claude Code for any length of time you might be familiar with it's plan "timelines" which always span many days but for medium size projects get implemented in about an hour. I've had CC build semi-complex Tauri, PyQT6, Rust and SvelteKit apps for me without me having ever touched that language. Is the code quality good? Probably not. But all…

Disagree - the timelines are completely reasonable for an actual software project, and that's what the training data is based on, not projects written with LLMs.

Yes, this is my experience as well.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#85

Post by Nevermark »

There are quite a few comments here on code obfuscation.

The hardest form of code obfuscation is called homomorphic computing, which is code transformed to act on encrypted data isomorphically to regular code on regular data. The homomorphic code is hard obfuscated by this transformation.

Now create a homomorphic virtual machine, that operates on encrypted code over encrypted data. Very hard to understand.

Now add data encryption/decryption algorithms, both homomorphically encrypted to be run by the virtual machine, to prepare and recover inputs, outputs or effects of any data or event information, for the homomorphic application code. Now that all data within the system is encrypted by means which are hard obfuscated, running on code which is hard obfuscated, the entire system becomes hard^2 (not a formal measure) opaque.

This isn't realistic in practice. Homomorphic implementations of even simple functions are extremely inefficient for the time being. But it is possible, and improvements in efficiency have not been exhausted.

Equivalent but different implementations of homomorphic code can obviously be made. However, given the only credible explanations for design decisions of the new code are, to exactly match the original code, this precludes any "clean room" defenses.

--

Implementing software with neural network models wouldn't stop replication, but would decompile as source that was clearly not developed independent from the original implementation.

Even distilling (training a new model on the "decompiled" model) would be dead giveaway that it was derived directly from the source, not a clean room implementation.

--

I have wondered, if quantum computing wouldn't enable an efficient version of homomorphic computing over classical data.

Just some wild thoughts.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#86

Post by tadfisher »

euroderf wrote: post #20
ronsor wrote: post #16

Earlier quoted context omitted.

I've used LLMs to help with decompilation since the original release of GPT-4. They're excellent at recognizing the purpose of functions and refactoring IDA or Ghidra pseudo-C into readable code.

Someone please try this on an original (early 1980s) IBM-PC BIOS.

I don't believe that was written in a compiled language, so any old 8086 disassembler should suffice. I would love to see what comments an LLM adds to the assembly code, though.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#87

Post by voiper1 »

t_mann wrote: post #45

> The ‘give up after ten attempts’ threshold aims to prevent Claude from wasting tokens when further progress is unlikely. It was only partially successful, as Claude would still sometimes make dozens of attempts. Not what I would have expected from a 'one-shot'. Maybe self-supervised would be a more suitable term?

I definitely didn't expect one-shot to mean "let it run itself in an indefinite loop"

Re: The unexpected effectiveness of one-shot decompilation with Claude

#88

Post by motoboi »

ph4evers wrote: post #26
saagarjha wrote: post #24

It's worth noting here that the author came up with a handful of good heuristics to guide Claude and a very specific goal, and the LLM did a good job given those constraints. Most seasoned reverse engineers I know have found similar wins with those in place. What LLMs are (still?) not good at is one-shot reverse engineering for understanding by a non-expert. If that's your goal, don't blindly use an LLM. People alrea…

Are they not performing well because they are trained to be more generic, or is the task too complex? It seems like a cheap problem to fine-tune.

The knowledge probably is o the pre-training data (the internet documenta the LLM is trained at to get a good grasp), but probably very poorly represented in the reinforcement learning phase.

Which is to say that probably antropic don’t have good training documents and evals to teach the model how to do that.

Well they didn’t. But now they have some.

If the author want to improve his efficiency even more, I’d suggest he starts creating tools that allow a human to create a text trace of a good run on decompilating this project.

Those traces can be hosted in a place Antropic can see and then after the next model pre-training there will be a good chance the model become even better at this task.

Re: The unexpected effectiveness of one-shot decompilation with Claude

#89

Post by reactordev »

I’ve been having fun sending Claude down the old school MUD route, giving it access to a SMAUG derivative and once it’s mastered the play, give it admin powers to create new play experiences.

I stayed away from decompilation and reverse engineering, for legal reasons.

Claude is amazing. It can sometimes get stuck in a reason loop but will break away, reassess, and continue on until it finds its way.

Claude was murdered in a dark instance dungeon when it managed to defeat the dragon but ran out of lamp oil and torches to find its way out. Because of the light system it kept getting “You can’t seem to see anything in the darkness” and randomly walked into a skeleton lair.

Super fun to watch from an observer. Super terrifying that this will replace us at the office.

Post reply on HN