Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

quesma.com

Post reply on HN
Print view View original

91–100 of 108 posts

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#91

Post by ronald_petty »

anamexis wrote: post #73
stared wrote: post #69

Earlier quoted context omitted.

No. To give it a fair test, we didn't tinker with model-specific context-engineering. Adding skills, examples, etc is very likely to improve performance. So is any interactive feedback. Our example instruction is here: https://github.com/QuesmaOrg/BinaryAudit/blob/main/tasks/lig...

Why, though? That would make sense if you were just trying to do a comparative analysis of different agent's ability to use specific tools without context, but if your thesis is: > However, [the approach of using AI agents for malware detection] is not ready for production. Then the methodology does not support that. It's "the approach of using AI agents for malware detection with next to zero documentation or guidan…

Not the author. Just my thoughts on supplying context during tests like these. When I do tests, I am focused on "out of the box" experiences. I suspect the vast majority of actors (good and bad, junior and senior) will use out of the box more then they will try to affect the outcome based on context engineering. We do expect tweaking prompts to provide better outcomes, but that also requires work (for now). Maybe another way to think is reducing system complexity by starting at the bottom (no configuration) before moving to top (more configuration). We can't even replicate out of the box today much less any level of configuration (randomness is going to random).

Agree it is a good test to try, but there are huge benefits beings able to understand (better recreate) 0-conf tests.

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#92

Post by celeryd »

mbh159 wrote: post #88

The methodology debate in this thread is the most important part. The commenter who says "add obfuscation and success drops to zero" is right but that's also the wrong approach imo. The experiment isn't claiming AI can defeat a competent attacker. It's asking whether AI agents can replicate what a skilled (RE) specialist does on an unobfuscated binary. That's a legitimate, deployable use case (internal audit, code re…

Why does that matter? Being oblivious to obfuscated binaries is like failing the captcha test.

Let's say instead of reversing, the job was to pick apples. Let's say an AI can pick all the apples in an orchard in normal weather conditions, but add overcast skies and success drops to zero. Is this, in your opinion, still a skilled apple picking specialist?

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#93

Post by sonofhans »

celeryd wrote: post #92
mbh159 wrote: post #88

The methodology debate in this thread is the most important part. The commenter who says "add obfuscation and success drops to zero" is right but that's also the wrong approach imo. The experiment isn't claiming AI can defeat a competent attacker. It's asking whether AI agents can replicate what a skilled (RE) specialist does on an unobfuscated binary. That's a legitimate, deployable use case (internal audit, code re…

Why does that matter? Being oblivious to obfuscated binaries is like failing the captcha test. Let's say instead of reversing, the job was to pick apples. Let's say an AI can pick all the apples in an orchard in normal weather conditions, but add overcast skies and success drops to zero. Is this, in your opinion, still a skilled apple picking specialist?

What if it’s 10x as fast during clear conditions? Then it doesn’t matter.

No hate. My only point is that’s it’s easy for analogies to fail. I can’t tell the point of either of your analogies, where the OP made several clear and cogent points.

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#94

Post by comex »

comex wrote: post #75
achille wrote: post #59

Earlier quoted context omitted.

in the article they explicitly said they stripped symbols. If you look at the actual backdoors many are already minimal and quite obfuscated, see: - https://github.com/QuesmaOrg/BinaryAudit/blob/main/tasks/dns... - https://github.com/QuesmaOrg/BinaryAudit/blob/main/tasks/dro...

The first one was probably found due to the reference to the string /bin/sh, which is a pretty obvious tell in this context. The second one is more impressive. I'd like to see the reasoning trace.

Reply to self: I managed to get their code running, since they seemingly haven’t published their trajectories. At least in my run (using Opus 4.6), it turns out that Claude is able to find the backdoored function because it’s literally the first function Claude checks.

Before even looking at the binary, Claude announces it will“look at the authentication functions, especially password checking logic which is a common backdoor target.” It finds the password checking function (svr_auth_password) using strings. And that is the function they decided to backdoor.

I’m experienced with reverse engineering but not experienced with these kinds of CTF-type challenges, so it didn’t occur to me that this function would be a stereotypical backdoor target…

They have a different task (dropbear-brokenauth2-detect) which puts a backdoor in a different function, and zero agents were able to find that one.

On the original task (dropbear-brokenauth-detect), in their runs, Claude reports the right function as backdoored 2 out of 3 times, but it also reports some function as backdoored 2 out of 2 times in the control experiment (dropbear-brokenauth-detect-negative), so it might just be getting lucky. The benchmark seemingly only checks whether the agent identifies which function is backdoored, not the specific nature of the backdoor. Since Claude guessed the right function in advance, it could hallucinate any backdoor and still pass.

But I don’t want to underestimate Claude. My run is not finished yet. Once it’s finished, I’ll check whether it identified the right function and, if so, whether it actually found the backdoor.

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#95

Post by xboxnolifes »

celeryd wrote: post #92
mbh159 wrote: post #88

The methodology debate in this thread is the most important part. The commenter who says "add obfuscation and success drops to zero" is right but that's also the wrong approach imo. The experiment isn't claiming AI can defeat a competent attacker. It's asking whether AI agents can replicate what a skilled (RE) specialist does on an unobfuscated binary. That's a legitimate, deployable use case (internal audit, code re…

Why does that matter? Being oblivious to obfuscated binaries is like failing the captcha test. Let's say instead of reversing, the job was to pick apples. Let's say an AI can pick all the apples in an orchard in normal weather conditions, but add overcast skies and success drops to zero. Is this, in your opinion, still a skilled apple picking specialist?

Maybe not, but also maybe you would no longer need skilled apple picking specialists.

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#96

Post by bitexploder »

radlad wrote: post #49
bitexploder wrote: post #23

Earlier quoted context omitted.

I can tell you how I am seeing agents be used with reasonable results. I will keep this high level. I don't rely on the agents solely. You build agents that augment your capabilities. They can make diagrams for you, give you an attack surface mapping, and dig for you while you do more manual work. As you work on an audit you will often find things of interest in a binary or code base that you want to investigate furt…

> But if you have a reasonable subset of "skills" / "agents" you can deploy for various auditing tasks it can absolutely speed you up some. Are people sharing these somewhere?

I put the terms in quotes because it can be as simple as a set of prompts you develop for various contexts. It really doesn't have to be too heavy of an idea.

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#97

Post by Avamander »

7777332215 wrote: post #26

I know they said they didn't obfuscate anything, but if you hide imports/symbols and obfuscate strings, which is the bare minimum for any competent attacker, the success rate will immediately drop to zero. This is detecting the pattern of an anomaly in language associated with malicious activity, which is not impressive for an LLM.

I have seen LLMs be surprisingly effective at figuring out such oddities. After all it has ingested knowledge of a myriad of data formats, encryption schemes and obfuscation methods.

If anything, complex logic is what'll defeat an LLM. But a good model will also highlight such logic being intractable.

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#98

Post by not_a9 »

achierius wrote: post #77
not_a9 wrote: post #43

Earlier quoted context omitted.

Things are pretty brutal and some categories are more affected than others. A/D seems to be somewhat less affected.

Brutal as in, heavy AI usage? What sort of categories are more affected?

From what someone told me rev/crypto/misc are the most broken, with pwn/web being more iffy and depending on challenge specifics.

I can't speak on AI usage very clearly (fun fact: just putting the challenge into ChatGPT's web UI sometimes works!), but I think the most egregious is orchestration platforms for agents (with MCP/whatever else) to autonomously solve challenges.

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#99

Post by godelski »

greazy wrote: post #89
godelski wrote: post #81

Earlier quoted context omitted.

It's a pretty common threshold, like 10% is. Be it the 80/20 "Pareto" rule, it's the value of one finger on one hand, or if you really want you stretch the p-value of 0.05 is 1 in 20 odds but that's definitely a stretch though arbitrary anyways. But 20 is a very human number and very common. It's just a division of 5 rather than 4 (I'm assuming you wouldn't have questioned a cutoff at 25%)

You've missed my point, it's not the thresholds, it's the categories assigned to the thresholds that need explaining.

You're right and I still don't understand your point.

Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

#100

Post by comex »

comex wrote: post #94
comex wrote: post #75

Earlier quoted context omitted.

The first one was probably found due to the reference to the string /bin/sh, which is a pretty obvious tell in this context. The second one is more impressive. I'd like to see the reasoning trace.

Reply to self: I managed to get their code running, since they seemingly haven’t published their trajectories. At least in my run (using Opus 4.6), it turns out that Claude is able to find the backdoored function because it’s literally the first function Claude checks. Before even looking at the binary, Claude announces it will“look at the authentication functions, especially password checking logic which is a common…

Update: It did find the backdoor! It spent an hour and a half mostly barking up various wrong trees and was about to "give my final answer" identifying the wrong function, but then said: "Actually, wait. Let me reconsider once more. [..] Let me look at one more thing - the password auth function. I want to double-check if there's a subtle bypass I missed." It disassembled it again, and this time it knew what the callee functions did and noticed the wrong function being called after failure.

Amusingly, it cited some Dropbear function names that it had not seen before, so it must have been relying in part on memorized knowledge of the Dropbear codebase.

Post reply on HN