And this one demonstration why these "1000 CTOs claim no effectiveness improvement after introducing AI in their companies" are 100% BS.
They may have not noticed an improvement, but it doesn't mean there isn't any.
We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
31–40 of 108 posts
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#32I highly doubt some of those results, GPT 5.2/+codex is incredible for cyber security and CTFs, and 5.3 Codex (not on API yet) even moreso. There is absolutely no way it's below Deepseek or Haiku. Seems like a harness issue, or they tested those models at none/low reasoning?
As I do eval and training data sets for living, in niche skills, you can find plenty of surprises. The code is open-source; you can run it yourself using Harbor Framework: git clone git@github.com:QuesmaOrg/BinaryAudit.git export OPENROUTER_API_KEY=... harbor run --path tasks --task-name lighttpd-* --agent terminus-2 --model openrouter/anthropic/claude-opus-4.6 --model openrouter/google/gemini-3-pro-preview --model o…
Just for fun, I ran dnsmasq-backdoor-detect-printf (which has a 0% pass rate in your leaderboard with GPT models) with --agent codex instead of terminus-2 with gpt-5.2-codex and it identified the backdoor successfully on the first try. I honestly think it's a harness issue, could you re-run the benchmarks with Codex for gpt-5.2-codex and gpt-5.2?
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#33And this one demonstration why these "1000 CTOs claim no effectiveness improvement after introducing AI in their companies" are 100% BS. They may have not noticed an improvement, but it doesn't mean there isn't any.
it also generally takes a heck of a noisy bang for internal developments to make it to the c-suite
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#34[dead]
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#35Earlier quoted context omitted.
Start with trying to find the xz vulnerability and other software possibly tying into that. Is there code that does something completely different than its comments claim?
Another way to phrase what I am asking is ... Does AI understand the context of code deep enough to know everything a piece of code can do , everything a service can do vs. what it was intended to do . If it can understand code that far then it could understand all the potential paths data could flow and thus all the potential vulnerabilities that several piece of code together could achieve when used in concert with…
Some models can,
depending on the length of the piece of code,
is probably the most honest answer right now.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#36GPT is impressive with a consistent 0% false positive rate across models, yet its ability to detect is as high as 18%. Meanwhile Claude Opus 4.6 is able to detect up to 46% of backdoors, but has a 22% false positive rate. It would be interesting to have an experiment where these models are able to test exploiting but their alignment may not allow that to happen. Perhaps combining models together can lead to that kind…
It would be really cool if someone developed some standard language and methodology for measuring the success of binary classificaiton tasks...
Oh, wait, we have had that for a hundred years - somehow it's just entirely forgotten when generative models are involved.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#37> The executables in our benchmark often have hundreds or thousands of functions — while the backdoors are tiny, often just a dozen lines buried deep within. Finding them requires strategic thinking: identifying critical paths like network parsers or user input handlers and ignoring the noise. Perhaps it would make sense to provide LLMs with some strategy guides written in .md files.
That's what I thought of too. Given their task formulation (they basically said - "check these binaries with these tools at your disposal" - and that's it!) their results are already super impressive. With a proper guidance and professional oversight it's a tremendous force multiplier.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#38Earlier quoted context omitted.
Another way to phrase what I am asking is ... Does AI understand the context of code deep enough to know everything a piece of code can do , everything a service can do vs. what it was intended to do . If it can understand code that far then it could understand all the potential paths data could flow and thus all the potential vulnerabilities that several piece of code together could achieve when used in concert with…
Some models can , depending on the length of the piece of code, is probably the most honest answer right now.
Fair enough. I suspect when they reach such a point that length no longer matters then a plethora of old and currently used state sponsored complex malware will be realized. Beyond that I think the next step would be to attain attribution to both individuals and perhaps whom they were really employed by. Bonus if the model can rewrite sanitize each piece of code to remove the malicious capabilities without breaking the officially intended functions.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#39Earlier quoted context omitted.
As I do eval and training data sets for living, in niche skills, you can find plenty of surprises. The code is open-source; you can run it yourself using Harbor Framework: git clone git@github.com:QuesmaOrg/BinaryAudit.git export OPENROUTER_API_KEY=... harbor run --path tasks --task-name lighttpd-* --agent terminus-2 --model openrouter/anthropic/claude-opus-4.6 --model openrouter/google/gemini-3-pro-preview --model o…
Are the existing trajectories from your runs published anywhere? Or is the only way is for me to run them again?
I can provide trajectories. Though probably we are not going to publish them this time. This would need some extra safeguards.
Email me. The address is in profile.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#40I know they said they didn't obfuscate anything, but if you hide imports/symbols and obfuscate strings, which is the bare minimum for any competent attacker, the success rate will immediately drop to zero. This is detecting the pattern of an anomaly in language associated with malicious activity, which is not impressive for an LLM.
When I was developing my ghidra-cli tool for LLMs to use, I was using crackmes as tests and it had no problem getting through obfuscation as long as it was prompted about it. In practice when reverse engineering real software it can sometimes spin in circles for a while until it finally notices that it's dealing with obfuscated code, but as long as you update your CLAUDE.md/whatever with its findings, it generally moves smoothly from then on.