[dead]
We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
41–50 of 108 posts
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#42[dead]
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#43Random thoughts, only vaguely related: what’s the impact of AI on CTFs? I would assume that kills part of the fun of such events?
Things are pretty brutal and some categories are more affected than others.
A/D seems to be somewhat less affected.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#44>While end-to-end malware detection is not reliable yet, AI can make it easier for developers to perform initial security audits. A developer without reverse engineering experience can now get a first-pass analysis of a suspicious binary. [...] The whole field of working with binaries becomes accessible to a much wider range of software engineers. It opens opportunities not only in security, but also in performing low-level optimization, debugging and reverse engineering hardware, and porting code between architectures.
THIS is the takeaway. These tools are allowing *adjacency* to become a powerful guiding indicator. You don't need to be a reverser, you can just understand how your software works and drive the robot to be a fallible hypothesis generator in regions where you can validate only some of the findings.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#45Shameless plug: https://github.com/akiselev/ghidra-cli I’ve been using Ghidra to reverse engineer Altium’s file format (at least the Delphi parts) and it’s insane how effective it is. Models are not quite good enough to write an entire parser from scratch but before LLMs I would have never even attempted the reverse engineering. I definitely would not depend on it for security audits but the latest models are more th…
How does this approach compare to the various Ghidra MCP servers?
I also did this approach (scripts + home-brew cli)...because I didn't know Ghidra MCP servers existed when I got started.
So I don't have a clear idea of what the comparison would be but it worked pretty well for me!
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#46Shameless plug: https://github.com/akiselev/ghidra-cli I’ve been using Ghidra to reverse engineer Altium’s file format (at least the Delphi parts) and it’s insane how effective it is. Models are not quite good enough to write an entire parser from scratch but before LLMs I would have never even attempted the reverse engineering. I definitely would not depend on it for security audits but the latest models are more th…
This is really cool! Thanks for sharing. It's a lot more sophisticated than what I did w/ Ghidra + LLMs.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#47I know they said they didn't obfuscate anything, but if you hide imports/symbols and obfuscate strings, which is the bare minimum for any competent attacker, the success rate will immediately drop to zero. This is detecting the pattern of an anomaly in language associated with malicious activity, which is not impressive for an LLM.
Isn’t LLM supposed to be better at analyzing obfuscated than heuristics? Because of its ability to pattern match it can deduce what obfuscated code does?
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#48> The executables in our benchmark often have hundreds or thousands of functions — while the backdoors are tiny, often just a dozen lines buried deep within. Finding them requires strategic thinking: identifying critical paths like network parsers or user input handlers and ignoring the noise. Perhaps it would make sense to provide LLMs with some strategy guides written in .md files.
That's what I thought of too. Given their task formulation (they basically said - "check these binaries with these tools at your disposal" - and that's it!) their results are already super impressive. With a proper guidance and professional oversight it's a tremendous force multiplier.
We are in this super weird space where the comparable tasks are one-shot, e.g. "make me a to-do app" or "check these binaries", but any real work is multi-turn and dynamically structured.
But when we're trying to share results, "a talented engineer sat with the thread and wrote tests/docs/harnesses to guide the model" is less impressive than "we asked it and it figured it out," even though the latter is how real work will happen.
It creates this perverse scenario (which is no one's fault!) where we talk about one-shot performance but one-shot performance is useful in exactly 0 interesting cases.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#49Shameless plug: https://github.com/akiselev/ghidra-cli I’ve been using Ghidra to reverse engineer Altium’s file format (at least the Delphi parts) and it’s insane how effective it is. Models are not quite good enough to write an entire parser from scratch but before LLMs I would have never even attempted the reverse engineering. I definitely would not depend on it for security audits but the latest models are more th…
I can tell you how I am seeing agents be used with reasonable results. I will keep this high level. I don't rely on the agents solely. You build agents that augment your capabilities. They can make diagrams for you, give you an attack surface mapping, and dig for you while you do more manual work. As you work on an audit you will often find things of interest in a binary or code base that you want to investigate furt…
> But if you have a reasonable subset of "skills" / "agents" you can deploy for various auditing tasks it can absolutely speed you up some.
Are people sharing these somewhere?
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#50The fact that Gemini returns the highest rate of fake positives aligns with my experience using the Gemini models. I use ChatGPT, Claude and Gemini regularly and Gemini is clearly the most sycophantic of the three. If I ask those three models to evaluate something or estimate odds of success, Gemini always comes back with the rosiest outlook.
I had been searching for a good benchmark that provided some empirical evidence of this sycophancy, but I hadn't found much. Measuring false positives when you ask the model to complete a detection related task may be a good way of doing that.