We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
1–10 of 108 posts
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#2See direct benchmark link: https://quesma.com/benchmarks/binaryaudit/
Open-source GitHub: https://github.com/QuesmaOrg/BinaryAudit
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#3Along this line can AI's find backdoors spread across multiple pieces of code and/or services? i.e. by themselves they are not back-doors, advanced penetration testers would not suspect anything is afoot but when used together they provide access.
e.g. an intentional weakness in systemd + udev + binfmt magic when used together == authentication and mandatory access control bypass. Each weakness reviewed individually just looks like benign sub-optimal code.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#4[flagged]
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#5Along this line can AI's find backdoors spread across multiple pieces of code and/or services? i.e. by themselves they are not back-doors, advanced penetration testers would not suspect anything is afoot but when used together they provide access. e.g. an intentional weakness in systemd + udev + binfmt magic when used together == authentication and mandatory access control bypass. Each weakness reviewed individually…
Start with trying to find the xz vulnerability and other software possibly tying into that.
Is there code that does something completely different than its comments claim?
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#6> The executables in our benchmark often have hundreds or thousands of functions — while the backdoors are tiny, often just a dozen lines buried deep within. Finding them requires strategic thinking: identifying critical paths like network parsers or user input handlers and ignoring the noise.
Perhaps it would make sense to provide LLMs with some strategy guides written in .md files.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#7Shameless plug: https://github.com/akiselev/ghidra-cli
I’ve been using Ghidra to reverse engineer Altium’s file format (at least the Delphi parts) and it’s insane how effective it is. Models are not quite good enough to write an entire parser from scratch but before LLMs I would have never even attempted the reverse engineering.
I definitely would not depend on it for security audits but the latest models are more than good enough to reverse engineer file formats.
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#8Very, very cool. Besides the top-performing models, it's interesting (if I'm reading this correctly) that gpt-5.2 did ~2x better than gpt-5.2-codex.. why?
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#9Shameless plug: https://github.com/akiselev/ghidra-cli I’ve been using Ghidra to reverse engineer Altium’s file format (at least the Delphi parts) and it’s insane how effective it is. Models are not quite good enough to write an entire parser from scratch but before LLMs I would have never even attempted the reverse engineering. I definitely would not depend on it for security audits but the latest models are more th…
[dead]
Re: We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them
#10Shameless plug: https://github.com/akiselev/ghidra-cli I’ve been using Ghidra to reverse engineer Altium’s file format (at least the Delphi parts) and it’s insane how effective it is. Models are not quite good enough to write an entire parser from scratch but before LLMs I would have never even attempted the reverse engineering. I definitely would not depend on it for security audits but the latest models are more th…
Oh, nice find... We end up using PyGhidra, but the models waste some cycles because of bad ergonomics. Perhaps your cli would be easier.
Still, Ghidra's most painful limitation was extremely slow time with Go Lang. We had to exclude that example from the benchmark.