Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The rev.ng decompiler goes open source

rev.ng

Post reply on HN
Print view View original

21–30 of 63 posts

Re: The rev.ng decompiler goes open source

#21

Post by aleclm »

dvzk wrote: post #10
saagarjha wrote: post #9

Earlier quoted context omitted.

I hear this a lot and in my experience people who Ghidra or IDA and don’t use the decompiler are exceptionally rare. Why would you suffer that when you can use something else for what you actually want?

I didn't say I never use it, just that it's not always the core feature. This will depend heavily on your field, but in my past work, the features that were way more essential are: scripting (+ IR lifting), xrefs, CFGs, labels/notes (in a persistent DB). In my experience decompilers will totally ignore or fail on certain types of malicious code, so they mainly exist to assist disassembly analysis. And for that purpos…

For scripting, our approach is to give you access to the project file (just a YAML file), and you can make changes from any scripting language you want. Everything the user can customize is in there, all the rest is deterministically produced from that file.

I really disliked the fact that you usually need to buy into the version of Python that $TOOL requires you to use, or the fact itself that you need to use a specific language.

Can parse YAML? You're mostly done.

The "project file" is what we call the model: https://docs.rev.ng/user-manual/model-tutorial/

For xrefs, CFG and the rest: we have all of that in the UI, but we also produce them in a rich way. For instance, when we emit disassembly and decompiled code, we actually emit plain text + HTML-like markup to provide metainformation for navigation (basically, xrefs) and highlighting. So you can use all that from any language that can parse HTML/XML. It's called PTML: https://docs.rev.ng/references/ptml/

For lifting: we use LLVM IR as our internal representation. This means that: 1) you don't have to learn an IR that no one else uses, 2) you can use off the shelf tools (e.g., KLEE for symbolic execution) but you can also use all the standard LLVM optimizations and analyses and 3) you can recompile it, but we're not into the binary translation business anymore.

Re: The rev.ng decompiler goes open source

#22

Post by aleclm »

dvzk wrote: post #8
Fnoord wrote: post #2

Price model: > Very briefly: > The rev.ng framework is fully open source. You can decompile anything you want from the CLI. > The UI will be available in the following forms: > free to use in the cloud for public projects; > available through a subscription in the cloud for private projects; > available at a cost as a fully standalone, fully offline application. In comparison, Hopper costs 100 USD with one year of up…

Decompilation is often the least important (and least reliable) part of IDA/Ghidra, so comparing the two is unfair. That said, the scene is perpetually starved for good C decompilers, so more attempts are always exciting.

> Decompilation is often the least important (and least reliable) part of IDA/Ghidra

This is something all people using decompilers say and sort of shows how low is trust towards decompilers. Expectations have always been rather low.

I've been there, but this does not have to be the case, the whole reason why we started rev.ng is to prove that expectations can be raised.

Apart from accuracy, which is difficult but engineering work, why don't decompilers emit syntactically valid C? Have you ever tried to re-compile code from any decompiler? It's a terrible experience.

rev.ng only emits valid C code, and we test it with a bunch of -Wall -Wextra:

https://github.com/revng/revng-c/blob/develop/share/revng-c/...

Other key topic: data structures. When reversing I spend half of the time renaming things and half of the time detecting data structures. The help I get from decompilers in latter is basically none.

rev.ng, by default, detects data structures on the whole binary, interprocedurally, including arrays. See the linked list example in the blog post. We also have plans to detect enums and other stuff.

Clearly we're not there yet, we still need to work on robustness, but our goal is to increase the confidence in decompilers and actually offer features that save time. Certain tools have made progress in improving the UI and the scripting experience, but there's other things to do beyond that.

I see this a bit like the transition from the phase in which C developers where using macros to ensure things were being inlined/unrolled to the phase where they stopped doing that because compilers got smart enough to the right thing and to do it much more effectively.

Re: The rev.ng decompiler goes open source

#23

Post by aleclm »

yakkityyak wrote: post #20

I hope collaborative workflows get a lot of attention. I haven't used IDA teams or anything, but a reverse engineering experience that felt as frictionless as Google Docs would be amazing.

That's our goal. We used to use QtCreator as a basis for the UI, terrible idea.

Then we switched to VSCode, which happens to be able to run in the browser. So we added some magic kubernetes sauce and voilà, you got the cloud decompiler with exactly the same user experience as the fully standalone one.

We still need to perform some QA on collaboration, but basically works. One daemon, many clients. Very simple architecture.

I think we got inspiration to do this from a CTF where we were doing "collaboration" using IDA with multiple windows on a X session on a server with multiple cursors. Very cursed, but effective.

Re: The rev.ng decompiler goes open source

#24

Post by znpy »

aleclm wrote: post #21
dvzk wrote: post #10

Earlier quoted context omitted.

I didn't say I never use it, just that it's not always the core feature. This will depend heavily on your field, but in my past work, the features that were way more essential are: scripting (+ IR lifting), xrefs, CFGs, labels/notes (in a persistent DB). In my experience decompilers will totally ignore or fail on certain types of malicious code, so they mainly exist to assist disassembly analysis. And for that purpos…

For scripting, our approach is to give you access to the project file (just a YAML file), and you can make changes from any scripting language you want. Everything the user can customize is in there, all the rest is deterministically produced from that file. I really disliked the fact that you usually need to buy into the version of Python that $TOOL requires you to use, or the fact itself that you need to use a spec…

> 3) you can recompile it, but we're not into the binary translation business anymore

How comes?

Re: The rev.ng decompiler goes open source

#25

Post by aleclm »

znpy wrote: post #24
aleclm wrote: post #21

Earlier quoted context omitted.

For scripting, our approach is to give you access to the project file (just a YAML file), and you can make changes from any scripting language you want. Everything the user can customize is in there, all the rest is deterministically produced from that file. I really disliked the fact that you usually need to buy into the version of Python that $TOOL requires you to use, or the fact itself that you need to use a spec…

> 3) you can recompile it, but we're not into the binary translation business anymore How comes?

Short answer: if you want to execute a program (maybe with some instrumentation, for fuzzing purposes) it's much easier to adopt a dynamic approach (i.e., emulation or virtualization). With static binary translation you can get better performance, but there's a lot of other things you need to get 100% right and that with a dynamic approach are a given (e.g., the CFG).

There's much more space of improvement in the field of analyzing code (as opposed to running it), so we're investing our energies there.

Then we're strong believers in integrating dynamic and static information, for instance see PageBuster: https://rev.ng/blog/pagebuster

But other than that, static binary translation is a feature of rev.ng in maintenance mode.

Re: The rev.ng decompiler goes open source

#26

Post by londons_explore »

aleclm wrote: post #17
londons_explore wrote: post #13

Idea: automatically name variables and members of structs based on how code interacts with them. Eg. The next pointer in a linked list should be easy to identify as 'next'. That would be done by downloading all of GitHub, then seeing what variables in GitHub code have the most similar layouts and interactions, and then if the confidence is high enough, using those names.

In the past we were thinking to do something like this by hand. For instance, we detect induction variables, we could rename them into `i`. However, nowadays, it seems pretty obvious that the right way to do this things is using LLMs. This said, at this stage, we see ourselves as people building robust infrastructure. Once the infrastructure is there, using some off the shelf model to rename things or add comments is…

If an LLM is used, it's unclear how to best do it.

One could try to train ones own LLM from scratch, using an encoder-decoder (translation - aka seq2seq) architecture trying to predict the correct variable name given the decompiled output.

One could try to use something like GPT-4 with a carefully designed prompt "Given this datastructure, what might be the name for this field?"

One could try to use something pretrained like llama, but then finetune it based on hundreds of thousands of compiled and decompiled programs.

Re: The rev.ng decompiler goes open source

#27

Post by mrexodia »

userbinator wrote: post #18
8organicbits wrote: post #12

Earlier quoted context omitted.

I've been planning to use a non-enforcement model for a future project. Some users will always pay, because of corporate policy or ethics. Some will never pay and will reverse engineer out any software license checks. Asking the user if they have a license keeps the honest ones honest and permits ad-hoc free trials, emergency use, and other reasonable "unlicensed use".

Some will never pay and will reverse engineer out any software license checks. For a long time (and might still be; not paying much attention anymore), it was a "rite of passage" in the scene to crack IDA... using itself.

It never was a “rite of passage”, because removing IDA’s license checks has always been trivial…

Re: The rev.ng decompiler goes open source

#28

Post by albertzeyer »

albertzeyer wrote: post #5

Checking the team about: https://rev.ng/about And looking at the code contributions: https://github.com/revng/revng/graphs/contributors Isn't it a bit weird that the CEO (aleclearmind) has most commits, even much more than the CTO (pfez)? I often hear the complaints from other CEOs that they don't really find any time anymore to code... Even the CTO usually is more on the managing side and less active in actual codin…

I wonder a bit about the downvotes. I didn't mean this as a criticism or so in any way. In fact, I like this very much. I just found this interesting and unlike what I saw elsewhere.

So the downvotes are because this is not interesting or not unusual?

Re: The rev.ng decompiler goes open source

#29

Post by saagarjha »

aleclm wrote: post #22
dvzk wrote: post #8

Earlier quoted context omitted.

Decompilation is often the least important (and least reliable) part of IDA/Ghidra, so comparing the two is unfair. That said, the scene is perpetually starved for good C decompilers, so more attempts are always exciting.

> Decompilation is often the least important (and least reliable) part of IDA/Ghidra This is something all people using decompilers say and sort of shows how low is trust towards decompilers. Expectations have always been rather low. I've been there, but this does not have to be the case, the whole reason why we started rev.ng is to prove that expectations can be raised. Apart from accuracy, which is difficult but en…

Curious what you do when you encounter an instruction you don't model

Re: The rev.ng decompiler goes open source

#30

Post by aleclm »

saagarjha wrote: post #29
aleclm wrote: post #22

Earlier quoted context omitted.

> Decompilation is often the least important (and least reliable) part of IDA/Ghidra This is something all people using decompilers say and sort of shows how low is trust towards decompilers. Expectations have always been rather low. I've been there, but this does not have to be the case, the whole reason why we started rev.ng is to prove that expectations can be raised. Apart from accuracy, which is difficult but en…

Curious what you do when you encounter an instruction you don't model

That's unlikely, since we use QEMU as a lifter, which sometimes supports new instructions before they hit silicon.

However, I think we'll emit a call to some `noreturn` function. Basically we emit a call to `abort`.

Post reply on HN