Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The rev.ng decompiler goes open source

rev.ng

Post reply on HN
Print view View original

51–60 of 63 posts

Re: The rev.ng decompiler goes open source

#51

Post by aleclm »

dark-star wrote: post #46

It doesn't work with my ELF file: [orchestra] [darkstar@shiina revng]$ ./revng artifact --analyze --progress decompile-to-single-file ../maytag.ko [=======================================] 100% 0.57s Analysis list revng-initial-auto-analysis (5): import-binary [===================> ] 50% 0.57s Run analyses lists (2): revng-initial-auto-analysis [=========> ] 25% 0.57s revng-artifact (2): Run analyses Only ELF executa…

Can you open an issue on GitHub and attach the binary? I don't think it should be too hard to load that.

Re: The rev.ng decompiler goes open source

#52

Post by aleclm »

JonChesterfield wrote: post #50
aleclm wrote: post #41

Earlier quoted context omitted.

I think most of your concerns about messing with the environment are sensible only under the assumption that you actually do `source environment`. In truth, we suggest to do that only so you use the GCC we distribute for the demo binary. The actual way this is intended to be used is through the `./revng` script. In that way, the environment changes only affect the invocation of `revng`. This is documented here: https…

Not setting environment variables is indeed solved by not setting environment variables - but `source ./environment` is what's written on the announcement page at the top of this thread. './revng' doesn't appear anywhere on it. You haven't set LD_LIBRARY_PATH but other people will do. Also LIBRARY_PATH, and put other stuff on PATH and so forth. Module systems are especially prone to this, but ending up with .bashrc d…

> `source ./environment` is what's written on the announcement page at the top of this thread. './revng' doesn't appear anywhere on it.

You're right, but after that there's a link to the docs where we say to use `./revng`. The blog post is for the impatient :) On the long run the docs is what most people will look at.

I don't think we want to support use cases that might break system packages too. If you set LD_LIBRARY_PATH to a directory where you have an LLVM installation, that might break any system program using LLVM too... Why should we try to fix that using `DT_RPATH` (which is a deprecated way of doing things) when system components don't do it?

We might cleanup the environment from LD_LIBRARY_PATH and other stuff, that might be a sensible default, yeah. Also we might have some sanity check printing a warning if weird libraries are pulled in.

But it's hard to take a decision without a specific use case in mind. If you have an example, bring it forward and I'm happy to discuss what should be the right approach there.

Re: The rev.ng decompiler goes open source

#53

Post by aleclm »

jcranmer wrote: post #49
aleclm wrote: post #22

Earlier quoted context omitted.

> Decompilation is often the least important (and least reliable) part of IDA/Ghidra This is something all people using decompilers say and sort of shows how low is trust towards decompilers. Expectations have always been rather low. I've been there, but this does not have to be the case, the whole reason why we started rev.ng is to prove that expectations can be raised. Apart from accuracy, which is difficult but en…

Here's my issue with decompilers: I don't want to look at assembly code. I'd rather see expression trees, expressed in C-like syntax, than trying to piece together variables from two-address or three-address instructions. Looking at assembly tends to lead to brain farts like "wait, was the first or second operand the output operand?" (really, fuck AT&T syntax) or "wait, does ja implement ugt or sgt?" So that means I…

I 120% agree with what you're saying, but emitting valid C is kinda part of what you're asking, in design terms.

Our goal is: omit all the casts that can be omitted without changing the semantics according to C. In fact, we have a PR doing exactly this (still on the old repo, hopefully it will go in soon).

But, how can you expect to be able to be strict with what C allows you to do implicitly, if you're not even emitting valid C? For instance, thanks to the fact that we emit valid C, we could test if the assembly emitted by a compiler is the same before and after removing redundant casts.

My point is that emitting valid C is kind of a prerequisite for what you're asking, a rather low bar to pass, but that, in practice, no mainstream decompiler passes. It's pretty obvious the decompiled code will often be redundant and outright wrong if you don't even guarantee it's syntactically valid. Then clearly it's not a panacea, but it's an important design criterion and shows the direction we want to go.

As for comments: we still haven't implemented inline comments, but they will be attached to program addresses, so they will be available both in disassembly and decompiled C. It's not very hard to do, but that needs some love.

Re: The rev.ng decompiler goes open source

#55

Post by aleclm »

costco wrote: post #54

Congrats. Do you have any regrets about outsourcing lifting to the QEMU TCG or has it worked well?

Thanks!

It has been working very well. Two regrets:

1. Not rebasing our fork of QEMU for years has put us in a bad spot. But just today a member of our team managed to lift stuff with the latest QEMU. And he has also been able to lift Qualcomm Hexagon code, for which we helped to add support in QEMU. Eventually we'll be the first proper Hexagon decompiler :)

2. Focusing too much on QEMU led our frontend to be tightly coupled with QEMU. It will now take some effort to enable support for additional frontends, non-QEMU based. But not impossible: our idea is to let user add support for a new architecture by defining, in C, a struct for the CPU state and a bunch of functions acting on it. That's it. No need to learn any internal representation.

tl;dr QEMU was a great choice, it worked so well that we didn't work on that part of the codebase for too much time and now there's some technical debt there. But we're addressing it.

Re: The rev.ng decompiler goes open source

#56

Post by jcranmer »

aleclm wrote: post #53
jcranmer wrote: post #49

Earlier quoted context omitted.

Here's my issue with decompilers: I don't want to look at assembly code. I'd rather see expression trees, expressed in C-like syntax, than trying to piece together variables from two-address or three-address instructions. Looking at assembly tends to lead to brain farts like "wait, was the first or second operand the output operand?" (really, fuck AT&T syntax) or "wait, does ja implement ugt or sgt?" So that means I…

I 120% agree with what you're saying, but emitting valid C is kinda part of what you're asking, in design terms. Our goal is: omit all the casts that can be omitted without changing the semantics according to C. In fact, we have a PR doing exactly this (still on the old repo, hopefully it will go in soon). But, how can you expect to be able to be strict with what C allows you to do implicitly, if you're not even emit…

One of the blog posts I keep meaning to write but never quite get around to is a post that C is not portable assembly. What is necessary is decompilation to a portable C-like assembly, but that target is not C, and I think focusing on creating valid C tends to drag you towards suboptimal decisions, even leaving aside issues like "should SLL decompile to x In my experience with Ghidra, I've just seen far too many times where Ghidra starts with wrong types for something and the result becomes gibberish--even just plain dropping stuff altogether. There are some cases where it's clear it's just poor analysis on Ghidra's part (e.g., it doesn't seem to understand stack slot reuse, and memcpy-via-xmm is very confusing to it). And Ghidra's type system lacks function pointer types, which is very annoying when you're doing vtable-heavy C++ code.

I do like the appeal of a recompileable target language. But that language need not be C--in fact, I'm actually sketching out the design of such a language for my own purposes in being able to read LLVM IR without going crazy (which means I need to distinguish between, e.g., add nuw and just plain add).

Analysis necessarily involves multiple levels. Given that a lot of the type analysis today tends to be crap, I'd rather prefer to have the ability to see a more solid first-level analysis that does variable recovery and works out function calling conventions so that it can inform my ability to reverse engineer structures or things like "does this C++ method return a non-trivial struct that is an implicit first parameter?"

(Also, since I'm largely looking at C++ code in practice, I'd absolutely love to be able to import C++ header files to fill in known structure types.)

Re: The rev.ng decompiler goes open source

#57

Post by aleclm »

jcranmer wrote: post #56
aleclm wrote: post #53

Earlier quoted context omitted.

I 120% agree with what you're saying, but emitting valid C is kinda part of what you're asking, in design terms. Our goal is: omit all the casts that can be omitted without changing the semantics according to C. In fact, we have a PR doing exactly this (still on the old repo, hopefully it will go in soon). But, how can you expect to be able to be strict with what C allows you to do implicitly, if you're not even emit…

One of the blog posts I keep meaning to write but never quite get around to is a post that C is not portable assembly. What is necessary is decompilation to a portable C-like assembly, but that target is not C, and I think focusing on creating valid C tends to drag you towards suboptimal decisions, even leaving aside issues like "should SLL decompile to x In my experience with Ghidra, I've just seen far too many time…

> should SLL decompile to x I think this a bit of a misguided question. The hardware has a precise semantic defined, usually. QEMU's 32), but this means that the lifter (still QEMU) will account for this and emit code preserving the semantics.

tl;dr: the code we emit should do the right thing depending on what the original instruction did, without making assumptions on what happens in case of C undefined behaviors.

> Ghidra's type system lacks function pointer types

Weird limitation, we support those.

> it doesn't seem to understand stack slot reuse

That's a tricky one. We're now re-designing certain parts of the pipeline to enable LLVM to promote stack accesses to SSA values, which basically solves the stack slot reuse. This is probably one of the most important features experienced reversers ask for.

> that language need not be C--

Making up your own language is temptation one should resist.

Anyway, we're rewriting our backend using an MLIR dialect (we call it clift) which targets C but should be good enough to emit something "similar to C but slightly different". It might make sense to have a different backend there. But a "standard C" backend has to be the first use case.

We thought about emitting C++, it would make our life simpler. But I think targeting non-C as the first and foremost backend would be a mistake.

Also, a Python backend would be cool.

> Analysis necessarily involves...

I would be interested in discussing more what exactly you mean here. Why don't you join our discord server?

> I'd absolutely love to be able to import C++ header files to fill in known structure types

We have a project for importing from header files. Basically we want use a compiler to turn them into DWARF debug symbols and then import those. Not too hard.

Re: The rev.ng decompiler goes open source

#58

Post by JonChesterfield »

aleclm wrote: post #52
JonChesterfield wrote: post #50

Earlier quoted context omitted.

Not setting environment variables is indeed solved by not setting environment variables - but `source ./environment` is what's written on the announcement page at the top of this thread. './revng' doesn't appear anywhere on it. You haven't set LD_LIBRARY_PATH but other people will do. Also LIBRARY_PATH, and put other stuff on PATH and so forth. Module systems are especially prone to this, but ending up with .bashrc d…

> `source ./environment` is what's written on the announcement page at the top of this thread. './revng' doesn't appear anywhere on it. You're right, but after that there's a link to the docs where we say to use `./revng`. The blog post is for the impatient :) On the long run the docs is what most people will look at. I don't think we want to support use cases that might break system packages too. If you set LD_LIBRA…

LLVM picking up the wrong libraries from the environment has cost me at least a couple of months over the last decade or so. Maybe twenty instances of customers being broken, ten hours or so in meetings explaining the problem and trying to persuade people that the right thing really is different for the system compiler vs your bespoke thing.

If you think it's better for your product to find unrelated libraries with the same name at runtime, you go for it.

Detecting that failure mode would be an interesting exercise - you could crawl your own address space after startup and try to guess whether the libraries you got are the ones you wanted. Probably implementable.

Re: The rev.ng decompiler goes open source

#59

Post by Sesse__ »

aleclm wrote: post #44
Sesse__ wrote: post #43

Earlier quoted context omitted.

> Other key topic: data structures. When reversing I spend half of the time renaming things and half of the time detecting data structures. The help I get from decompilers in latter is basically none. That's funny, because I've used both Hex-Rays and Ghidra, and gotten lots of help with data structures. The interactivity really helps a bunch with filling in the blanks.

In IDA you basically have only detection of stack frame layout (in a quite confusing fashion) and "create struct out of this pointer", which is something you have to do manually and its intraprocedural. Imagine this being done automatically, across all of the binary. If you pass a pointer to another function the type is correct and you build the type from all the functions using it. Then obviously the user needs to f…

I'm sure user-defined structs can benefit from combining information from multiple functions, but saying that what you get today is “basically none” is a bit of an overstatement. Also, the special (and important!) case of operating system ABI structs is great, and that information propagates throughout function calls.

Re: The rev.ng decompiler goes open source

#60

Post by dark-star »

aleclm wrote: post #51
dark-star wrote: post #46

It doesn't work with my ELF file: [orchestra] [darkstar@shiina revng]$ ./revng artifact --analyze --progress decompile-to-single-file ../maytag.ko [=======================================] 100% 0.57s Analysis list revng-initial-auto-analysis (5): import-binary [===================> ] 50% 0.57s Run analyses lists (2): revng-initial-auto-analysis [=========> ] 25% 0.57s revng-artifact (2): Run analyses Only ELF executa…

Can you open an issue on GitHub and attach the binary? I don't think it should be too hard to load that.

can I somehow share the binary privately? It's a proprietary module that I probably shouldn't share publically (also it's ... rather large)

I opened issue #366 for it already

Post reply on HN