The rev.ng decompiler goes open source

In truth, we suggest to do that only so you use the GCC we distribute for the demo binary. The actual way this is intended to be used is through the `./revng` script. In that way, the environment changes only affect the invocation of `revng`.

This is documented here: https://docs.rev.ng/user-manual/working-environment/ We should probably add a warning about `source ./environment`.

Now, let's get to each of your comments :D

> though thankfully not LD_LIBRARY_PATH

We spent a lot of time to have a completely self-contained set of binaries where each ELF refers to its dependencies through relative paths. LD_LIBRARY_PATH is evil.

> Mostly prefixed "HARD_"

Those are just used by our compiler wrappers, I don't think those environment variables collide with anything in practice.

> It sets `AWS_EC2_METADATA_DISABLED="true"`

Original discussion: https://github.com/revng/revng/pull/309#discussion_r12805759...

I guess we could patch the AWS SDK to avoid this. Anyway, it affects only when rev.ng is running in the cloud.

> export RPATH_PLACEHOLDER=... > export HARD_FLAGS_CXX_CLANG=...

Those are used when linking binaries translated by revng. If you're not interested in end-to-end binary translation, they don't matter.

> it means 'runpath' which is a similar but much less useful construct

We specifically want DT_RUNPATH. DT_RPATH is deprecated and there might an use case for overriding our libraries with LD_LIBRARY_PATH.

> There's a good chance you can completely remove the environment mangling

I think your observations concerning "mangling the environment" are only valid for non-private environment variables. The following variables are private: RPATH_PLACEHOLDER, HARD_*, REVNG_*. Also, they are all only for binary translation purposes. We could push them down into some smaller-scoped compiler wrappers, but those make sense only if we can get rid of environment entirely, which we can't because we ship Python.

> a combination of setting different flags when building clang

No, the flags also affect the linker and there's some features of our wrappers that cannot simply be burned in. We can push them in more private places, though.

> a lot of failure modes can be removed > libc++abi and libunwind can and probably should be statically linked into the libc++

We no longer have issues with that, our build system is pretty reliable in that regard. LLVM is just one of the components, these things need to work robustly in general, and they do (with quite some effort).

You seem to be wary of using dynamic linking, we put some effort in it, now it works pretty good and always looks up things in the right place, and without ever hardcoding absolute paths anywhere, nor any install phase that "patches" the binaries. The unpacked directory can be moved wherever you want.

> I am completely out of patience with distributing dynamically linked programs on Linux

You're thinking of some other solution, our solution does not use LD_LIBRARY_PATH and all the binaries reference each other in a robust way using `$ORIGIN`. Try:

    ./root/bin/python ./root/bin/revng artifact --help

But again, doing `source environment` is mostly for demo purposes, in the actual use case, you just do `./revng` and your environment is untouched.

We ship our Python, but you don't have to use it: you're supposed to just do ./revng (or interact over the network in daemon mode).

Our approach is: use whatever tool you like for scripting as long as it can parse our YAML project file, make changes to it, and then invoke `./revng artifact` (or interact with the daemon): https://docs.rev.ng/user-manual/model-tutorial/

Result: we get to use our Python version (the latest) and you get to use whatever language you like. Then we'll provide on pypi wrappers that help you with that and are compatible with large set of Python versions.

tl;dr Don't `source ./environment`, use `./revng`.

> Thanks again for shipping, and I hope some of the above feedback is helpful!

I'm happy there's someone that cares about this :D

Our next big iteration of this might involve simplifying things a lot by adopting nix + mount namespace to make /nix/store available without root.

Maybe this is not the right place for discussing this, we can chat on our discord server if you'd like :)

That's funny, because I've used both Hex-Rays and Ghidra, and gotten lots of help with data structures. The interactivity really helps a bunch with filling in the blanks.

One could take an pretrained model like llama, train it on only a few thousands of compiled and decompiled programs, then feed it compiled programs and have it decompile them and evaluate that output to make a new dataset and fine tune it again. Repeat until satisfactory.

I don't want to look at assembly code. I'd rather see expression trees, expressed in C-like syntax, than trying to piece together variables from two-address or three-address instructions. Looking at assembly tends to lead to brain farts like "wait, was the first or second operand the output operand?" (really, fuck AT&T syntax) or "wait, does ja implement ugt or sgt?"

So that means I want to look at something vaguely C-like. But the problem is that the C type system is too powerful for decompilers to robustly lift to, and the resulting code is generally at best filled with distractions of wait-I-can-fix-this excessive casting and at worst just wrong. And when it's wrong, I have to resort to staring at the assembly, which (for Ghidra at least) means throwing away a lot of the notes I've accumulated because they don't correlate back to underlying assembly.

So what I really want isn't something that can emit recompilable C code, that's optimizing for something that doesn't help me in the end. What I want is robust decompilation to something that lets me ignore the assembly entirely. I'm a compiler writer, I can handle a language where integers aren't signed but the operands are.