Probably one or two years ago I randomly met this wonderful tool amidst dealing with a VSCode Remote SSH problem via search. The story begins with using VSCode remote SSH on a HPC that is running CentOS 6. VSCode Remote SSH ship with a Node binary that is dynamically linked to a glibc that is not supported on older OSes such as CentOS 6 https://code.visualstudio.com/docs/remote/linux . I am not the system manager on the HPC and could not update the system myself, and VSCode team is too arrogant to support those old OSes. Alhough I have a homebrew environment and has a new glibc built there, the Node VSCode does not recognize that from environment variable. Manually update string in the node binary’s strings does provide me with solution at least: https://github.com/microsoft/vscode-remote-release/issues/10...
As a researcher I am really thankful of the developer who created this wonderful tool.
PatchELF: Simple utility for modifying existing ELF executables and libraries
31–40 of 42 posts
Re: PatchELF: Simple utility for modifying existing ELF executables and libraries
#32Probably one or two years ago I randomly met this wonderful tool amidst dealing with a VSCode Remote SSH problem via search. The story begins with using VSCode remote SSH on a HPC that is running CentOS 6. VSCode Remote SSH ship with a Node binary that is dynamically linked to a glibc that is not supported on older OSes such as CentOS 6 https://code.visualstudio.com/docs/remote/linux . I am not the system manager on…
Also another reason I have to come up with this is though VSCode is open-sourced, VSCode’s remote SSH extension is private. I had to look into the extension module’s uglified source code to figure out what is happening out there… Alternative solution would involve edit the uglified js file https://github.com/microsoft/vscode-remote-release/issues/10...
Re: PatchELF: Simple utility for modifying existing ELF executables and libraries
#33Patchelf is almost always broken, and when they merge fixes they don't release a new version right away. Between 2014-2019 running patchelf+strip would corrupt binaries, after the fix running patchelf twice on the same binary would corrupt it. And now patchelf doesn't work on all binaries, there's a bugfix merged https://github.com/NixOS/patchelf/pull/230 merged Nov 19, 2020, but no release since then. Also patchelf…
elf files ARE hard, the documentation is best described as "adequate". You can often learn a lot of things looking at how GCC generates ELF files that might not be obvious from looking at the ELF spec.
Re: PatchELF: Simple utility for modifying existing ELF executables and libraries
#34Patchelf is almost always broken, and when they merge fixes they don't release a new version right away. Between 2014-2019 running patchelf+strip would corrupt binaries, after the fix running patchelf twice on the same binary would corrupt it. And now patchelf doesn't work on all binaries, there's a bugfix merged https://github.com/NixOS/patchelf/pull/230 merged Nov 19, 2020, but no release since then. Also patchelf…
Nixos has been my main distro for ages now, and I gotta admit, I'm still a little bit surprised every time patchelf works
Re: PatchELF: Simple utility for modifying existing ELF executables and libraries
#35Patchelf is almost always broken, and when they merge fixes they don't release a new version right away. Between 2014-2019 running patchelf+strip would corrupt binaries, after the fix running patchelf twice on the same binary would corrupt it. And now patchelf doesn't work on all binaries, there's a bugfix merged https://github.com/NixOS/patchelf/pull/230 merged Nov 19, 2020, but no release since then. Also patchelf…
patchelf is a lifesaver for handling Android/iOS with libraries compiled outside the regular toolchain (both platforms can be very finicky about soname/rpath).
On one occasion, however, I did spend days chasing a crash that turned out to be a bug when using an older version of patchelf on ARM binaries.
So great tool, but definitely be wary of introducing weird flakiness.
Re: PatchELF: Simple utility for modifying existing ELF executables and libraries
#36Earlier quoted context omitted.
I'm not really sure why it has to be that hard, like, why don't we just use base64-encoded JSON or something?
> why don't we just use base64-encoded JSON or something? Base64 would be counterproductive, increasing both space and parsing time... presumably out of fear that JSON would contain a naughty byte for a greenfield file format. It would be much better to just design tho format to not have any naughty bytes. Parsing time for exacutables and libraries is definitely on the critical startup path. You really want a length-…
That's a good point about the critical path - I was thinking it would be a bit bigger and slower since you'd have to decode it, but I hadn't realized what an impact that would probably have. No bit-perfect round trips is also absolutely horrifying and I would never have even thought that would be a thing.
>Compiler writers and tool authors are perfectly comfortable working with binary file formats. There's nothing more inherently future-compatible about JSON than a forward-compatible binary format like flatbuffers. Having to escape and then unescape naughty bytes is a huge downside for text-based formats that are hardly ever read by humans.
Well, the problem isn't binary or non-binary, the problem is that these formats like ELF are, apparently, really annoying to deal with, have weird limitations, and are difficult to extend. The reason I thought of JSON in particular is because it doesn't really need to be extended to encode anything (unless you include escaping or base64 encoding binary data you want to put inside of a JSON document as "extending" it). You can encode all of the fields in ELF (or any other format) inside of JSON, while it doesn't make sense to consider the converse because ELF has fixed fields with fixed meanings.
That's the problem with these bespoke binary formats like ELF - they're not designed to encode arbitrary schemas of data, they're designed for very specific tasks and then when they get used outside of their intended environment, we get problems like have been described in this thread. Nobody has ever had these problems with a JSON document - maybe with something that consumed one, but the file format itself simply does not have the same kind of limitations like ELF does. It has different limitations, but they're not of a fundamental and semantic nature like they are in a more rigid format.
You're right that it would be a problem to have to escape/unescape every section every time you wanted to run something because that's very slow, but I think that's basically the only problem that these bespoke binary formats solve. If that's the case, I wonder why something like Matroska wouldn't work for binaries? My understanding is that it's basically binary XML and allows for basically a completely arbitrary dictionary structure. It doesn't have nice tooling like JSON or XML do, but there's no weird restrictions on things like field length that I'm aware of. I guess it doesn't exactly have any "momentum", though, but maybe the NixOS people will get sick enough of ELF to consider such a drastic solution :P
Re: PatchELF: Simple utility for modifying existing ELF executables and libraries
#37BTW, is there any recommended open source dynamic binary patching tool?
Have been using Intel Pin, but fine-grained patches would incur noticeable performance regression permanently.
Thanks!
Re: PatchELF: Simple utility for modifying existing ELF executables and libraries
#38Patchelf is almost always broken, and when they merge fixes they don't release a new version right away. Between 2014-2019 running patchelf+strip would corrupt binaries, after the fix running patchelf twice on the same binary would corrupt it. And now patchelf doesn't work on all binaries, there's a bugfix merged https://github.com/NixOS/patchelf/pull/230 merged Nov 19, 2020, but no release since then. Also patchelf…
I'm not really sure why it has to be that hard, like, why don't we just use base64-encoded JSON or something?
> why don't we just use base64-encoded JSON or something?
You should try it and let us know how that works.
Re: PatchELF: Simple utility for modifying existing ELF executables and libraries
#39Earlier quoted context omitted.
> why don't we just use base64-encoded JSON or something? Base64 would be counterproductive, increasing both space and parsing time... presumably out of fear that JSON would contain a naughty byte for a greenfield file format. It would be much better to just design tho format to not have any naughty bytes. Parsing time for exacutables and libraries is definitely on the critical startup path. You really want a length-…
That's a good point about the critical path - I was thinking it would be a bit bigger and slower since you'd have to decode it, but I hadn't realized what an impact that would probably have. No bit-perfect round trips is also absolutely horrifying and I would never have even thought that would be a thing. >Compiler writers and tool authors are perfectly comfortable working with binary file formats. There's nothing mo…
> That's the problem with these bespoke binary formats like ELF - they're not designed to encode arbitrary schemas of data, they're designed for very specific tasks and then when they get used outside of their intended environment, we get problems like have been described in this thread. Nobody has ever had these problems with a JSON document - maybe with something that consumed one, but the file format itself simply does not have the same kind of limitations like ELF does. It has different limitations, but they're not of a fundamental and semantic nature like they are in a more rigid format
This is nothing specific to binary formats, but specific to insufficiently extensible formats. Note that I specifically mentioned flatbuffers, which provide for extensibility while keeping parsing latency low.
Also, ELF was designed to be extensible by adding new sections. You could totally add functionality by adding a new section holding JSON data.
Don't confuse JSON with extensibility. I've seen plenty of headaches with poorly thought out JSON schemas where forward compatibility wasn't sufficiently well thought out. There are also tons of elegantly extensible binary formats. ELF is just old; much older than JSON. A new binary format would probably be more elegantly extensible.
Re: PatchELF: Simple utility for modifying existing ELF executables and libraries
#40BTW, is there any recommended open source dynamic binary patching tool? Have been using Intel Pin, but fine-grained patches would incur noticeable performance regression permanently. Thanks!
I don't know exactly what Pin does, but probably dyninst, which also supports Arm8 and POWER. It won't avoid slowdown, though.