Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is a binary executable organized? Let's explore it (2014)

jvns.ca

Post reply on HN
Print view View original

71–80 of 93 posts

Re: How is a binary executable organized? Let's explore it (2014)

#71

Post by kazinator »

matheusmoreira wrote: post #65
jstrieb wrote: post #2

As I've said in other threads https://news.ycombinator.com/item?id=38847750#38862450 , I highly recommend writing an ELF by hand at least once. It's a great exercise to understand the basic parts of an executable. It's also helpful if you want to go the opposite direction of this article - bottom-up instead of top-down. Lots of other great discussion in various threads on that other HN post.

Modifying existing ELFs can also be extremely educational and fun. It's a bit frustrating at first because it's more or less impossible to debug this stuff when it doesn't work but when things finally start working it's awesome. Turns out it's possible to patch ELFs in all sorts of interesting ways. With the auxiliary vector you can even have introspection at runtime: Linux gives us the address of the program header…

These tricks can be easily done simply and portably, without caring about the underlying executable format.

One trick is that you can reserve some global array in the executable, which is prefixed by a byte sequence that doesn't occur anywhere. A small utility can find that byte sequence and write custom data after it to create a customized executable.

I think that, also, many executable formats don't mind if something is appended to the executable. If the executable somehow knows its original size (you can write that size somewhere using the previous trick: no grotty executable format parsing required), it can open itself, seek to that offset and read the data.

I think this might be how CLISP creates an .exe file on Windows; I think it takes the base clisp.exe and combines it with the lispinit.mem image into one file.

Re: How is a binary executable organized? Let's explore it (2014)

#72

Post by cylinder714 »

jstrieb wrote: post #2

As I've said in other threads https://news.ycombinator.com/item?id=38847750#38862450 , I highly recommend writing an ELF by hand at least once. It's a great exercise to understand the basic parts of an executable. It's also helpful if you want to go the opposite direction of this article - bottom-up instead of top-down. Lots of other great discussion in various threads on that other HN post.

Chris Wellons' "A Magnetized Needle and a Steady Hand," a piece on building an ELF executable from scratch:

https://nullprogram.com/blog/2016/11/17/

Re: How is a binary executable organized? Let's explore it (2014)

#73

Post by raarts »

akdas wrote: post #22
jstrieb wrote: post #2

As I've said in other threads https://news.ycombinator.com/item?id=38847750#38862450 , I highly recommend writing an ELF by hand at least once. It's a great exercise to understand the basic parts of an executable. It's also helpful if you want to go the opposite direction of this article - bottom-up instead of top-down. Lots of other great discussion in various threads on that other HN post.

Writing an ELF file by hand is something I did recently: https://github.com/avik-das/garlic/blob/master/recursive/elf... To explain the format to myself and others, I also created an interactive visualization for the bytes in the file. It helps me to click on a byte, see an explanation for it and see related bytes elsewhere in the file highlighted. https://scratchpad.avikdas.com/elf-explanation/elf-explanati...

Wow. Cool! I’m a CS teacher and definitely going to use this. Thanks for your work! (Anyone aware of a Windows or Mac equivalent?)

Re: How is a binary executable organized? Let's explore it (2014)

#74

Post by saagarjha »

csdvrx wrote: post #26
norir wrote: post #18

Earlier quoted context omitted.

I personally am not convinced that portable binaries are a net positive. I believe in the era of fast computers that source distribution and local compilation is superior to binary distribution. Unfortunately, much of the software we rely on is so large, and compilers so relatively slow, that binary distribution is something of a necessary evil. I'd rather see more effort towards simpler software components (that nat…

You can have both. APE are generally faster and smaller. Fat APEs (aarch64 + x86_64) are larger, but interesting in their own way.

Not in general.

Re: How is a binary executable organized? Let's explore it (2014)

#75

Post by matheusmoreira »

kazinator wrote: post #71
matheusmoreira wrote: post #65

Earlier quoted context omitted.

Modifying existing ELFs can also be extremely educational and fun. It's a bit frustrating at first because it's more or less impossible to debug this stuff when it doesn't work but when things finally start working it's awesome. Turns out it's possible to patch ELFs in all sorts of interesting ways. With the auxiliary vector you can even have introspection at runtime: Linux gives us the address of the program header…

These tricks can be easily done simply and portably, without caring about the underlying executable format. One trick is that you can reserve some global array in the executable, which is prefixed by a byte sequence that doesn't occur anywhere. A small utility can find that byte sequence and write custom data after it to create a customized executable. I think that, also, many executable formats don't mind if somethi…

> reserve some global array in the executable

This is neat but has a limitation in that it cannot be expanded after the program is compiled and linked. Resizing the array would invalidate all pointers that follow as well as render incorrect any code that takes its size.

This can be solved with a layer of indirection: just append the data to the executable and write its size and file offset in the array. That way the data block can be freely resized. That's the solution people told me to use and indeed the one that I usually see in existing repositories. The problem is you run into some additional complexity later which results in the loss of portability and thus the main reason to choose this method.

> it can open itself

That's the crux of the issue. How does an executable open itself? That's where portability goes out the window. I've seen source code that opens argv[0] which is under the control of the parent program and therefore unreliable. I've seen code which opens /proc/self/exe which is Linux specific. I've seen code that calls Win32 API functions to get the path to the executable. All this just so it can open and load into memory a file which the kernel has already loaded, just so it can read some additional data off of it.

My solution sidesteps that question entirely. It just adds a LOAD segment for the embedded data which instructs the kernel to map it in automatically before the program even runs. There's no need to open, seek or read anything, it will already be there by the time execution begins.

The auxiliary vector contains a pointer to the program's segments table so it can reach the data from there. Then it's just a matter of walking this table looking for the custom descriptor segment. It's all done in a structured way, using the standard magic number locations and ranges. There's no chance of a magic number being recognized by mistake.

The only possible portability issue is the availability of the AT_PHDR, AT_PHENT and AT_PHNUM entries in the auxiliary vector. I'm not sure if they're standard. I know Linux has them and it's all I personally care about but if these entries do turn out to be standard then I can confidently say that my method is portable to any ELF-based operating system.

Re: How is a binary executable organized? Let's explore it (2014)

#78

Post by versteegen »

kazinator wrote: post #71
matheusmoreira wrote: post #65

Earlier quoted context omitted.

Modifying existing ELFs can also be extremely educational and fun. It's a bit frustrating at first because it's more or less impossible to debug this stuff when it doesn't work but when things finally start working it's awesome. Turns out it's possible to patch ELFs in all sorts of interesting ways. With the auxiliary vector you can even have introspection at runtime: Linux gives us the address of the program header…

These tricks can be easily done simply and portably, without caring about the underlying executable format. One trick is that you can reserve some global array in the executable, which is prefixed by a byte sequence that doesn't occur anywhere. A small utility can find that byte sequence and write custom data after it to create a customized executable. I think that, also, many executable formats don't mind if somethi…

Put the offset at the end! Famously, the table of contents of a .zip file are the end rather than the beginning, which has many useful properties (such as being able to patch the contents by only appending to an existing file). And you can concatenate an executable and a .zip and get a file which is both.

Re: How is a binary executable organized? Let's explore it (2014)

#79

Post by matheusmoreira »

versteegen wrote: post #78
kazinator wrote: post #71

Earlier quoted context omitted.

These tricks can be easily done simply and portably, without caring about the underlying executable format. One trick is that you can reserve some global array in the executable, which is prefixed by a byte sequence that doesn't occur anywhere. A small utility can find that byte sequence and write custom data after it to create a customized executable. I think that, also, many executable formats don't mind if somethi…

Put the offset at the end! Famously, the table of contents of a .zip file are the end rather than the beginning, which has many useful properties (such as being able to patch the contents by only appending to an existing file). And you can concatenate an executable and a .zip and get a file which is both.

Yes. Cosmopolitan libc has support for exactly this. It contains a lot of platform-specific hacks in order to open the executable though. I went through the implementation.

I think the problem is this notion of a "memory image". It would be so much easier if the kernel just copied the entire file into memory and called it a day.

Re: How is a binary executable organized? Let's explore it (2014)

#80

Post by vlovich123 »

cesarb wrote: post #46
vlovich123 wrote: post #41

Earlier quoted context omitted.

I’m not aware of this problem in MSVC, clang and gcc and those are the most popular afaik.

> I’m not aware of this problem in MSVC The compiler I was thinking of was indeed MSVC. From a quick web search, it seems that more recent versions of MSVC have changed them to be thread safe by default, so if you can make sure that your code will never be compiled on older MSVC versions (and that nobody will ever use the compiler option which disables the thread-safe initialization), it might be fine to depend on it…

True as of msvc 2015 which is 8.5 years old at this point. I agree it’s weird that you can disable it but libraries retaining correctness in the face of random compiler flags is hard (eg ffast-math is a common one that can break your floating point library)
Post reply on HN