Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is a binary executable organized? Let's explore it (2014)

jvns.ca

Post reply on HN
Print view View original

81–90 of 93 posts

Re: How is a binary executable organized? Let's explore it (2014)

#81

Post by vlovich123 »

secondcoming wrote: post #45
kccqzy wrote: post #17

Earlier quoted context omitted.

The style guide at both my previous and current employers explicitly forbids having global non-0 statics for this exact reason: code that runs before main() is very unusual. Many assumptions do not hold. A far better way is to use function-local statics. A static variable inside a function is initialized when execution reaches that point when the function is being called. Furthermore, such initialization is thread sa…

The issue with Meyers Singletons is that every time they're accessed a flag must be checked first. This is bad in hop loops.

It’s not as bad as you might think because the CPU should speculate through it pretty easily.

Re: How is a binary executable organized? Let's explore it (2014)

#82

Post by vlovich123 »

matheusmoreira wrote: post #61
vlovich123 wrote: post #8

Earlier quoted context omitted.

> Things that I don’t understand very well, including calling main. So I won’t explain them. It depends on the language runtime, but a common task will be initializing global non-0 statics. For languages like Rust/C/C++ you can also inject variables to be initialized via linker flags. Before start if the program is dynamically linked then I believe the linker runtime is run to resolve the links and then transfer cont…

> initializing global non-0 statics Does that mean this doesn't work in a freestanding environment? Yet another reason to assiduously avoid global variables. I suppose that's why I never ran into this issue. > Basically hacks on hacks on hacks And then there's this insanity right here: https://blogs.oracle.com/solaris/post/init-and-fini-processi... I like to imagine that I won't have to actually implement this when I…

> Does that mean this doesn't work in a freestanding environment? Yet another reason to assiduously avoid global variables. I suppose that's why I never ran into this issue.

Why do you say that? A freestanding environment will still enter through _start and execute the compiler generated code to initialize globals before main. What I can’t recall is how compile time non-0 values are initialized - I think it could be part of the bss and initialized by the loader instead (but freestanding environments would implement that too as part of being a target for a language) but both them and runtime initialized globals initialized between _start and main would work.

Basically freestanding targets might not give you access to runtime APIs (eg POSIX) but the language is still the language and all features defined as language features should work and it’s the responsibility of the compiler and target environment to provide that guarantee.

Re: How is a binary executable organized? Let's explore it (2014)

#83

Post by rep_lodsb »

vitiral wrote: post #49
actionfromafar wrote: post #12

Not a criticism, not even a nit-pick, but a reflection "(binaries are kind of the definition of platform-specific, so this is all platform-specific) (this is true!) When "Actually Portable Executable" took the (geek) proved that the same binary could run on a bunch of platforms, that was a surreal moment I still haven't mentally recovered from. Here we spent decades trying to solve the cross-platform problem, in so m…

I may have misunderstood, but I'm pretty sure APE is not a "binary format" per se. It is a script that can be executed on any system. That script can then LOAD a binary. IIRC the original needed to decode it from base64 before it could be loaded. So... it's an executable binary loader

It's a script that starts with an EXE header ("MZ"). Having both EXE and ELF headers at the same location is obviously impossible, since they start with different bytes.

A possible, although very limited way to have an actual binary program execute on different platforms would be to create a DOS .COM file (which has no header, just the raw machine code) with a valid ELF header. It would then also work on 32-bit Windows via its built-in DOS emulator, and presumably on 64-bit Windows with WSL2.

The start bytes for a 32-bit ELF header decode as 16-bit x86 into:

    7F 45    JG    +45h
    4C       DEC   SP
    46       INC   SI
    01 01    ADD   [BX+DI],AX
    01 xx    ADD   ...
The first instruction is a jump past the end of the ELF header, unfortunately it's conditional. But we have 9 reserved bytes to continue this code, which is enough to undo the effects of the DEC and ADD instructions and then jump to the same address. I've written a 138 byte "Hello world" that works on Linux, DOS and also CP/M-80 that way.

It's possible to have the code that executes under Linux be a small (less than 2K bytes) loader program that creates 16-bit code and data segments and installs a handler for SIGSEGV. It can then jump into the same code that would run under 16-bit DOS, trapping every INT 21h and translating the most important syscalls into their Linux equivalents, kind of like a minimal version of Wine.

I have a proof of concept for that, it only handles the "read", "write" and "exit" syscalls, which is enough to write something like rot13 or hexdump. With a lot more work, it could be possible to produce really non-trivial software that runs in such a restricted environment...

Re: How is a binary executable organized? Let's explore it (2014)

#84

Post by [deleted] »

matheusmoreira wrote: post #75
kazinator wrote: post #71

Earlier quoted context omitted.

These tricks can be easily done simply and portably, without caring about the underlying executable format. One trick is that you can reserve some global array in the executable, which is prefixed by a byte sequence that doesn't occur anywhere. A small utility can find that byte sequence and write custom data after it to create a customized executable. I think that, also, many executable formats don't mind if somethi…

> reserve some global array in the executable This is neat but has a limitation in that it cannot be expanded after the program is compiled and linked. Resizing the array would invalidate all pointers that follow as well as render incorrect any code that takes its size. This can be solved with a layer of indirection: just append the data to the executable and write its size and file offset in the array. That way the…

[deleted]

Re: How is a binary executable organized? Let's explore it (2014)

#85

Post by kazinator »

matheusmoreira wrote: post #75
kazinator wrote: post #71

Earlier quoted context omitted.

These tricks can be easily done simply and portably, without caring about the underlying executable format. One trick is that you can reserve some global array in the executable, which is prefixed by a byte sequence that doesn't occur anywhere. A small utility can find that byte sequence and write custom data after it to create a customized executable. I think that, also, many executable formats don't mind if somethi…

> reserve some global array in the executable This is neat but has a limitation in that it cannot be expanded after the program is compiled and linked. Resizing the array would invalidate all pointers that follow as well as render incorrect any code that takes its size. This can be solved with a layer of indirection: just append the data to the executable and write its size and file offset in the array. That way the…

> How does an executable open itself? That's where portability goes out the window.

It's a way easier nut to crack than parsing executable formats. Various OSs have a method for this:

Linux: /proc/self/exe

Windows: GetModuleFileNameW(NULL, buf, MAX_PATH)

MacOS: _NSGetExecutablePath(buf, &size)

Solaris: getexecname()

BSDs: ...?

These are just "ask and you shall receive" APIs; nothing to analyze.

You may want to solve that problem anyway for other reasons: for instance, if you want your programming language to have a relocatable installation which finds related files using its own path.

On a system where you cannot solve this problem at all, you can use trick 1. Have an array somewhere in the program where you write the installation path. Your installer has to do this, and if the program is moved after installation, some utility program has to fix that up. (I am not crazy about the idea because it changes the checksum of the file at the installation site. You can no longer use an off-the-shelf program like sha256sum to check the integrity of the executable against the provisioned materials upstream.)

Re: How is a binary executable organized? Let's explore it (2014)

#86

Post by akdas »

matheusmoreira wrote: post #64
akdas wrote: post #55

Earlier quoted context omitted.

I wrote it all by hand :) Lately, I've been using Svelte for interactive visualizations (see my post on using a tool called Astro with Svelte: https://avikdas.com/2023/12/30/interactive-demos-using-astro... ). But this one is all hand-written JS!

You and your web site are huge inspirations for me.

Thank you! That means a lot to me

Re: How is a binary executable organized? Let's explore it (2014)

#87

Post by akdas »

raarts wrote: post #73
akdas wrote: post #22

Earlier quoted context omitted.

Writing an ELF file by hand is something I did recently: https://github.com/avik-das/garlic/blob/master/recursive/elf... To explain the format to myself and others, I also created an interactive visualization for the bytes in the file. It helps me to click on a byte, see an explanation for it and see related bytes elsewhere in the file highlighted. https://scratchpad.avikdas.com/elf-explanation/elf-explanati...

Wow. Cool! I’m a CS teacher and definitely going to use this. Thanks for your work! (Anyone aware of a Windows or Mac equivalent?)

Thank you! My passion is teaching, and I created interactive visualizations for a CS theory class I taught: https://cstheory.avikdas.com/

If you're interested in collaborating for your classes, reach out to avik at avikdas dot com

Re: How is a binary executable organized? Let's explore it (2014)

#88

Post by matheusmoreira »

vlovich123 wrote: post #82
matheusmoreira wrote: post #61

Earlier quoted context omitted.

> initializing global non-0 statics Does that mean this doesn't work in a freestanding environment? Yet another reason to assiduously avoid global variables. I suppose that's why I never ran into this issue. > Basically hacks on hacks on hacks And then there's this insanity right here: https://blogs.oracle.com/solaris/post/init-and-fini-processi... I like to imagine that I won't have to actually implement this when I…

> Does that mean this doesn't work in a freestanding environment? Yet another reason to assiduously avoid global variables. I suppose that's why I never ran into this issue. Why do you say that? A freestanding environment will still enter through _start and execute the compiler generated code to initialize globals before main. What I can’t recall is how compile time non-0 values are initialized - I think it could be…

> Why do you say that?

Your post gave me the impression the compiler generates initializers for static globals and calls them during program initialization. Is this not correct?

> A freestanding environment will still enter through _start and execute the compiler generated code to initialize globals before main.

Freestanding C generally implies not linking in any standard library code though. It doesn't make any sense to choose the freestanding C dialect only to link in the standard hosted C library.

The _start symbol is provided by the C library's so called standard system startup files. The standard _start calls all the initialization functions that the C program expects. If you override it with your own _start, then none of these functions will be called. I thought you meant the static data initialization was handled in one of these functions.

Re: How is a binary executable organized? Let's explore it (2014)

#89

Post by versteegen »

zerotolerance wrote: post #38
actionfromafar wrote: post #11

Earlier quoted context omitted.

Can you elaborate?

The other replies are pretty good. You can find all sorts of goodies in string data inside a binary: hostnames, URL fragments, error messages or templates, credentials. Pretty much any string constants that a program might use.

It's even quite interesting to run strings on programs I wrote and I do it regularly. For example it can be faster to get the version string from an .exe using strings if you know what to grep for than to run "wine program.exe --version".

Re: How is a binary executable organized? Let's explore it (2014)

#90

Post by vitiral »

rep_lodsb wrote: post #83
vitiral wrote: post #49

Earlier quoted context omitted.

I may have misunderstood, but I'm pretty sure APE is not a "binary format" per se. It is a script that can be executed on any system. That script can then LOAD a binary. IIRC the original needed to decode it from base64 before it could be loaded. So... it's an executable binary loader

It's a script that starts with an EXE header ("MZ"). Having both EXE and ELF headers at the same location is obviously impossible, since they start with different bytes. A possible, although very limited way to have an actual binary program execute on different platforms would be to create a DOS .COM file (which has no header, just the raw machine code) with a valid ELF header. It would then also work on 32-bit Windo…

Don't tell jart this or she'll probably add it to cosmopolitan
Post reply on HN