Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ELF files on Linux

linux-audit.com

Post reply on HN
Print view View original

11–20 of 27 posts

Re: ELF files on Linux

#11

Post by jcranmer »

AceJohnny2 wrote: post #9
haberman wrote: post #3

I wrote Bloaty ( https://github.com/google/bloaty ) which involved writing a totally custom ELF file parser. Here are some epiphanies I had about ELF while writing it. ELF (and Mach-O, PE, etc) are designed to optimize the creation of a process image. The runtime loader mainly just has to mmap() a bunch of file ranges into memory with various permissions. This is quite different than loading a .jar file, .pyc, etc. w…

I've been wondering about qualitative differences between Mach-O and ELF, after hearing a Mach developer trash the ELF format, but I don't know enough about either to comment. Do you have any insight?

I only know the ELF format in detail, but I think the major complaint is that ELF uses a flat namespace for symbols whereas Mach-O has a two-level namespace. Furthermore, ELF lets you preload dynamic libraries such that you can override calls even to symbols provided in the same shared object.

Re: ELF files on Linux

#12

Post by IshKebab »

haberman wrote: post #3

I wrote Bloaty ( https://github.com/google/bloaty ) which involved writing a totally custom ELF file parser. Here are some epiphanies I had about ELF while writing it. ELF (and Mach-O, PE, etc) are designed to optimize the creation of a process image. The runtime loader mainly just has to mmap() a bunch of file ranges into memory with various permissions. This is quite different than loading a .jar file, .pyc, etc. w…

Sections and segments are so badly named. They may as well have called them data and different-data.

I guess it is too late to rename them though.

Re: ELF files on Linux

#13

Post by jcranmer »

vectorEQ wrote: post #2

nice article but wish people would elaborate more on relocations instead of always skipping that. it's a very important part of understanding how ELF works when it's executed.

Relocations boil down to this:

A relocation entry contains a location of the patch, the symbol to use in relocation, an optional addend, and a type. The type tells you how to compute the relocation and is completely defined by the processor-specific ABI.

The simple relocations boil down to "add the addend to the address of the symbol, subtract the address of the relocation, and store it as signed N-bit number". There are more complex relocations that involve things such as symbol sizes, TLS relocations, or the GOT and the PLT.

Re: ELF files on Linux

#14

Post by haberman »

monocasa wrote: post #10
AceJohnny2 wrote: post #9

Earlier quoted context omitted.

I've been wondering about qualitative differences between Mach-O and ELF, after hearing a Mach developer trash the ELF format, but I don't know enough about either to comment. Do you have any insight?

Not haberman, but I've written loaders for both Mach-O and ELF, and really prefer ELF. ELF is mainly structured like descriptive tables of how the relevant pieces look in memory; Mach-O is more structured like a script of commands that you run to load the binary. There's a couple places where the model breaks down for ELF, DWARF and GNU_STACK both feel more Mach-O, but if you're playing with binaries for non standard…

Interesting, what was your job that required writing both a Mach-O and ELF loader?

Re: ELF files on Linux

#15

Post by mzs »

zimbatm wrote: post #4

If you ever need to tweak or inspect an existing binary, https://github.com/NixOS/patchelf is great.

/usr/ccs/bin/dump on solaris is great for shell scripts too.

Re: ELF files on Linux

#16

Post by haberman »

AceJohnny2 wrote: post #9
haberman wrote: post #3

I wrote Bloaty ( https://github.com/google/bloaty ) which involved writing a totally custom ELF file parser. Here are some epiphanies I had about ELF while writing it. ELF (and Mach-O, PE, etc) are designed to optimize the creation of a process image. The runtime loader mainly just has to mmap() a bunch of file ranges into memory with various permissions. This is quite different than loading a .jar file, .pyc, etc. w…

I've been wondering about qualitative differences between Mach-O and ELF, after hearing a Mach developer trash the ELF format, but I don't know enough about either to comment. Do you have any insight?

That is a deep and interesting question. I'm not sure I can give a great answer, but here are a few thoughts.

If we look at the file format itself (separate from the features/semantics of the linker and loader), I think ELF is simpler and more orthogonal. You can iterate over the section/segment tables of an ELF file without knowing anything about what each section/segment means. ELF nicely decouples the high-level "container" aspect of the file format from the lower-level semantics of how you interpret each section/segment in the linker and loader.

Mach on the other hand couples these two concepts together. The top-level table is an array of "load commands", each with its own type, but you can't even parse a load command until you know what type it is. Unlike ELF, the entries of this table do not have a generic format or even a consistent size. If you haven't written code to specifically recognize a given command type, all you can do as fallback behavior is skip it. To me ELF feels like a refactoring of Mach to make it a little more general and layered.

If we consider the actual semantics and features of the file formats, there are pros and cons to both. Mach-O has built-in support for fat (multi-architecture) binaries, which is kind of nifty, though I've never actually used it myself. Mach-O distinguishes between "dylib" and "bundle" for shared libraries -- for the life of me I can never remember the difference between these two -- whereas ELF just has one type of shared library. (https://docstore.mik.ua/orelly/unix3/mac/ch05_03.htm). The distinction seems to add complexity and I'm not sure I understand the benefit. Mach-O has two-level namespaces (dynamic symbols are resolved by both name and the library they come from) -- colliding symbols aren't generally a problem I've seen with ELF, but maybe it's useful in some cases. ELF makes symbol interpositioning easy with LD_PRELOAD, though Mach-O seems to have its own version of this that I've never tried: https://stackoverflow.com/questions/12609728/changing-functi.... Overall I prefer ELF.

Re: ELF files on Linux

#17

Post by AceJohnny2 »

haberman wrote: post #16
AceJohnny2 wrote: post #9

Earlier quoted context omitted.

I've been wondering about qualitative differences between Mach-O and ELF, after hearing a Mach developer trash the ELF format, but I don't know enough about either to comment. Do you have any insight?

That is a deep and interesting question. I'm not sure I can give a great answer, but here are a few thoughts. If we look at the file format itself (separate from the features/semantics of the linker and loader), I think ELF is simpler and more orthogonal. You can iterate over the section/segment tables of an ELF file without knowing anything about what each section/segment means. ELF nicely decouples the high-level "…

Thanks!

Regarding multi-arch support, Ryan C. Gordon (Linux game porter extraordinaire, icculus.org) had proposed FatELF [1] back in 2009 (LWN coverage [2]). It seemed simple enough to implement, but never really picked up steam (IMHO for reasons that speak of the culture of the Linux ecosystem).

[1] http://icculus.org/fatelf/

[2] https://lwn.net/Articles/359070/

Re: ELF files on Linux

#18

Post by kccqzy »

One thing that's on my mind but haven't been able to spend time investigating is the fact that on my machine (Ubuntu 19.04), almost all distribution-installed executables are not ELF executables per se, but ELF shared objects. Running `file /bin/ls` shows that it's an ELF 64-bit LSB shared object. Running `readelf -h /bin/ls` also says that the type is DYN. Is the executable type basically deprecated now?

Re: ELF files on Linux

#19

Post by CalChris »

haberman wrote: post #3

I wrote Bloaty ( https://github.com/google/bloaty ) which involved writing a totally custom ELF file parser. Here are some epiphanies I had about ELF while writing it. ELF (and Mach-O, PE, etc) are designed to optimize the creation of a process image. The runtime loader mainly just has to mmap() a bunch of file ranges into memory with various permissions. This is quite different than loading a .jar file, .pyc, etc. w…

I've worked with both ELF and Mach-O. They're really very similar from a loading standpoint although Mach-O is a little more wordy and precise. One difference is that Mach-O allows you to specify the initial stack (x86_thread_state64_t.rsp) and ELF (for some reason which I've never understood) doesn't; but maybe there are ELF extensions in the embedded space that I don't know about.

With most Mach-O files, rsp is set to 0 and a default is used. The code is in bsd_i386.c/thread_userstack():

  if (state25->rsp) {
    *user_stack = state25->rsp;
    if (customstack)
      *customstack = 1;
  } else {
    *user_stack = VM_USRSTACK64;
    if (customstack)
      *customstack = 0;
  }
This can be set on the OSX ld command line with -stack_addr. OSX uses an obscenely old fork of GNU ld for Mach-O. No linker scripts.

Things like zero pages are explicit (but required!) in OSX while they are implicit in ELF+Linux. Also llvm lld's ELF support is first class whereas its Mach-O support is less fleshed out. With Mach-O it helps to read the Darwin source files to figure out what it does with a file. Same to a lesser degree with Linux.

Re: ELF files on Linux

#20

Post by glandium »

haberman wrote: post #16
AceJohnny2 wrote: post #9

Earlier quoted context omitted.

I've been wondering about qualitative differences between Mach-O and ELF, after hearing a Mach developer trash the ELF format, but I don't know enough about either to comment. Do you have any insight?

That is a deep and interesting question. I'm not sure I can give a great answer, but here are a few thoughts. If we look at the file format itself (separate from the features/semantics of the linker and loader), I think ELF is simpler and more orthogonal. You can iterate over the section/segment tables of an ELF file without knowing anything about what each section/segment means. ELF nicely decouples the high-level "…

Mach-O also has a more compact bytecode-like representation for relocations, while ELF just wastes tons of space. See https://glandium.org/blog/?p=1177
Post reply on HN