Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dynamic linking

drewdevault.com

Post reply on HN
Print view View original

101–110 of 249 posts

Re: Dynamic linking

#101

Post by danShumway »

dan-robertson wrote: post #92
earthboundkid wrote: post #40

Say we have program X with dependency Y. X+Y is either dynamic or static. X can either have responsive maintainers or unresponsive maintainers. Y can either change to fix a bug or change to add a bug. (With Heartbleed, I remember our server was fine because we were on some ancient version of OpenSSL.) Here are the scenarios: - dynamic responsive remove bug: Positive/neutral. Team X would have done it anyway. - dynami…

If you’re a Linux distribution and you have the source code to the software you distribute then the responsiveness of the maintainers doesn’t matter so much: any change to a library which is compatible with the previous version (in the sense that you could dynamically link to the new version instead of the old version) is going to be compatible under static linking too.

That's a really good point that I hadn't considered. All of the software I'm pulling from Pacman is being compiled from source, the authors aren't uploading binaries.

So the difference in effort between changing the static library and changing the dynamic library is... maybe not nothing, but not nearly as high as I was assuming.

Re: Dynamic linking

#102

Post by mforney »

cryptonector wrote: post #23

By far the most important reason for dynamic linking for C is semantics: static linking semantics are stuck in 1978 and suck (more on that below), while dynamic linking semantics make C a much better language. In particular, static linking for C has two serious problems: 1. symbol collisions -> accidental interposition (and crashes); 2. you have to flatten the dependency tree into a topological sort at the final link…

You bring up some good points here. Here are some of my experiences with these problems when working on oasis (my static linux distro).

> 1. symbol collisions -> accidental interposition (and crashes);

I've encountered symbol collisions only twice, but both resulted in linker errors due to multiple function definitions. I'm not sure how this could happen accidentally. Maybe you are referring to variables in the common section getting merged into a single symbol? Recent gcc enables -fno-common by default, so those will be caught by the linker as well.

> 2. you have to flatten the dependency tree into a topological sort at the final link-edit.

Yes, this is pretty annoying. pkg-config can solve this to some degree with its --static option, but that only works if your libraries supply a .pc file (this is often the case, though).

I think libtool also can handle transitive dependencies of static libraries, but it tries hard to intercept the -static option before it reaches the compiler so it links everything but libc statically. You can trick it by passing `-static --static`.

For oasis, I use a separate approach to linking involving RSP files (i.e. linking with @libfoo.rsp), which really are just lists of other libraries they depend on.

> Besides fixing these issues, the C dynamic linking universe also enables things like: > - run-time code injection via LD_PRELOAD and intended interposition

Yes, this can be a problem. I wanted to do this recently to test out the new malloc being developed for musl libc, but ended up having to manually integrate it into the musl sources instead of just using LD_PRELOAD.

> - run-time code loading/injection via dlopen(3)

In particular, this is a big problem for scripting languages that want to use modules written in compiled languages, as well as OpenGL which uses dlopen to load a vendor-specific driver.

> Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.

But doesn't the dynamic linker still have to do extra work to resolve the relocations in the executable, even when the dependency libraries are already loaded?

Re: Dynamic linking

#103

Post by jart »

cryptonector wrote: post #88
pengaru wrote: post #69

Static linking has been such a nuisance for the libSDL folks that they implemented dynamic loading of itself [0], controlled via an environment variable, as an escape hatch from executables w/libSDL linked statically. It's understandable that games, especially proprietary ones, distribute statically-linked binaries ensuring any third-party dependencies will be present and be a compatible version. But the value of tha…

> [...] Having said that, I do wish the average linux distro still statically linked everything in /bin and /sbin. It was nice to still be able to administrate the system even when the dynamic libraries were hosed. [...] This argument came up back when Solaris 10 was in development and the project to get rid of static link archives for system libraries came up (search for Solaris "unified process model"). The disposi…

Sometimes solutions give rise to new categories of issues, and it's difficult to connect the dots to the root cause. If you believe dynamic linking hasn't introduced an even broader array of difficulties for C coders needing to support both, then please read Ulrich Drepper's DSO tutorial which gives a pretty good rundown: https://software.intel.com/sites/default/files/m/a/1/e/dsoho... If I remember correctly, it was largely SCO Group that pushed UNIX vendors back in the 1990's to switch to a WIN32 linking model. I didn't find their arguments that compelling, to be honest, due to not citing alternatives considered.

Re: Dynamic linking

#104

Post by danShumway »

riquito wrote: post #50
eeereerews wrote: post #37

Earlier quoted context omitted.

What do you mean by "source-based distribution"?

One without packages shipped as prebuilt binaries: Gentoo [1] would be a good example. You compile every binary yourself (sometime you can use prebuilt binaries for the biggest software, if you are so inclined and the distro allows it). [1] https://www.gentoo.org/

My Arch install doesn't compile anything locally unless I'm installing from AUR.

Maybe it's doing it behind the scenes or something, but I'm doubtful, because otherwise I think my upgrades would take a lot longer.

Re: Dynamic linking

#105

Post by MaulingMonkey »

bjourne wrote: post #62
loeg wrote: post #25

Earlier quoted context omitted.

> Put the cold hard numbers right in front of someone's face and still the cargo cult wins out. Come on. You did not actually measure the figure GP mentioned and which you are disputing. Your methodology and assumption — that 4% external symbol use translates into 4% size used — is a plausible guess, but you haven't supported it with data. Even if you had measured the figure you're accusing GP of ignoring, the tone o…

> There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary. But most code isn't performance critical. Thus trying to align functions to page boundaries is just wasting memory. Even in performance critical code, alignin…

Not only is it wasting memory, but I wouldn't be convinced it'd not be outright hurting performance by increasing cache line aliasing, increasing TLB overhead and misses/invalidations! Avoiding page alignment can be a performance gain! https://pvk.ca/Blog/2012/07/30/binary-search-is-a-pathologic...

Like, sure, mapping your executable's code section in at a page boundary is probably fine, but I think trying to align individual functions to page boundaries would be a counterproductive mistake as a general strategy.

Re: Dynamic linking

#106

Post by saagarjha »

dan-robertson wrote: post #95

I’m curious about the supposed memory advantages of dynamic linking: on average how many different executables share each page of memory? What about when memory is in high demand? How high does that average become? What is the probability that a page of a shared library is already in memory (cache or otherwise) when it needs to be loaded, and in particular the probability that it is there because another program load…

I think this depends heavily on your platform and how often libraries are reused. On macOS, for example, most libraries aside from libc are loaded R/O or CoW quite literally hundreds of times, because every app shares AppKit and WebKit and Security and the dozens of other platform frameworks (and their private dependencies!) that are basically "free" to use and ship with the system and so have very high adoption. On more "coherent" Linux distributions I'm sure things like GTK, glib, OpenSSL, zlib are used by a lot of things too. Sure, there's going to be a lot of one-off dynamic libraries too, but there's a lot of duplication with the popular dependencies and then a long tail.

Re: Dynamic linking

#107

Post by binarycrusader »

jeffbee wrote: post #79
binarycrusader wrote: post #65

Earlier quoted context omitted.

It is a performance feature if you are memory constrained. Shared libraries are “shared” for a reason. On a server with high multi-tenancy the savings can be significant.

Not really. Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems. As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything.

Not really.

Yes, really.

Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems.

Definitely not true. Every bit of memory that's not available that could be shared memory instead is a reduction in memory available for filesystem caches, etc.

As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything.

Cloud vendor computing models are not generally the computing model of the rest of the world. Comparison to their environment is not relevant to the general populace.

I worked for a "big iron" OEM vendor until late 2017 and the savings were definitely still significant then both for their customers and the vendor themselves.

There are numerous benefits to shared linking, that doesn't mean it's always the appropriate solution, but is not correct to claim that there are no performance benefits.

Especially on more memory-constrained consumer devices, the shared memory benefits of dynamic linking are still significantly beneficial.

Re: Dynamic linking

#108

Post by cryptonector »

sigjuice wrote: post #98
cryptonector wrote: post #97

Earlier quoted context omitted.

Specifically for C, static linking is a trap due to its poor semantics. Don't do it. Or fix the C link-editors, then static linking C will be fine.

What would be examples of poor static linking semantics? Thanks!

https://news.ycombinator.com/item?id=23656173

Re: Dynamic linking

#109

Post by cryptonector »

mforney wrote: post #102
cryptonector wrote: post #23

By far the most important reason for dynamic linking for C is semantics: static linking semantics are stuck in 1978 and suck (more on that below), while dynamic linking semantics make C a much better language. In particular, static linking for C has two serious problems: 1. symbol collisions -> accidental interposition (and crashes); 2. you have to flatten the dependency tree into a topological sort at the final link…

You bring up some good points here. Here are some of my experiences with these problems when working on oasis (my static linux distro). > 1. symbol collisions -> accidental interposition (and crashes); I've encountered symbol collisions only twice, but both resulted in linker errors due to multiple function definitions. I'm not sure how this could happen accidentally. Maybe you are referring to variables in the commo…

> > 1. symbol collisions -> accidental interposition (and crashes);

> I've encountered symbol collisions only twice, but both resulted in linker errors due to multiple function definitions. I'm not sure how this could happen accidentally. Maybe you are referring to variables in the common section getting merged into a single symbol? Recent gcc enables -fno-common by default, so those will be caught by the linker as well.

No, this comes up all the time. Try building an all-in-one busybox-style program, and you'll quickly run into conflicts.

If static link archives had all the metadata that ELF files have, then the link-editor could resolve conflicts correctly. That is the correct fix, but no one is putting effort into it. The static linkers haven't changed much since symbol length limits were raised from 14 bytes!

> > 2. you have to flatten the dependency tree into a topological sort at the final link-edit.

> Yes, this is pretty annoying. pkg-config can solve this to some degree with its --static option, but that only works if your libraries supply a .pc file (this is often the case, though).

pkg-config alleviates the problem, but it's not enough. Among other things building a build system that can build with both, static and dynamic linking is a real pain. But more importantly, this flattening of dependency trees loses information and makes it difficult for link-editors to resolve symbol conflicts correctly (see above).

> > Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library.

> But doesn't the dynamic linker still have to do extra work to resolve the relocations in the executable, even when the dependency libraries are already loaded?

It's still faster than I/O. (Or at least it was back in the days of hard drives. But I think it's still true even in the days of SSDs.)

Re: Dynamic linking

#110

Post by btilly »

eikenberry wrote: post #63
BeetleB wrote: post #46

Earlier quoted context omitted.

> First, this analysis was done on Arch Linux, a source-based distribution. Since you know at compile time what your environment is, I would expect the benefits to be smaller. And of course, this means you're willing to do a lot of recompiles. I run Gentoo as my Linux box. It takes a long time to compile stuff as it is (and I have a 22 core machine). Being forced to compile a lot more because everything is linked sta…

If compilers weren't so pathetically slow this wouldn't be that much of an issue. If this became widespread it might have the positive impact on projects like LLVM and get them to pay some attention to compiling time, not just optimization.

The problem with compile times is mostly not the compiler's fault. It is with the project setup.

See http://www.real-linux.org.uk/recursivemake.pdf for a classic explanation.

Post reply on HN