Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dynamic linking

drewdevault.com

Post reply on HN
Print view View original

151–160 of 249 posts

Re: Dynamic linking

#151

Post by danShumway »

techntoke wrote: post #125
cookiecaper wrote: post #115

Earlier quoted context omitted.

It's not Gentoo, but in practice, many frequently-used packages remain in the AUR and an AUR helper to make that transparent is standard kit for anyone using the box as a workstation. It's much more "source-based" than it may appear at first glance. I have ~15G of self-built packages sitting in my PKGDEST right now, if that counts for anything.

Have you checked if they have official packages yet? I was surprised to find that half my AUR packages already were in official recently. Font packages and Android Studio can easily take up 5G. Also, you don't need an AUR helper.

Agreed; I main Arch and I very rarely install packages from AUR. Rare enough that I've never felt the need to install a helper, it would just be unnecessary clutter.

I suspect many users will need to dip into AUR once or twice for unsual packages, but I'm mildly skeptical that most users need to dip into AUR often enough that compile times would be a serious argument against static linking.

Re: Dynamic linking

#152

Post by psykotic »

mforney wrote: post #102
cryptonector wrote: post #23

By far the most important reason for dynamic linking for C is semantics: static linking semantics are stuck in 1978 and suck (more on that below), while dynamic linking semantics make C a much better language. In particular, static linking for C has two serious problems: 1. symbol collisions -> accidental interposition (and crashes); 2. you have to flatten the dependency tree into a topological sort at the final link…

You bring up some good points here. Here are some of my experiences with these problems when working on oasis (my static linux distro). > 1. symbol collisions -> accidental interposition (and crashes); I've encountered symbol collisions only twice, but both resulted in linker errors due to multiple function definitions. I'm not sure how this could happen accidentally. Maybe you are referring to variables in the commo…

> But doesn't the dynamic linker still have to do extra work to resolve the relocations in the executable, even when the dependency libraries are already loaded?

For dynamic linking there's usually only a single reference that needs to be fixed up in the PLT or GOT for each referenced symbol and the fix-ups are all localized, so that part is typically a very small cost. But this means every call to a dynamically linked library goes through an extra stub function, which adds I$ and BTB pressure compared to static linking.

(If you do things manually with dlsym there's also an extra indirection but it's a little different mechanically since you do an indirect call of a function pointer rather than doing a direct call to a forwarding stub. The advantage of the forwarding stub is that even if there's a BTB miss for either of the two direct calls, you don't suffer a branch mispredict but just a few front-end cycles waiting for the instruction decode, which VTune labels as a "branch resteer". The direct call option also leads to smaller code size for each call site which helps with I$ pressure. Basically it's the difference between CALL rel32 and MOV tmp, [RIP + disp32]; CALL [tmp].)

Re: Dynamic linking

#153

Post by danans »

binarycrusader wrote: post #107
jeffbee wrote: post #79

Earlier quoted context omitted.

Not really. Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems. As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything.

Not really. Yes, really. Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems. Definitely not true. Every bit of memory that's not available that could be shared memory instead is a reduction in memory available for filesystem caches, etc. As for "high multi-tenancy" there's nobody out there with server occupancy as high as G…

> Cloud vendor computing models are not generally the computing model of the rest of the world. > Comparison to their environment is not relevant to the general populace.

The way that Google shares server compute and memory resources is by having a service oriented architecture. A single high scale service serves many different applications, usually colocated in the same cluster or data center. Each service is based on multiple instances of a statically linked binary.

At that scale, there is no point in try to use shared dynamically linked libraries to reduce consumption because you save more by either reducing increasing your own app's efficiency or relying on one of the major services rather than linking more functionality into your own app.

Re: Dynamic linking

#154

Post by saagarjha »

wahern wrote: post #131
saagarjha wrote: post #83

Earlier quoted context omitted.

I'm not sure I understand the first link you posted: if a binary is statically linked, why does it need a GOT? It's literally calling functions in its own binary…

But where in its own binary? To support ASLR of code (a ROP mitigation), especially fine-grain randomization, function call sites can't use static addressing, either absolute or relative, pointing directly to a function. To support ASLR of code you can either rewrite every function call site on load (kinda similar to DLLs on Windows), or use one or more tables that are updated at runtime (PLTs, GOT, etc). BSD and Lin…

I know how ASLR/ROP/PLT/GOT/RELRO work, but I'm still not understanding what's going on here. When you statically link a binary, you get…one file. And that gets loaded into memory together, and to make it PIE you have all the jumps be pc-relative. Like, a straight up jump instruction to a fixed offset. So where the the room for a table like this?

(Unrelated, but since you brought it up: kbind is IMO not a very good mitigation. It seems that all you have to do to bypass it is leak a cookie and ROP to that one place in ld.so that is "blessed" and then you not only have the ability to scribble all over your read-only GOT but as far as I can tell you can overwrite any read-only memory, which means it opens up an extremely valuable exploit primitive…)

Re: Dynamic linking

#155

Post by temac »

earthboundkid wrote: post #40

Say we have program X with dependency Y. X+Y is either dynamic or static. X can either have responsive maintainers or unresponsive maintainers. Y can either change to fix a bug or change to add a bug. (With Heartbleed, I remember our server was fine because we were on some ancient version of OpenSSL.) Here are the scenarios: - dynamic responsive remove bug: Positive/neutral. Team X would have done it anyway. - dynami…

You don't necessarily have to give equal weights.

Plus the whole reasoning tends to imply old versions are overall better (you don't have just one bug added or removed from time to time, you have to consider the overall bugginess over time, and if the project is well maintained, it will hopefully eventually decrease, especially if your usage scope remains constant)

But the reality is simply: it depends. On the libraries. On the application. On the platform. On the languages. On the tooling. On the maturity of all of that.

And even so, you may want (or be forced) to take an hybrid path (classic example: a program for Windows -- static is reasonable, but not for platform libs, and actually just not even available in this case)

Re: Dynamic linking

#156

Post by zenhack »

cookiecaper wrote: post #115
abjKT26nO8 wrote: post #39

Earlier quoted context omitted.

> First, this analysis was done on Arch Linux, a source-based distribution. Since you know at compile time what your environment is, I would expect the benefits to be smaller. And of course, this means you're willing to do a lot of recompiles. I'd like to see analysis done on more traditional (& common) binary distros. Arch Linux is not Gentoo. And AUR is only a secondary method of installing software. So I'm not sur…

It's not Gentoo, but in practice, many frequently-used packages remain in the AUR and an AUR helper to make that transparent is standard kit for anyone using the box as a workstation. It's much more "source-based" than it may appear at first glance. I have ~15G of self-built packages sitting in my PKGDEST right now, if that counts for anything.

> an AUR helper to make that transparent is standard kit for anyone using the box as a workstation

Fwiw, I've been using arch as my daily driver continuously for 14 years and haven't ever really felt the strong need for an AUR helper.

Re: Dynamic linking

#157

Post by cryptonector »

kick wrote: post #138
cryptonector wrote: post #93

Earlier quoted context omitted.

Actually, this is a very serious topic, and TFA is wrong in a number of ways. HN commenters being able to say so is everything that's right with HN. My own commentary on TFA is all over the comments on this HN post. I put more effort and detail into my comments on this post than GP, but GP is not wrong. Commenting that a comment is exemplary of all that's wrong with HN is... popular, I'll grant you that, but it grate…

HN commenters being able to say so is everything that's right with HN. This would be correct, if the objection were right. But the commentator was objectively incorrect, in a way that could have been avoided with something as small as a web search, and the comment had the audacity to do so in a condescending way. I didn't knock anyone who had a genuine objection to the post, I replied to an incredibly low-effort and…

I saw nothing wrong with u/dwheeler's comment. I've addressed issues with static linking in more detail up and down this thread, so perhaps you won't tell me to "shut up", but really, it seems all you've got is "shut up". Maybe I could say something clever about comments like yours being representative of all that's wrong with HN, except I don't believe that -- no, I'm glad you get to say "shut up" -unpleasant though it is-, and I'm glad u/dwheeler can express skepticism without going into as much detail as you'd like even though you'll be there telling him to "shut up".

Re: Dynamic linking

#158

Post by binarycrusader »

danans wrote: post #153
binarycrusader wrote: post #107

Earlier quoted context omitted.

Not really. Yes, really. Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems. Definitely not true. Every bit of memory that's not available that could be shared memory instead is a reduction in memory available for filesystem caches, etc. As for "high multi-tenancy" there's nobody out there with server occupancy as high as G…

> Cloud vendor computing models are not generally the computing model of the rest of the world. > Comparison to their environment is not relevant to the general populace. The way that Google shares server compute and memory resources is by having a service oriented architecture. A single high scale service serves many different applications, usually colocated in the same cluster or data center. Each service is based…

I'm aware, which is why I specifically said that cloud vendor computing models aren't really relevant to the general populace. Google's model in particular is (or at least was) very different from other vendors.

Re: Dynamic linking

#159

Post by temac »

jnwatson wrote: post #17
old-gregg wrote: post #10

Earlier quoted context omitted.

Run htop or similar, sort by "shared memory" column and see how much more memory you'd need per process if shared linking did not exist. I think the author's using a wrong method to make a point. Dynamic linking feels out of place for most long-running server-side apps (typical SaaS workload). One can argue that in a mostly CLI-environment there's also not much benefit. But even an empty Ubuntu desktop runs ~400 proc…

Go executables are statically linked. It makes deployment a breeze. I think you overestimate how much saving you get from dynamically linking libc. Each executable uses only a small portion of libc, so the average savings is going to be in the handful of kilobytes per executable.

> Go executables are statically linked. It makes deployment a breeze.

But what are we comparing to what then ?

A few custom Go applications, compared to whole classic Linux distros, on which deployment is both not really the same thing, but still a breeze ?

So yeah, to different needs, different tools.

Re: Dynamic linking

#160

Post by cryptonector »

saagarjha wrote: post #123
btilly wrote: post #116

Earlier quoted context omitted.

I prefer that comment to your reply. By a lot. It attempted to apply reason. You only mocked the form the arguments took. I am firmly of the opinion that content is more important than form. That comment did make a major mistake. Arch Linux is not a source distribution. But the mistake notwithstanding, it was otherwise well-reasoned. Your response was not. You can mock the point that there is still a benefit in shari…

> I am firmly of the opinion that content is more important than form. This is quite possibly the stupidest thing I have heard this year. How could you possibly have come up with something that brain-dead? Is this how you normally are, or did you have to reach deep into your mind to find the most inane thing you could possibly say and pollute Hacker News with it? Goodness, I hope so. I don't even want to think about…

> This is quite possibly the stupidest thing I have heard this year.

I doubt any comment in this thread could possibly be the stupidest thing any of us has heard or read in 2020.

Post reply on HN