Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dynamic linking

drewdevault.com

Post reply on HN
Print view View original

231–240 of 249 posts

Re: Dynamic linking

#231

Post by saurik »

cryptonector wrote: post #171
saurik wrote: post #168

Earlier quoted context omitted.

Most linkers support archive "groups" at the command line, so you don't need to do the topological sort. Most linkers also support relocatable objects, so you can trivially solve the symbol collision problem by just linking together a subset of your app and then stripping that symbol before linking the conflict. And like, it frankly just sounds like you have never heard of libtool, which adds most of the stuff in you…

I know all about libtool, thank you, and I'm quite aware that it writes dependency information into its .la files. I use libstool every day. It sucks. This functionality has to be built into the linker.

Making the linker--which is architecture specific and generally "just barely what is required to munge together object files"--into some fat stack of code (that I guess you would expect to work the same for every single linker) because you don't believe wrappers are appropriate for some reason is a disagreement over "clean reusable architecture" and not that the ecosystem as a whole is somehow "stuck in the 80s" :/.

Re: Dynamic linking

#232

Post by ncmncm »

cryptonector wrote: post #162
ncmncm wrote: post #128

Paging Laura Creighton! Your time is now.

What should I search for to get a clue as to what you're referring to?

Her name, and USENET. She was the most vocal opponent of the move to dynamic-linked libraries, back when it happened. Her expectation was that programs that once worked would stop working as the libraries rotted.

Re: Dynamic linking

#233

Post by 08-15 »

pjmlp wrote: post #225
08-15 wrote: post #222

Earlier quoted context omitted.

> Without shared libraries, the only way to do plugins is via IPC. That's primitive superstition: - dlopen() works in statically linked programs too. - Even without dlopen(), you can load code dynamically. It's not magic, especially if the plugin is linked statically. - Besides, how is IPC bad? I take fcgi over Apache's modules any day.

dlopen is useless in static linked programs when the idea is to provide plugins after the fact. The idea of using plugins is exactly that various parts are able to ship them at various times during the lifetime of the applications. Patching files compiled statically is obviously not what one wants from plugins. Incidently plugins support is exactly why Go added support for dynamic linking into their toolchain.

One of us is confused; I honestly don't know which one.

Where is the problem in this scenario? My application (statically) linked provides a plugin interface. It works by calling dlopen() on the plugin and then passes a record of functions to the entry points. The plugin can be provided by a third party, and top make that easier, I provide an SDK (a header file).

No file is patched, and I have no idea what this has to do with Go. Enlighten me?

Re: Dynamic linking

#234

Post by pjmlp »

08-15 wrote: post #233
pjmlp wrote: post #225

Earlier quoted context omitted.

dlopen is useless in static linked programs when the idea is to provide plugins after the fact. The idea of using plugins is exactly that various parts are able to ship them at various times during the lifetime of the applications. Patching files compiled statically is obviously not what one wants from plugins. Incidently plugins support is exactly why Go added support for dynamic linking into their toolchain.

One of us is confused; I honestly don't know which one. Where is the problem in this scenario? My application (statically) linked provides a plugin interface. It works by calling dlopen() on the plugin and then passes a record of functions to the entry points. The plugin can be provided by a third party, and top make that easier, I provide an SDK (a header file). No file is patched, and I have no idea what this has t…

Your solution requires compiling everything from scratch in a single executable, that is not how plugins are supposed to be designed.

We have been there before with solutions like graphics drivers for Borland's BGI library for their MS-DOS compilers. Where a driver interface is provided, and then each "plugin" registers themselves on application startup.

Adding new plugins to an existing application in this scenario requires recompilation and updating the uses/#include being used for plugins, or patch the executable from an existing .obj file.

If you are using dlopen on a third party binary instead of your own, then you are already using dynamic linking by definition, by loading third party code dynamically and revolving the proper address locations for all symbols.

Go only had static compilation on the beginning as the only true way, but it failed short exactly in this scenario, to the point that eventually plugin package came to be and dynamic compilation support is now a feature of Go's toolchain as well, although many seem to still not be aware that Go compilers can also produced dynamic libraries.

Re: Dynamic linking

#235

Post by tannhaeuser »

pjmlp wrote: post #234
08-15 wrote: post #233

Earlier quoted context omitted.

One of us is confused; I honestly don't know which one. Where is the problem in this scenario? My application (statically) linked provides a plugin interface. It works by calling dlopen() on the plugin and then passes a record of functions to the entry points. The plugin can be provided by a third party, and top make that easier, I provide an SDK (a header file). No file is patched, and I have no idea what this has t…

Your solution requires compiling everything from scratch in a single executable, that is not how plugins are supposed to be designed. We have been there before with solutions like graphics drivers for Borland's BGI library for their MS-DOS compilers. Where a driver interface is provided, and then each "plugin" registers themselves on application startup. Adding new plugins to an existing application in this scenario…

That may all be true, but I guess GP was asking what that has to do with statically linking your main binary (or not), which of course you can do, yet still use shared libs as plugin loading mechanism. For example, ODBC drivers (with or without the various unixODBC/iODBC proxies) can be dlopen()ed from otherwise statically compiled main binaries just fine, can't they (even though for ODBC the typical use case is using a driver manager rather than invoking dlopen() directly)?

Re: Dynamic linking

#236

Post by thu2111 »

ridiculous_fish wrote: post #167

This is a crazy conversation! Dynamic linking allows the system to decide the UI. Static linking means that the UI cannot evolve. Imagine statically linking UIKit or Android's UI library!

UI libraries tend to evolve by introducing new APIs or modes, because changing UI implementations has a habit of breaking apps.

For example, on Android even the switch to hw accelerated rendering was a mode app devs had to opt in to, that didn't even change the look!

Re: Dynamic linking

#237

Post by cryptonector »

saurik wrote: post #231
cryptonector wrote: post #171

Earlier quoted context omitted.

I know all about libtool, thank you, and I'm quite aware that it writes dependency information into its .la files. I use libstool every day. It sucks. This functionality has to be built into the linker.

Making the linker--which is architecture specific and generally "just barely what is required to munge together object files"--into some fat stack of code (that I guess you would expect to work the same for every single linker) because you don't believe wrappers are appropriate for some reason is a disagreement over "clean reusable architecture" and not that the ecosystem as a whole is somehow "stuck in the 80s" :/.

Part of what has to be done can be done in wrappers, it's true: the part that has to do with recording additional metadata. But the other part -using that metadata to resolve symbol conflicts- cannot be done in a wrapper -- it has to be done in the linker-editor. IMO this is a very strong argument for putting all of this in the linker-editor.

Re: Dynamic linking

#238

Post by 08-15 »

pjmlp wrote: post #234
08-15 wrote: post #233

Earlier quoted context omitted.

One of us is confused; I honestly don't know which one. Where is the problem in this scenario? My application (statically) linked provides a plugin interface. It works by calling dlopen() on the plugin and then passes a record of functions to the entry points. The plugin can be provided by a third party, and top make that easier, I provide an SDK (a header file). No file is patched, and I have no idea what this has t…

Your solution requires compiling everything from scratch in a single executable, that is not how plugins are supposed to be designed. We have been there before with solutions like graphics drivers for Borland's BGI library for their MS-DOS compilers. Where a driver interface is provided, and then each "plugin" registers themselves on application startup. Adding new plugins to an existing application in this scenario…

> then you are already using dynamic linking by definition

"Well, technically..."

I proposed using the dlopen() mechanism (or a custom linker) from a program that is itself statically linked. Because that's what the argument is about: statically linking the stuff you always need. Now, technically, that's dynamic linking. And because I'm linking the plugins dynamically, I might as well link everything dynamically, right? No, absolutely not. Because the latter gives me DLL hell, and rpaths, and library maintainers who think that changing LD_LIBRARY_PATH is perfectly sensible. The former doesn't. Not equivalent. Not at all.

We're not talking about BGI or overlays or Go. We're talking about statically linked binaries calling the dynamic linker. You didn't actually know that was possible, did you?

Re: Dynamic linking

#239

Post by vlovich123 »

pjmlp wrote: post #207
vlovich123 wrote: post #142

Earlier quoted context omitted.

From distributions where you build everything from source (most cloud providers today, mobile phone operators, etc), such advantages are purely theoretical & never play out in practice. In such environments you're pushing out updates of the entire system on whatever your regular cadence is. Additionally, because the old library is still mapped & running you have to know to restart all the processes that have the old…

Where are those mobile phone operators building everything from scratch? Because from my telecommunications and mobile OS development knowledge I have hard time remembering at least one. And binary deployments? They are done all the time.

When I said mobile phone operators, I meant more Apple and Google, not the telecom operators. I haven't worked at telecom operators but I wouldn't be surprised if that world is wildly different - IT is generally considered a cost center rather than as way to remove costs in other parts of the org.

Apple, Google, Microsoft, Amazon, Facebook all build everything from source. Source: I worked at 3 of those & have friends coworkers at the rest. For cloud users I don't have as good a knowledge of that space. I imagine the majority of them use off-the-shelf prebuilt libraries that come with the OS they run on.

Re: Dynamic linking

#240

Post by vlovich123 »

zelly wrote: post #146
vlovich123 wrote: post #142

Earlier quoted context omitted.

From distributions where you build everything from source (most cloud providers today, mobile phone operators, etc), such advantages are purely theoretical & never play out in practice. In such environments you're pushing out updates of the entire system on whatever your regular cadence is. Additionally, because the old library is still mapped & running you have to know to restart all the processes that have the old…

Suppose 100 processes on the same box call the same function in glibc. If each process had its own redundant version of it mapped into memory, then that's 100x more cache misses compared to them all sharing the same page. I get that only a handful of libraries are actually used, but that's a very fat tail and the performance will be very bad if that handful of libraries are always statically linked. This may be less…

I would highly recommend you shift your thinking & consider even something as basic as threads as a distributed systems problem. The distributed systems field has a much richer successful history of how to design robust systems at scale (not just in terms of compute resources but also in terms of # of people working on the problem & number of components). Sharing memory is actually not a simple & solved problem but distributed systems problem generally have lots of formal methods & tools for dealing with unreliability of any given component.

That's also ignoring that measuring the distributed cost of a component is fundamentally simpler than measuring it for a library mapped into 50 million processes.

Shared libraries can be useful but their value is often vastly overstated.

Post reply on HN