Dllicious – shared object usage analysis on Linux • Hacker News Board

Re: Dllicious – shared object usage analysis on Linux

#21

Post by choeger » Sun, Feb 06, 2022, 5:37 PM UTC

adrian_b wrote: post #14

choeger wrote: post #7

> DLL’s add a level of complexity to writing and using software, and newer languages like Rust and Go have eschewed them, Yeah, no. Every sane compiler developer would always prefer DLLs over static linking because of modularity. It makes the compiler much smaller if you can defer the combination of modules to a linker and thus have proper separate compilation. Unfortunately, separate compilation with a C-like ABI on…

I do not understand what you mean. There is no relationship between whether static or dynamic linking is used and modularity. When a program is decomposed in modules, those are compiled separately and the complete executable program is made by linking, either statically or dynamically. The static vs. dynamic option does not influence the semantics of the program regarding modularity. Dynamic linking offers a few extr…

> When a program is decomposed in modules, those are compiled separately

No, they're not. At least not in languages with nontrivial features and optimizations. Consider the identity function id = \\x.x in some module A and it's usage A.id 42 in some module B. Unless you commit to uniform object representation, excluding several optimizations, there's no way to compile A separately from it's use in B (because specialization of A.id is required). That fact excludes the option of creating dynamic libraries because you would expect the dynamic library compiled from A to be used in stead of A (with the necessary type interface data). Similar problems occur with polymorphic data structures.

Separate compilation, modularity, and dynamic linking are all aspects of the same problem.

I think that's why Rust uses a global compilation approach and if I am not completely mistaken, only Swift tries to have dynamic linking with polymorphism.

Re: Dllicious – shared object usage analysis on Linux

#22

Post by choeger » Sun, Feb 06, 2022, 5:38 PM UTC

aij wrote: post #9

choeger wrote: post #7

> DLL’s add a level of complexity to writing and using software, and newer languages like Rust and Go have eschewed them, Yeah, no. Every sane compiler developer would always prefer DLLs over static linking because of modularity. It makes the compiler much smaller if you can defer the combination of modules to a linker and thus have proper separate compilation. Unfortunately, separate compilation with a C-like ABI on…

Static linking does not preclude separate compilation though. Go and Rust both use separate compilation with static linking by default (at least last I used them... it's been a while). I know Rust at least used to support dynamic linking too -- it just wasn't the default. And C also supports static linking of course.

I don't think Rust uses separate compilation. And go just seems to not care about a stable ABI.

Re: Dllicious – shared object usage analysis on Linux

#23

Post by choeger » Sun, Feb 06, 2022, 5:38 PM UTC

gspr wrote: post #17

adrian_b wrote: post #14

Earlier quoted context omitted.

I do not understand what you mean. There is no relationship between whether static or dynamic linking is used and modularity. When a program is decomposed in modules, those are compiled separately and the complete executable program is made by linking, either statically or dynamically. The static vs. dynamic option does not influence the semantics of the program regarding modularity. Dynamic linking offers a few extr…

I don't think the parent meant to use "modularity" to mean "pertaining to a programming language's concept of modules", but rather something along the lines of "pertaining to the act of splitting something into self-contained, clearly delineated, interoperating pieces".

Exactly. Modularity on the frontend is dead simple. Modularity all the way down to executing machine code is hard.

Re: Dllicious – shared object usage analysis on Linux

#24

Post by pjmlp » Sun, Feb 06, 2022, 11:06 PM UTC

choeger wrote: post #22

aij wrote: post #9

Earlier quoted context omitted.

Static linking does not preclude separate compilation though. Go and Rust both use separate compilation with static linking by default (at least last I used them... it's been a while). I know Rust at least used to support dynamic linking too -- it just wasn't the default. And C also supports static linking of course.

I don't think Rust uses separate compilation. And go just seems to not care about a stable ABI.

It surely does, you can pack crates as rlibs.

Re: Dllicious – shared object usage analysis on Linux

#25

Post by mid-kid » Mon, Feb 07, 2022, 9:34 PM UTC

One thing I noticed is that you're running `ldd` on every executable file on the system. Incidentally, `ldd` returns valid output when ran on .so files (which are always executable). This means you're counting an extra use of certain libraries not just for the executables that use them but also the libraries that depend on them.

Also it seems like you're also scanning a bunch of flatpak apps and steam apps, which provide an environment for common libraries, but expect each program to statically link anything not shipped within these common environments.

Both of these skew the graph of most used libraries, and the former skews the additional space required for static linking a little.

It's also worth mentioning that inlining, the lack of a requirement for symbol names/version info/dependency info, and dead code stripping, make static binaries significantly smaller than the size of program + so.

I'm a huge proponent of shared libraries, but these numbers wouldn't really mean much in reality, unfortunately. Though for me, the main benefit of shared libraries is being to individually patch and upgrade them, without a care for how many apps actually use them nor how. This has been a godsent in terms of adding functionality and making my computer behave like I want it to.