Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dllicious – shared object usage analysis on Linux

wiki.alopex.li

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Dllicious – shared object usage analysis on Linux

#11

Post by aij »

fabianhjr wrote: post #5

On my NixOS computer filtering for the most used: 10093 libcap.so.2 49344 libc.so.6 27555 libdl.so.2 13296 libffi.so.8 16112 libgcc_s.so.1 10086 libgcrypt.so.20 11352 libglib-2.0.so.0 10128 libgpg-error.so.0 13015 liblzma.so.5 22664 libm.so.6 11337 libpcre.so.1 29942 libpthread.so.0 11036 libresolv.so.2 19510 librt.so.1 17287 libz.so.1 11362 libzstd.so.1 Though, some usages are not the same as others even if they hav…

Is that having GCed all but the current generation of packages? Or filtering for a single nixpkgs version some other way?

Even then, I would expect NixOS to do some unsharing of shared libraries, so it would probably be best to use the full path to determine how much each shared object is actually shared.

For example, I have 87 different `libzstd.so.1` in my `/nix/store` right now.

Re: Dllicious – shared object usage analysis on Linux

#12

Post by 5a04fcd55b8 »

azalemeth wrote: post #8
5a04fcd55b8 wrote: post #6

Earlier quoted context omitted.

I think dynamic linking is great for saving (DRAM) memory, but for caches I am having trouble imagining a case in which this would make a practical difference. You would need two distinct , long-lived dynamically-linked executables, both being hot at the same time, and both having libc (or another common library) in the hotpath at the same time. To elaborate, - If two hot processes are the same executable, static lin…

> You would need two distinct, long-lived dynamically-linked executables, both being hot at the same time, and both having libc (or another common library) in the hotpath at the same time. Like, for example, bash and a process you are executing in parallel in a loop?

I may be missing something... Is bash exec()ing something in the loop? If so, caches will be the least of our concerns; there will be context switches, system calls, and even I/O, all orders of magnitude slower than even DRAM access. Otherwise, what is bash doing in the loop? When is bash ever CPU-bound?... in a library function call... that is also called by the other process?

Re: Dllicious – shared object usage analysis on Linux

#13

Post by mprovost »

fabianhjr wrote: post #5

On my NixOS computer filtering for the most used: 10093 libcap.so.2 49344 libc.so.6 27555 libdl.so.2 13296 libffi.so.8 16112 libgcc_s.so.1 10086 libgcrypt.so.20 11352 libglib-2.0.so.0 10128 libgpg-error.so.0 13015 liblzma.so.5 22664 libm.so.6 11337 libpcre.so.1 29942 libpthread.so.0 11036 libresolv.so.2 19510 librt.so.1 17287 libz.so.1 11362 libzstd.so.1 Though, some usages are not the same as others even if they hav…

Sorted:

  49344 libc.so.6
  29942 libpthread.so.0
  27555 libdl.so.2
  22664 libm.so.6
  19510 librt.so.1
  17287 libz.so.1
  16112 libgcc_s.so.1
  13296 libffi.so.8
  13015 liblzma.so.5
  11362 libzstd.so.1
  11352 libglib-2.0.so.0
  11337 libpcre.so.1
  11036 libresolv.so.2
  10128 libgpg-error.so.0
  10093 libcap.so.2
  10086 libgcrypt.so.20

Re: Dllicious – shared object usage analysis on Linux

#14

Post by adrian_b »

choeger wrote: post #7

> DLL’s add a level of complexity to writing and using software, and newer languages like Rust and Go have eschewed them, Yeah, no. Every sane compiler developer would always prefer DLLs over static linking because of modularity. It makes the compiler much smaller if you can defer the combination of modules to a linker and thus have proper separate compilation. Unfortunately, separate compilation with a C-like ABI on…

I do not understand what you mean.

There is no relationship between whether static or dynamic linking is used and modularity.

When a program is decomposed in modules, those are compiled separately and the complete executable program is made by linking, either statically or dynamically.

The static vs. dynamic option does not influence the semantics of the program regarding modularity.

Dynamic linking offers a few extra features, e.g. delaying the linking of a library to some time after a program starts and choosing one between more libraries at that time, but exactly the same functionality can be implemented in a statically linked program (using pointers to functions), with no difference in behavior (but with different costs in memory space and execution time; which costs are larger will be different for each particular case).

Actually a compiler that targets only static linkers will be slightly smaller, because many of the standards for dynamic linking, e.g. the UNIX SVR4/ELF ABI, require the compiler to emit additional instructions and data structures whenever external variables or functions are accessed, in comparison with the case when only static linking is used.

Dynamic linking is an additional complication for the compiler, not a simplification. Proper separate compilation of modules is the easiest with only static linking, when the compiler just has to emit appropriate relocation and linking data (which was actually the job of an assembler for the traditional UNIX compilers, which generated an assembly program, not an ELF object file), besides what it needs to do for compiling a monolithic program.

Re: Dllicious – shared object usage analysis on Linux

#15

Post by zeroping »

gbin wrote: post #10

The size computation is overblown no? A static link would probably remove most of the symbols as they are probably unused?

The author specifically called that out. You are correct.

They also didn't look at DLLs used by other DLLs.

This is like a back-of-the-envelope calculation. A quick first pass to get an order of magnitude.

Re: Dllicious – shared object usage analysis on Linux

#16

Post by Vogtinator »

zeroping wrote: post #15
gbin wrote: post #10

The size computation is overblown no? A static link would probably remove most of the symbols as they are probably unused?

The author specifically called that out. You are correct. They also didn't look at DLLs used by other DLLs. This is like a back-of-the-envelope calculation. A quick first pass to get an order of magnitude.

> They also didn't look at DLLs used by other DLLs.

If so, then the data is too wrong to be of any value.

However, ldd lists also indirect dependencies so I think this was already accounted for (though that might've been by accident):

~> readelf -d /bin/bash | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libreadline.so.8] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] ~> ldd /bin/bash linux-vdso.so.1 (0x00007fff15191000) libreadline.so.8 => /lib64/libreadline.so.8 (0x00007f2a24818000) libc.so.6 => /lib64/libc.so.6 (0x00007f2a2460e000) libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f2a245db000) /lib64/ld-linux-x86-64.so.2 (0x00007f2a249b1000)

Re: Dllicious – shared object usage analysis on Linux

#17

Post by gspr »

adrian_b wrote: post #14
choeger wrote: post #7

> DLL’s add a level of complexity to writing and using software, and newer languages like Rust and Go have eschewed them, Yeah, no. Every sane compiler developer would always prefer DLLs over static linking because of modularity. It makes the compiler much smaller if you can defer the combination of modules to a linker and thus have proper separate compilation. Unfortunately, separate compilation with a C-like ABI on…

I do not understand what you mean. There is no relationship between whether static or dynamic linking is used and modularity. When a program is decomposed in modules, those are compiled separately and the complete executable program is made by linking, either statically or dynamically. The static vs. dynamic option does not influence the semantics of the program regarding modularity. Dynamic linking offers a few extr…

I don't think the parent meant to use "modularity" to mean "pertaining to a programming language's concept of modules", but rather something along the lines of "pertaining to the act of splitting something into self-contained, clearly delineated, interoperating pieces".

Re: Dllicious – shared object usage analysis on Linux

#18

Post by matthias509 »

5a04fcd55b8 wrote: post #6
matthias509 wrote: post #3

Saving disk space isn’t nearly as exciting as saving CPU cache space which is much more precious. That means that something like libc will almost always be in cache from something calling it.

I think dynamic linking is great for saving (DRAM) memory, but for caches I am having trouble imagining a case in which this would make a practical difference. You would need two distinct , long-lived dynamically-linked executables, both being hot at the same time, and both having libc (or another common library) in the hotpath at the same time. To elaborate, - If two hot processes are the same executable, static lin…

Imagine the other extreme where every single executable on the system statically linked libc. Assuming libc calls are pretty common(which I think is safe) I would expect the cache to get trashed.

Re: Dllicious – shared object usage analysis on Linux

#19

Post by jcranmer »

choeger wrote: post #7

> DLL’s add a level of complexity to writing and using software, and newer languages like Rust and Go have eschewed them, Yeah, no. Every sane compiler developer would always prefer DLLs over static linking because of modularity. It makes the compiler much smaller if you can defer the combination of modules to a linker and thus have proper separate compilation. Unfortunately, separate compilation with a C-like ABI on…

As a compiler developer, yeah, no, you're wrong. Dynamic linking creates several more challenges for compilation than static linking.

If you're trying to be cross-platform, you have the major issue that dynamic libraries have very different models on different platforms. Windows DLLs require all of the exported--and imported--names to be explicitly identified at compile time; MACH-O and ELF files don't require that. On the other hand, ELF files have symbol visibility which can be toggled to do something similar, but the standard expectations in C are completely different: every symbol is capable of being exported (or overridden--thanks symbol preemption!). (I don't have much familiarity with MACH-O's two-namespace system, so I won't talk about it further).

There are other issues. DSOs make it hard to use linker features such as arranging things in a section to make a distributed static list (e.g., for registering reflection stuff). Even doing static constructors in dynamic libraries is hard. Arranging for single static addresses (as C/C++ require) can be challenging. On Windows, malloc in one DLL can't be freed by free in another... sometimes. It also requires defining a stable ABI, since you have no reasonable expectation that the other side of the dynamic library is using the same compiler version you are. In general, crossing the dynamic library boundary is like doing FFI, given how little control you have over the process.

No, static linking is much easier. You don't have to hold in your head so much more crazy semantics with dynamic linking.

Post reply on HN