Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sorry state of dynamic libraries on Linux

macieira.org

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Sorry state of dynamic libraries on Linux

#21

Post by marshray »

ice799 wrote: post #14
psykotic wrote: post #6

While we're complaining about the sorry state of Linux, something it's desperately lacking (for those of us who ship binary-only libraries and executables) is an equivalent of Windows's PDBs or OS X's dSYMs for post-mortem debugging without bloating the shipping binaries.

hi. linux does have this. you just strip the debug symbols out (and put them somewhere safe). then write a .gnu_debuglink section to the stripped ELF binary with a CRC that matches the stripped symbols. once something bad happens: you just take the core dump, the symbols you have tucked away, and you are able to debug just fine.

This is great! Seems like this may have just become available in the last few years?

Re: Sorry state of dynamic libraries on Linux

#22

Post by Rexxar »

marshray wrote: post #13
brigade wrote: post #4

I started writing a comment before the site went down (hah, I knew it would be WordPress) and now I'll probably forget to post it there so I'll just post it here. Is the address of externalFunction that is stored in the GOT really resolved to the address of the stub and not the final real address of the function? Because it has to be for data (or code simply wouldn't work), and I don't see why function symbols would…

C and C++ say that a function has the same address in all translation units Until perhaps very recently, the ISO C and C++ standards didn't actually support dynamic linking. (They didn't officially support multithread concurrency either but flexibility is one of those languages' strengths). if your final binary is over 2GB the linker could fail to put the symbol within range of the offset and fail I have had to code…

I'm very curious to know how you manage to need 2GB final binary ?

- Huge use of template metaprogramming ?

- Generated code ?

- May be it's only for the debug build with all symbols ?

Re: Sorry state of dynamic libraries on Linux

#23

Post by klodolph »

gioele wrote: post #20
dkarl wrote: post #5

All this for the possibility of interposition? Yes, it seems so. The impact is there for this little-known and little-used feature. Instead of optimising for the common-case scenario where the symbols are not overridden, the ABI optimises for the corner case. Using LD_PRELOAD isn't so rare and strange that you can propose throwing it out without quantifying its performance cost. I can't recall offhand why I needed it…

> All this for the possibility of interposition? Yes, it seems so. The impact is there for this little-known and little-used feature. Instead of optimising for the common-case scenario where the symbols are not overridden, the ABI optimises for the corner case. It also looks like PIE+PIC is required if you want a secure system with ASLR: " rel="nofollow">http://blog.flameeyes.eu/2009/11/02/the-pie-is-not-exactly-a...…

Of course, a bunch of this work is sabotaged by stupid default `LDFLAGS` you get if you use `pkg-config` and some package (like `gmodule`) throws random stuff in there like `-Wl,--export-dynamic` which is just totally unnecessary for at least 99% of executables which only ever used `gmodule` indirectly in the first place...

Re: Sorry state of dynamic libraries on Linux

#24

Post by ars »

dkarl wrote: post #5

All this for the possibility of interposition? Yes, it seems so. The impact is there for this little-known and little-used feature. Instead of optimising for the common-case scenario where the symbols are not overridden, the ABI optimises for the corner case. Using LD_PRELOAD isn't so rare and strange that you can propose throwing it out without quantifying its performance cost. I can't recall offhand why I needed it…

LD_PRELOAD is used by fakeroot, artsdsp, esddsp, alsadsp to name some of the most popular programs that use it.

Re: Sorry state of dynamic libraries on Linux

#25

Post by klodolph »

brigade wrote: post #4

I started writing a comment before the site went down (hah, I knew it would be WordPress) and now I'll probably forget to post it there so I'll just post it here. Is the address of externalFunction that is stored in the GOT really resolved to the address of the stub and not the final real address of the function? Because it has to be for data (or code simply wouldn't work), and I don't see why function symbols would…

Yes, when you call a function you often jump to the stub. When you take the address of the function, it does something more complicated for PIC executables. Try examining the assembly of the following code:

    void func1(void);
    void *func2(void) { return func1; }
You'll notice that unlike function calls, the code here differs with PIC enabled. I think your assumption is that function calls and function addresses in C use the same address, which is not true.

Re: Sorry state of dynamic libraries on Linux

#26

Post by bch »

I think it was here (on HN) that I read a comment that suggested shared libs in Unix originated at Sun Microsystems, originating due to some politics in their work w/ the X Window System (iirc). I've searched for that story since, but not found it -- I'm guessing there may be somebody reading this story who may know what I'm talking about. Retell the story?

Re: Sorry state of dynamic libraries on Linux

#27

Post by brigade »

klodolph wrote: post #25
brigade wrote: post #4

I started writing a comment before the site went down (hah, I knew it would be WordPress) and now I'll probably forget to post it there so I'll just post it here. Is the address of externalFunction that is stored in the GOT really resolved to the address of the stub and not the final real address of the function? Because it has to be for data (or code simply wouldn't work), and I don't see why function symbols would…

Yes, when you call a function you often jump to the stub. When you take the address of the function, it does something more complicated for PIC executables. Try examining the assembly of the following code: void func1(void); void *func2(void) { return func1; } You'll notice that unlike function calls , the code here differs with PIC enabled. I think your assumption is that function calls and function addresses in C u…

My assumption (in your example) was actually that with PIC, func2 would return the address of the real func1 and a hypothetical 'func1();' would call a stub that calls the real func1. On OS X, this is true. On Linux, func2 returns the address of the stub, and I see no reason why (DYLD_INSERT_LIBRARIES works fine on OS X.)

This is actually the specific thing that prompted the author to investigate and write up this whole post. It's so what the fuck to me that I didn't believe him at all until I tried it myself.

EDIT: actually, I finally realized one potential benefit to this: it lets you completely lazily resolve the functions address. But again, variables can't benefit from this so need to be resolved immediately. I really wonder whether the load-time gain is actually worth the runtime hit for this case...

Re: Sorry state of dynamic libraries on Linux

#28

Post by marshray »

Rexxar wrote: post #22
marshray wrote: post #13

Earlier quoted context omitted.

C and C++ say that a function has the same address in all translation units Until perhaps very recently, the ISO C and C++ standards didn't actually support dynamic linking. (They didn't officially support multithread concurrency either but flexibility is one of those languages' strengths). if your final binary is over 2GB the linker could fail to put the symbol within range of the offset and fail I have had to code…

I'm very curious to know how you manage to need 2GB final binary ? - Huge use of template metaprogramming ? - Generated code ? - May be it's only for the debug build with all symbols ?

One can include arbitrary amounts of data in binaries. I don't see this for ELF systems, but consider all the Windows installers that package entire applications into a single .exe.

Re: Sorry state of dynamic libraries on Linux

#29

Post by wavetossed »

For an interesting use of RPATH headers to make portable Linux binaries (and shared libraries) have a look at this script that I used to build Python 2.7.2 and a whole pile of 3rd party libraries https://github.com/wavetossed/pybuild

I think that things like RPATH and LD_PRELOAD are exactly what shared libraries should be doing. The reason for shared libraries in the modern age, is increased flexibility.

Re: Sorry state of dynamic libraries on Linux

#30

Post by forgotusername »

ars wrote: post #24
dkarl wrote: post #5

All this for the possibility of interposition? Yes, it seems so. The impact is there for this little-known and little-used feature. Instead of optimising for the common-case scenario where the symbols are not overridden, the ABI optimises for the corner case. Using LD_PRELOAD isn't so rare and strange that you can propose throwing it out without quantifying its performance cost. I can't recall offhand why I needed it…

LD_PRELOAD is used by fakeroot, artsdsp, esddsp, alsadsp to name some of the most popular programs that use it.

Only for overriding parts of libc, which already cannot be 'unexported'. The article's main beef is with internal program symbols also being made indirect (AIUI a patch adding a new visibility option was landed in gcc years ago that more or less automatically fixed this up, but I may be wrong)

Edit: rushing out the door here, last sentence was referring to this: http://gcc.gnu.org/wiki/Visibility

Post reply on HN