Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the Linker (2012)

opensource.apple.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Inside the Linker (2012)

#21

Post by jstimpfle »

ma2rten wrote: post #10

I work on C++ code at a large tech company. My workflow is such that I compile and run tests often after I made incremental changes. I feel like I often wait for the linker for ages. I am wondering if linking could be sped up. I feel like a lot of the steps mentioned here could be cached for instance.

I, too, have been wondering that. On a high level, a linker does these things:

- Read in objects file and parse linking information (needed symbols etc)

- Do some in-memory calculations - resolving symbols and some other stuff. Simple graph theory. Is there any need for more than O(n log n) runtime?.

- Write out object files

I don't think anything should take so long. No caching needed. I read the series on linkers by gold's author, where significant speedups were claimed. And also, that object-oriented C++ was the right approach to software architecture. I strongly disagree with that second point and didn't understand too much from the article series to be honest. So I am not surprised at all that there are today even much faster linkers.

Re: Inside the Linker (2012)

#22

Post by jstimpfle »

comex wrote: post #18
bear_child wrote: post #12

Earlier quoted context omitted.

Can you say more about the last paragraph? I have often wondered if you could avoid identifier mangling with a better designed object file format

Hmm… well, when it comes to identifier mangling, object formats aren't really the problem. Both ELF and Mach-O use nul termination for symbol (and section) names, so they can't contain 00 bytes, but there's nothing in the binary format preventing them from containing any other bytes. So you could make a symbol named foo::bar(int,int) …and most likely, everything that deals with binaries would have no problem with it.…

Why would you want run-time code generation for things that you can also generate at compile time? A linker is there to link, not to compile.

One of the big problems with templates is that you're not required to instantiate them explicitly. So you end up with lots of duplicate instantiations. Maybe try doing it explicitly, always - have a well defined place where the template is instantiated. You can also put the compiled code in a library, I think. (There seems to be an extern template feature since C++11).

Anyways, templates are a mess...

Re: Inside the Linker (2012)

#23

Post by jacobush »

johncolanduoni wrote: post #19
comex wrote: post #9

Earlier quoted context omitted.

As input to a linker, Mach-O and ELF files look pretty similar: they both split their contents into named "sections", and then there's a symbol table, which is basically a list of (name, address) pairs. In C code, each symbol generally represents (the start of) a separate function or variable. All references between them are explicitly marked as relocations; thus, the linker should be free to reorder them, and any fu…

> Heck, I don't understand why -dead_strip isn't enabled by default. My guess would be to facilitate debugging, where you might want to invoke a symbol that wouldn't otherwise be used at runtime (e.g. some sort of debug print).

Can it be also for something dynamically load a symbol and call it? Like for .so making

Re: Inside the Linker (2012)

#24

Post by userbinator »

jacobush wrote: post #23
johncolanduoni wrote: post #19

Earlier quoted context omitted.

> Heck, I don't understand why -dead_strip isn't enabled by default. My guess would be to facilitate debugging, where you might want to invoke a symbol that wouldn't otherwise be used at runtime (e.g. some sort of debug print).

Can it be also for something dynamically load a symbol and call it? Like for .so making

Yes, as someone with a Windows background I think this is one of the most unusual aspects of the dynamic linking system on Unix-likes --- everything is "exported" by default and the basic system has no concept of linking to symbols from a specific module, it only looks at the symbol name itself.

In Windows you have to explicitly specify which symbols to export, and only those exported ones can be imported.

Re: Inside the Linker (2012)

#25

Post by klodolph »

userbinator wrote: post #24
jacobush wrote: post #23

Earlier quoted context omitted.

Can it be also for something dynamically load a symbol and call it? Like for .so making

Yes, as someone with a Windows background I think this is one of the most unusual aspects of the dynamic linking system on Unix-likes --- everything is "exported" by default and the basic system has no concept of linking to symbols from a specific module, it only looks at the symbol name itself. In Windows you have to explicitly specify which symbols to export, and only those exported ones can be imported.

On Windows this has the consequence that if you have a DLL, it might use a different malloc than your application. As a result, you can’t safely free objects that were created in a DLL unless you go to all sorts of trouble. This can mean passing objects back into a DLL to free them or using some other mechanism to ensure that you use the same allocator everywhere. This is especially problematic with C++ because of inlining.

The “everything is exported” default can be fixed with a flag, you then set symbol visibility with attributes like __declspec or use a linker script. This doesn’t change the calling convention, unlike with DLLs.

Re: Inside the Linker (2012)

#26

Post by ljcn »

User23 wrote: post #6

This is really beautiful HTML

Indeed, retro. Not quite valid though unfortunately. Apart from the annoying character encoding, doctype, etc. problems, there's a

without a

, and a missing a preceding .

Re: Inside the Linker (2012)

#27

Post by comex »

jacobush wrote: post #23
johncolanduoni wrote: post #19

Earlier quoted context omitted.

> Heck, I don't understand why -dead_strip isn't enabled by default. My guess would be to facilitate debugging, where you might want to invoke a symbol that wouldn't otherwise be used at runtime (e.g. some sort of debug print).

Can it be also for something dynamically load a symbol and call it? Like for .so making

Only if the symbol is exported, but in that case it won’t be removed even if -dead_strip is used.

As userbinator noted, Unix has an awful tradition of exporting most symbols by default, which is also in effect on macOS. The modern approach is to pass -fvisibility=hidden to the compiler, which changes the default to unexported, and then mark symbols that are supposed to be exported with __attribute__((visibility(“default”))). Alternately, there’s a Darwin-specific ld option (-exported_symbols_list) where you write a list of names to export in a separate text file, and everything else is forced to be unexported.

Re: Inside the Linker (2012)

#28

Post by comex »

klodolph wrote: post #25
userbinator wrote: post #24

Earlier quoted context omitted.

Yes, as someone with a Windows background I think this is one of the most unusual aspects of the dynamic linking system on Unix-likes --- everything is "exported" by default and the basic system has no concept of linking to symbols from a specific module, it only looks at the symbol name itself. In Windows you have to explicitly specify which symbols to export, and only those exported ones can be imported.

On Windows this has the consequence that if you have a DLL, it might use a different malloc than your application. As a result, you can’t safely free objects that were created in a DLL unless you go to all sorts of trouble. This can mean passing objects back into a DLL to free them or using some other mechanism to ensure that you use the same allocator everywhere. This is especially problematic with C++ because of in…

What you said about malloc on Windows is true, but I wouldn’t blame it on symbols not being exported by default – but rather on Windows’ choice to ship N different C runtime libraries, one for each MSVC version, as separate DLLs. Most applications and libraries link to one of those DLLs, and if two images link to the same DLL, they do share C runtime state, including the malloc heap [1]. But that’s only possible if they were compiled with the same MSVC version. (There is also an option, -MT, to fully statically link the C runtime, but it’s less commonly used.)

In contrast, both Linux and Darwin have only one C runtime library for the system, in libc.so.6 (typically) or libSystem.dylib respectively, which maintains backwards compatibility over a longer time period. Sometimes there’s a need to make changes to libc that would normally be ABI-breaking, e.g. when off_t was changed to be 64 bits to support >4GB file sizes on 32-bit platforms. But to avoid breakage, special mechanisms are used to provide two different versions of the same symbol within the same library, for each affected symbol; existing binaries will use the old version, while programs compiled and linked on a newer system will automatically use the new version. (On Linux, a complicated mechanism called symbol versioning is used for this, while Darwin just renames symbols with the asm() directive in header files.)

On Linux, it is possible to statically link any of various libcs, but programs that do that typically can’t use shared libraries * at all*, so the issue of malloc across library boundaries doesn’t come up.

Oh, and for the record, apparently Windows 10 has a new “universal” CRT DLL for new code that will be maintained in place going forward, but it’s still separate from all the pre-existing versioned DLLs, and there’s a debug variant that’s a separate DLL from the normal one and probably doesn’t share state with it.

[1] https://msdn.microsoft.com/en-us/library/ms235460.aspx

Re: Inside the Linker (2012)

#29

Post by userbinator »

klodolph wrote: post #25
userbinator wrote: post #24

Earlier quoted context omitted.

Yes, as someone with a Windows background I think this is one of the most unusual aspects of the dynamic linking system on Unix-likes --- everything is "exported" by default and the basic system has no concept of linking to symbols from a specific module, it only looks at the symbol name itself. In Windows you have to explicitly specify which symbols to export, and only those exported ones can be imported.

On Windows this has the consequence that if you have a DLL, it might use a different malloc than your application. As a result, you can’t safely free objects that were created in a DLL unless you go to all sorts of trouble. This can mean passing objects back into a DLL to free them or using some other mechanism to ensure that you use the same allocator everywhere. This is especially problematic with C++ because of in…

This can mean passing objects back into a DLL to free them or using some other mechanism to ensure that you use the same allocator everywhere.

That's what the LocalAlloc/LocalFree functions are for.

Re: Inside the Linker (2012)

#30

Post by userbinator »

comex wrote: post #28
klodolph wrote: post #25

Earlier quoted context omitted.

On Windows this has the consequence that if you have a DLL, it might use a different malloc than your application. As a result, you can’t safely free objects that were created in a DLL unless you go to all sorts of trouble. This can mean passing objects back into a DLL to free them or using some other mechanism to ensure that you use the same allocator everywhere. This is especially problematic with C++ because of in…

What you said about malloc on Windows is true, but I wouldn’t blame it on symbols not being exported by default – but rather on Windows’ choice to ship N different C runtime libraries, one for each MSVC version, as separate DLLs. Most applications and libraries link to one of those DLLs, and if two images link to the same DLL, they do share C runtime state, including the malloc heap [1]. But that’s only possible if t…

There is one CRT that has existed since Win95 and is still there in Win10, it's called MSVCRT.DLL and getting versions of MSVC other than 6 to link to it is possible and has been done (although not trivial). MS rather strongly discourages this, but as evidenced by all the apps out there that do it and continue to work, it's pretty much the only way to get a single small dynamically-linked binary that'll run on every 32-bit version of Windows ever released.

In contrast the "universal" CRT is not very universal at all, and a horrible bloated mess. But certainly not unusual of MS...

Post reply on HN