Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mold: A Modern Linker

github.com

Post reply on HN
Print view View original

11–20 of 125 posts

Re: Mold: A Modern Linker

#11

Post by spijdar »

klodolph wrote: post #6
_a1_ wrote: post #5

> As an implementation strategy, we do not care about memory leak because we really can't save that much memory by doing precise memory management. It is because most objects that are allocated during an execution of mold are needed until the very end of the program. I'm sure this is an odd memory management scheme (or the lack thereof), but this is what LLVM lld does too. The fact that someone does it wrong doesn't…

If you think this approach is wrong, could you articulate the reasons why you think it is wrong? This is a classic memory management strategy... if a program is running as a batch program, all memory will be freed when the program exits. Any alternative memory management strategy would have to free and then reuse memory in order to show improvement. If it's a small amount of memory freed, or if the memory is unlikely…

Yeah, this is also the strategy GCC and co use generally AFAIK. In a program like GCC where a single invocation will operate over a single file/unit, there's just not much benefit to trying to re-use data; if GCC or LLVM were closer to "build servers" with persistent state that compiled and linked objects on demand then it'd make sense, but in their current model, it's easier and safer to just keep data around.

Re: Mold: A Modern Linker

#12

Post by dataflow »

rui314 wrote: post #3
CyberRabbi wrote: post #2

> I won't avoid Unix-ism when writing code (e.g. I'll probably use fork(2)). It’s probably fine to not avoid most Unix APIs but fork() is truly an exception here. Fork() is not friendly to any third party library you may use because their state may become invalidated after a fork but the library has no way to know if the process has been forked. This is especially bad if the library uses multi threading. The best way…

Author here. Good point. I ended up not using fork() without exec(), so that should be fine now, but here is my original plan to use fork(): I wanted to keep a linker process running as a daemon so that it doesn't read the same files over and over again. After loading input files, the linker becomes a daemon and calls fork() to create a worker process. Then the worker process does the rest of linking. In other word,…

> in the end I had to give up with the fork()-based worker process design.

Honestly that's a good thing, or your program just fundamentally couldn't possibly ever work on native Windows!

Re: Mold: A Modern Linker

#13

Post by jdblair »

From a marketing perspective, "mold" meaning "a form used to cast an object from liquid" is a lot more appealing than "green fungus growing on bread." A mold for casting objects is also a lot closer, metaphorically speaking, to what a linker does.

I honestly thought that was the meaning the author was trying to evoke before I saw the picture on the github page.

Re: Mold: A Modern Linker

#14

Post by pjmlp »

dataflow wrote: post #12
rui314 wrote: post #3

Earlier quoted context omitted.

Author here. Good point. I ended up not using fork() without exec(), so that should be fine now, but here is my original plan to use fork(): I wanted to keep a linker process running as a daemon so that it doesn't read the same files over and over again. After loading input files, the linker becomes a daemon and calls fork() to create a worker process. Then the worker process does the rest of linking. In other word,…

> in the end I had to give up with the fork()-based worker process design. Honestly that's a good thing, or your program just fundamentally couldn't possibly ever work on native Windows!

Or mainframe OSes.

Re: Mold: A Modern Linker

#15

Post by nothasan »

jdblair wrote: post #13

From a marketing perspective, "mold" meaning "a form used to cast an object from liquid" is a lot more appealing than "green fungus growing on bread." A mold for casting objects is also a lot closer, metaphorically speaking, to what a linker does. I honestly thought that was the meaning the author was trying to evoke before I saw the picture on the github page.

Would emulsifier roll off the tongue better?

Re: Mold: A Modern Linker

#16

Post by rui314 »

spijdar wrote: post #11
klodolph wrote: post #6

Earlier quoted context omitted.

If you think this approach is wrong, could you articulate the reasons why you think it is wrong? This is a classic memory management strategy... if a program is running as a batch program, all memory will be freed when the program exits. Any alternative memory management strategy would have to free and then reuse memory in order to show improvement. If it's a small amount of memory freed, or if the memory is unlikely…

Yeah, this is also the strategy GCC and co use generally AFAIK. In a program like GCC where a single invocation will operate over a single file/unit, there's just not much benefit to trying to re-use data; if GCC or LLVM were closer to "build servers" with persistent state that compiled and linked objects on demand then it'd make sense, but in their current model, it's easier and safer to just keep data around.

Another classic example is Apache's memory pool. In Apache, you allocate memory from memory pools associated with the current request or connection. Memory pools are freed as a whole when a request or a connection is complete. mold's memory management scheme is not very different from that if you think the entire linker as a single "session" which uses a single memory pool.

Re: Mold: A Modern Linker

#17

Post by saagarjha »

It’s interesting to see where the bottlenecks are. Some of these are clearly algorithmic and many are solved with simplifying the work or using concurrency, but a lot are platform-specific and nonobvious.

Re: Mold: A Modern Linker

#18

Post by rui314 »

jdblair wrote: post #13

From a marketing perspective, "mold" meaning "a form used to cast an object from liquid" is a lot more appealing than "green fungus growing on bread." A mold for casting objects is also a lot closer, metaphorically speaking, to what a linker does. I honestly thought that was the meaning the author was trying to evoke before I saw the picture on the github page.

Author here. Haha, that's perhaps true. But at the same time, it seems like a tradition to give a silly name (e.g. "git") to a tool, and I actually like that name and the image to show that I'm not too serious. This is a fun project but not ready for production use.

Re: Mold: A Modern Linker

#19

Post by _dh54 »

chubot wrote: post #10
CyberRabbi wrote: post #2

> I won't avoid Unix-ism when writing code (e.g. I'll probably use fork(2)). It’s probably fine to not avoid most Unix APIs but fork() is truly an exception here. Fork() is not friendly to any third party library you may use because their state may become invalidated after a fork but the library has no way to know if the process has been forked. This is especially bad if the library uses multi threading. The best way…

I can't think of any reason a linker needs to use many libraries, especially not one that start threads. A linker reads files and writes a file. It's a single-purpose tool, not a monolithic app with many dependencies. Either way, threads should be controlled by the application and not libraries. Well-written libraries like sqlite and Lua are parameterized by I/O and concurrency. They don't read files and start thread…

I agree with you on both points but when building large complex applications (or tools!) this stuff tends to unintentionally creep up over time and it’s not fun to debug non-deterministic bugs due to an opaquely broken implicit contract between you and a poorly written library that got pulled in transitively. Sometimes those libraries are unavoidable closed source system libraries (https://bugs.python.org/issue33725) and the issue ends up being not easily fixable. It’s just risk minimization.

Re: Mold: A Modern Linker

#20

Post by [deleted] »

nothasan wrote: post #15
jdblair wrote: post #13

From a marketing perspective, "mold" meaning "a form used to cast an object from liquid" is a lot more appealing than "green fungus growing on bread." A mold for casting objects is also a lot closer, metaphorically speaking, to what a linker does. I honestly thought that was the meaning the author was trying to evoke before I saw the picture on the github page.

Would emulsifier roll off the tongue better?

[deleted]
Post reply on HN