Live data from Hacker News

Oasis – a small, statically-linked Linux system

github.com

201–210 of 288 posts

Re: Oasis – a small, statically-linked Linux system

#201
post #73

Earlier quoted context omitted.

This alone “musl maintainers not even agreeing it’s a problem” should be a good reason to avoid musl imho

What's a better option for static linking? glibc is religiously against it; they have far worse dogmatic beliefs than this musl DNS thing. I'd be happy to choose a better alternative if one exists, but if one does not, I have to live with the options at hand. From where I'm standing, musl seems like the only game in town. uClibc doesn't seem like it's appropriate for general purpose Linux applications on desktop comp…

> glibc is religiously against it

from my understanding glibc is not "religiously" against it, they're against it for technical reasons. In the sense, this is not a dogma. It's about internal details of their implementations.

See: https://stackoverflow.com/questions/57476533/why-is-statical...

Re: Oasis – a small, statically-linked Linux system

#202
post #20

Doesn't linking everything statically imply that the base image -- and memory, at runtime -- will be bloated by many copies of libc and other common libraries? I do like the simplicity of static linking but it sort of seems to go against the idea of avoiding "bloat".

I have seen this sort of statement on HN before. I am guessing that the persons who propagate this idea have never actually experimented with replacing dynamically-linked programs having numerous dependencies with statically-compiled ones. It's a theory that makes sense in the abstract, but they have not actually tested it.

Though it is not a goal of mine to save storage space by using static binaries, and I actually expect to lose space as a tradeoff, I have actually saved storage space in some cases by using static binaries. This comes from being able to remove libraries from /usr/lib. TBH, I am not exactly sure why this is the case. Perhaps in part because one might be storing large libraries containing significant numbers of functions that one's programs never use.

For me using static binaries works well. Even "common" libraries can be removed in some cases by using a multi-call/crunched binary like busybox. This might not work for everyone. I think much depends on what selection of programs the computer owner prefers. (Namely, the dependencies required by those programs.)

Re: Oasis – a small, statically-linked Linux system

#203
post #133

Earlier quoted context omitted.

Static linked binaries are a generally lot smaller than a dynamically linked library and its dependencies, especially with link-time optimizations and inlining. You wouldn't want have 100 tools statically link the entirety of chromium, but for normal C library sizes you don't get bloat. The preference for dynamic libraries in Linux distros is just so they can roll out patch updates in one place instead of rebuilding…

But dynamically linked library only needs to be loaded to RAM once whereas with static linking you'd be loading the same code many times (unless you compile everything to single binary like BusyBox). This also gets you better cache utilization. Also I think inlining would typically increase the total size of output rather than decrease it.

Static linking gives you better instruction cache utilization as you are executing local code linearly rather than going through indirection with more boilerplate. This indirection costs a few cycles too.

Inlining external code reduces the size not only by saving the call, PLT and and stack dance, but also through specialization (removal of unused conditional, pruning of no longer referenced symbols) as the code is locally optimized. This further reduction in size further improves cache behavior and performance.

Duplication can be an issue (not necessarily for performance, but for total binary size), but compilers have heuristics for that. Even just having the symbol local saves some space and call overhead though (no PLT).

The case for the shared library having better caching implies multiple processes that are distinct executables (otherwise they share program memory regardless of linkage) trying to hammer it at once, sharing the continued caching, but such scenario is hurt by the call overhead and lower optimization opportunities, negating the result.

Re: Oasis – a small, statically-linked Linux system

#204

Earlier quoted context omitted.

Dynamic linking served us when OS upgrades came infrequently, user software was almost never upgraded short of mailing out new disks, and vendors had long lead times to incorporate security fixes. In the days of fast networks, embedded OSs, emphemeral containers, and big hard drives, a portable static binary is way less complex and only somewhat less secure (unless you're regularly rebuilding your containers/execs in…

As far as I can see, it would be unwise to roll back 30 years of (Linux) systems building with dynamic linking in favor of static linking. It mostly works very well and does save some memory, disk, and has nice security properties. Both have significant pros and cons. I've been thinking (not a Linux expert by any means) the ideal solution would be to have better dependency management: I think a solution could be if s…

Linus Torvalds disagrees: https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...

Re: Oasis – a small, statically-linked Linux system

#205
post #81

> Fast builds that are 100% reproducible. It's unclear to me what "100%" refers to here, but surely it does not include the Linux kernel or drivers? (I've recently read conversations about how difficult this would be.)

Got a link? Sounds interesting.

Here's a link to a recent HN discussion:

https://news.ycombinator.com/item?id=38852616

TLDR: Linux kernel doesn't have a stable binary kernel interface. And they don't want one.

Given this, the definition of "reproducible build" needs, well, a refined definition, if it includes the Linux kernel.

[1] https://www.kernel.org/doc/Documentation/process/stable-api-...

Re: Oasis – a small, statically-linked Linux system

#206
post #99

Earlier quoted context omitted.

> In the days of fast networks, embedded OSs, emphemeral containers, and big hard drives, a portable static binary is way less complex and only somewhat less secure If what you're trying to do is run a single program on a server somewhere, then yes absolutely a static binary is the way to go. There are lots of cases, especially end user desktops, where this doesn't really apply though. In my opinion the debate over s…

It applies very much to end user desktops as well, with snap, flatpak, etc. working towards it. Lots of software requires dependencies that aren't compatible with each other and result in absolute dependency hell or even a broken install when you dare to have more than one version of something. Because who would ever need that, right? Especially not in a dev desktop environment... Windows is basically all self-contai…

It should be noted though that flatpaks and related solutions are NOT equivalent to static linking. They do a lot more and serve a wildly different audience than something like Oasis. They are really much too extreme for non-GUI applications, and I would question the competence of anybody found running ordinary programs packaged in that manner.

I recognize that you probably weren't confused on this I'm just clarifying for others since the whole ecosystem can be a bit confusing.

Re: Oasis – a small, statically-linked Linux system

#207
post #115

Earlier quoted context omitted.

Why did you need to use bazel?

I didnt need to use bazel, I like bazel and want to learn more about it. I also have a small, but burning, passion for reproducible builds, distributed compilation and distributed caching. Being able to build an entire OS and essentially anything I want on top in a reproducible and relatively organic way (with incremental compilation) is pretty dope.

You should check out the ChromeOS Bazelification project[1]. It has those exact same goals. Not all packages are reproducible though because they embed timestamps.

[1]: https://chromium.googlesource.com/chromiumos/bazel/+/HEAD/do...

Re: Oasis – a small, statically-linked Linux system

#208
post #48
post #20

Doesn't linking everything statically imply that the base image -- and memory, at runtime -- will be bloated by many copies of libc and other common libraries? I do like the simplicity of static linking but it sort of seems to go against the idea of avoiding "bloat".

A linker typically only includes the parts of the library it needs for each binary so some parts will definately have many copies of the same code when you statically link but it will not make complete copies. But I wouldnt consider this bloat. To me it is just a better seperation of concerns. To me bloat would be to have a system that has to keep track of all library dependencies instead, both from a packaging persp…

> A linker typically only includes the parts of the library it needs for each binary […]

It is exactly the same with the dynamic linking due to the demand paging available in all modern UNIX systems: the dynamic library is not loaded into memory in its entirety, it is mapped into the process's virtual address space.

Initially, there is no code from the dynamic library loaded into memory until the process attempts to access the first instruction from the required code at which point a memory fault occurs, and the virtual memory management system loads the required page(s) into the process's memory. A dynamic library can be 10Gb in size and appear as a 10Gb in the process's memory map but only 1 page can be physically present in memory. Moreover, under the heavy memory pressure the kernel can invalidate the memory page(s) (using LRU or a more advanced memory page tracking technique) and the process (especially true for background or idlying processes) will reference zero pages with the code from the dynamic library.

Fundamentally, dynamic linking is the deferred static linking where the linking functions are delegated to the dynamic library loader. Dynamic libraries incur a [relatively] small overhead of slower (compared to statically linked binaries) process startup times due to the dynamic linker having to load the symbol table, the global offset table from the dynamic library and performing the symbol fixup according to the process's own virtual memory layout. It is a one-off step, though. For large, very large and frequently used dynamic libraries, caching can be employed to reduce such overhead.

Dynamic library mapping into the virtual address space != loading the dynamic library into memory, they are two disjoint things. It almost never happens when the entire dynamic library is loaded into memory as the 100% code coverage is exceedingly rare.

Re: Oasis – a small, statically-linked Linux system

#209
post #133

Earlier quoted context omitted.

But dynamically linked library only needs to be loaded to RAM once whereas with static linking you'd be loading the same code many times (unless you compile everything to single binary like BusyBox). This also gets you better cache utilization. Also I think inlining would typically increase the total size of output rather than decrease it.

Static linking gives you better instruction cache utilization as you are executing local code linearly rather than going through indirection with more boilerplate. This indirection costs a few cycles too. Inlining external code reduces the size not only by saving the call, PLT and and stack dance, but also through specialization (removal of unused conditional, pruning of no longer referenced symbols) as the code is l…

> The case for the shared library having better caching implies multiple processes that are distinct executables

But this is the most common case for desktops/multipurpose systems.

On my desktop there are tens or hundreds distinct processes sharing most of their code.

Re: Oasis – a small, statically-linked Linux system

#210
post #48

Earlier quoted context omitted.

A linker typically only includes the parts of the library it needs for each binary so some parts will definately have many copies of the same code when you statically link but it will not make complete copies. But I wouldnt consider this bloat. To me it is just a better seperation of concerns. To me bloat would be to have a system that has to keep track of all library dependencies instead, both from a packaging persp…

>A linker typically only includes the parts of the library it needs for each binary so some parts will definately have many copies of the same code when you statically link but it will not make complete copies. Just to add to what you said: in the old days the linker would include only the .o files in the .a library that were referenced. Really common libraries like libc should be made to have only a single function…

> Just to add to what you said: in the old days the linker would include only the .o files in the .a library that were referenced.

It was not exactly like that. Yes, the .o file granularity was there but the unused code from that .o file would also get linked in.

The original UNIX linker had a very simple and unsophisticated design (compared to its contemporaries) and would not attempt to optimise the final product being linked. Consider a scenario where the binary being linked references A from an «abcde.o» file, and the «abcde.o» file has A, B, C, D and E defined in it, so the original «ld» would link the entire «abcde.o» into the final product. Advanced optimisations came along much later on.

Post reply on HN