Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oasis – a small, statically-linked Linux system

github.com

Post reply on HN
Print view View original

261–270 of 288 posts

Re: Oasis – a small, statically-linked Linux system

#261

Post by marwis »

arghwhat wrote: post #234
marwis wrote: post #209

Earlier quoted context omitted.

> The case for the shared library having better caching implies multiple processes that are distinct executables But this is the most common case for desktops/multipurpose systems. On my desktop there are tens or hundreds distinct processes sharing most of their code.

No it is not. Depending on your CPU, you might have, say, 32KB of 8-way associative instruction cache per core. Just being shared does not make it fit in the cache. A shared library would only be there across processes of different executable images if its users primarily , continuously execute the same paths in shared libs rather than anything unique in their own executable image - e.g., they'd more or less need to…

> A shared library would only be there across processes of different executable images if its users primarily, continuously execute the same paths in shared libs rather than anything unique in their own executable image

Yes, like libc/WinApi, Cairo/whatever graphics library of your OS, Gtk/Qt etc

All of the apps on my desktop (I don't use electron) share vast majority of their code (see above) and spend most of their time drawing UI through shared libs or doing their own business logic but that's part of their code and not subject to shared lib overhead.

Re: Oasis – a small, statically-linked Linux system

#262

Post by charcircuit »

YoshiRulz wrote: post #235
charcircuit wrote: post #150

Earlier quoted context omitted.

>Executables do list their dependencies They list paths to libraries, but not the exact version that the executable depends on. It is a common occurrence for executables to load versions of libraries they were not designed to be used with.

If you're talking about ELF for desktop Linux, they for the most part don't contain file paths, and may specify the version but usually just have the major version (to allow for security updates). You can use ldd to read the list of deps and also do a dry run of fulfilling them from the search path, for example: $> ldd $(command -v ls) linux-vdso.so.1 (0x00007ffd5b3a0000) libcap.so.2 => /usr/lib/libcap.so.2 (0x00007f…

Libraries can cause bugs even if they have the same exact version as it may be compiled in a way that is not expected by the program. Ideally the list of libraries should be some form of a hash of the library to ensure it is loading exactly what it expects.

Re: Oasis – a small, statically-linked Linux system

#263

Post by dieortin »

colatkinson wrote: post #200
dijit wrote: post #4

I cant speak much about the system, it just works, but the community was really nice when I interacted with them over IRC I had the plan to build oasis with bazel for some immutable OS images that could run as kubernetes nodes. I succeeded with a little pointing.

If you don't mind I'm super curious as to what approach you ended up taking. Did you use rules_foreign_cc to build the ninja files they generate? Or generating BUILD files directly? Or something completely different? Sounds like a really cool project!

Same, I’m curious too!

Re: Oasis – a small, statically-linked Linux system

#264

Post by arghwhat »

marwis wrote: post #261
arghwhat wrote: post #234

Earlier quoted context omitted.

No it is not. Depending on your CPU, you might have, say, 32KB of 8-way associative instruction cache per core. Just being shared does not make it fit in the cache. A shared library would only be there across processes of different executable images if its users primarily , continuously execute the same paths in shared libs rather than anything unique in their own executable image - e.g., they'd more or less need to…

> A shared library would only be there across processes of different executable images if its users primarily, continuously execute the same paths in shared libs rather than anything unique in their own executable image Yes, like libc/WinApi, Cairo/whatever graphics library of your OS, Gtk/Qt etc All of the apps on my desktop (I don't use electron) share vast majority of their code (see above) and spend most of their…

It's really important to emphasize how small caches are (especially as even the KBs they have cannot always be fully utilized). When we talk about processor caches, we're trying to make the current routine of one process fit well* - not even the whole thing.

No, you don't have a desktop environment where a majority of your unique executables all spend time at the same time in the same small, compute-heavy libc routine with no other processes to trash the cache in between.

For Gtk for example, the applications are not taking the same inputs and drawing the same GUI at the same time, with no other processes in between to wipe the cache.

Instead they're primarily running their own application logic in their own time, and interaction with Gtk (accepting input, rendering) is on timescales so long that the cache has been wiped out over and over in between (16ms is practically infinity at these scales). In these cases, the cache will be filled at the time of execution with e.g. that apps runtime data, widget tree and what not.

At the same time, remember that a static routine is much smaller and faster. Even if way say you fetch a shared library routine from cache and save some cycles there, every single call to it incurs large performance hits over the static linkage: a few cycles to every call from the PLT, possibly many cycles from poorer optimization (e.g., branches you never need), and cycles from not fitting as well in the tiny instruction cache as the process executes it.

... And if that shared lib routine calls other shared libs, then you get to apply the overheads and lost optimization recursively. This recursion where the static linkage pruning is especially effective: you might have tens or hundreds of megabytes of dependencies dynamically, but single megabyte static output as unused functionality is pruned.

There is no realistic scenario where dynamic linkage wins on performance or cacheability - it just doesn't play well with how caches end up used. Overall system memory utilization can be slightly lower for dynamic linkage in some cases, but not in a notable way.

Re: Oasis – a small, statically-linked Linux system

#265

Post by arghwhat »

inkyoto wrote: post #213
arghwhat wrote: post #203

Earlier quoted context omitted.

Static linking gives you better instruction cache utilization as you are executing local code linearly rather than going through indirection with more boilerplate. This indirection costs a few cycles too. Inlining external code reduces the size not only by saving the call, PLT and and stack dance, but also through specialization (removal of unused conditional, pruning of no longer referenced symbols) as the code is l…

> Static linking gives you better instruction cache utilization as you are executing local code linearly rather than going through indirection with more boilerplate. No, it does not, it worsens it. For example, «strlen», if it comes from a dynamic library, will be loaded into the physical memory once and only once, and it will be mapped into each process's address space as many times as there are processes. Since «st…

The compiler has built-ins for parts of libc exactly because dynamic linkage is ridiculous for performance, but they cannot statically link with a dynamic libc. It's a hack to make dynamic linked libc have at least somewhat acceptable performance.

If your libc was statically linked, you would not need the built-in - the strlen impl from your libc would get inlined by LTO.

The chances of a particular routine being in L1 is absolutely miniscule - it's hard enough to keep a single process and it's data in L1 and L2. What might happen is that you find it in L3, but: 1. The code you're loading is now much larger (fitting less well in L1 so you'll get more L1 misses) and slower (cache aside, it has redirection overhead and has not been LTO'd for this use), and 2. The inlined version would probably also be found in L3 - either resident or prefetched as that section if the process executable obviously had to be loaded to switch to it. 3. Unless the system is idle, the cache will be trashed in between process switches by the loads from other processes.

So while you could technically have a case where the shared lib is in cache, I do not think a realistic scenario exists where that setup wins out. There are more distinct pages, but the pages didn't fit in the first place: by having each process access fewer pages overall it can miss less while it is running.

Re: Oasis – a small, statically-linked Linux system

#266

Post by marwis »

arghwhat wrote: post #264
marwis wrote: post #261

Earlier quoted context omitted.

> A shared library would only be there across processes of different executable images if its users primarily, continuously execute the same paths in shared libs rather than anything unique in their own executable image Yes, like libc/WinApi, Cairo/whatever graphics library of your OS, Gtk/Qt etc All of the apps on my desktop (I don't use electron) share vast majority of their code (see above) and spend most of their…

It's really important to emphasize how small caches are (especially as even the KBs they have cannot always be fully utilized). When we talk about processor caches, we're trying to make the current routine of one process fit well* - not even the whole thing. No, you don't have a desktop environment where a majority of your unique executables all spend time at the same time in the same small, compute-heavy libc routin…

I'll give you a scenario where dynamic linkage is a clear win.

I have 16GB of RAM and usually when working all of it is in use. If everything was compiled statically I would get massive swapping.

Re: Oasis – a small, statically-linked Linux system

#267

Post by mayli »

liampulles wrote: post #36
kentonv wrote: post #20

Doesn't linking everything statically imply that the base image -- and memory, at runtime -- will be bloated by many copies of libc and other common libraries? I do like the simplicity of static linking but it sort of seems to go against the idea of avoiding "bloat".

It would be bloated, but how big of a problem is that these days? A TB of storage is pretty cheap.

Imagine base windows install requires 1TB of storage.

Re: Oasis – a small, statically-linked Linux system

#268

Post by hn_go_brrrrr »

notfed wrote: post #205
hn_go_brrrrr wrote: post #87

Earlier quoted context omitted.

Got a link? Sounds interesting.

Here's a link to a recent HN discussion: https://news.ycombinator.com/item?id=38852616 TLDR: Linux kernel doesn't have a stable binary kernel interface. And they don't want one. Given this, the definition of "reproducible build" needs, well, a refined definition, if it includes the Linux kernel. [1] https://www.kernel.org/doc/Documentation/process/stable-api-...

I don't think these are mutually incompatible. Reproducible builds means the same sources will produce the same results when built twice. The kernel's lack of stability guarantees refers to its evolution over time.

Re: Oasis – a small, statically-linked Linux system

#269

Post by soulofmischief »

TimeBearingDown wrote: post #248
soulofmischief wrote: post #170

Earlier quoted context omitted.

Same, I love Qubes' philosophy and UX, but GPU passthrough support was a dealbreaker in the end and I switched to a KVM system.

I’m pretty sure GPU passthrough does work in Qubes HVMs, although I haven’t tried it myself. Here are three quick and recent tutorials I found including one with a newer VirtualGL approach that offloads work instead of passing the entire card. https://neowutran.ovh/qubes/articles/gaming_windows_hvm.html https://forum.qubes-os.org/t/nvidia-gpu-passthrough-into-lin... https://forum.qubes-os.org/t/seamless-gpu-passthrou…

Does this fix the Code 14 issue with NVIDIA cards? That is why I had to switch to KVM back in 2016, as KVM has support for bypassing NVIDIA's "bug" which prevents using consumer cards in a virtual environment. I have been away from Qubes for 7 years now so I'd hope some form of improvement has been made.

Re: Oasis – a small, statically-linked Linux system

#270

Post by helloimhonk »

Rochus wrote: post #70
helloimhonk wrote: post #58

Earlier quoted context omitted.

cproc supports C11, tcc only goes up to c99. There is also something to be said for cproc using QBE which is slowly growing backends like risc-v etc which tcc doesnt support afaik.

Ok, thanks, that makes sense. QBE looks interesting, but I'm missing 32 bit support. So currently I'm trying to reuse the TCC backend, which is far from trivial.

There seems to have been a little progress on that recently, I saw someone working on a ppc 32bit port here: https://bsd.network/@tobhe/111756322928965195 and I also saw somewhere someone working on a 68000 port, but I don't remember where.

I'd like an arm 32bit port, which might be a bit easier with this ppc port as an example of 32bit qbe. It'd be nice to try run some hare programs on a gameboy advance :)

Post reply on HN