An alternative to shared libraries
1–10 of 24 posts
Re: An alternative to shared libraries
#2What fascinates me is how close this synthetic filesystem is to making REST calls with a small client library. All your application needs to know is the path to call and what HTTP verb to use. Even the way the author introduces versioning sounds familiar: using a versioning file is similar to having a version at the beginning of your URL.
I appreciate how good architectures all resemble each other at some point, with only the transport layers differing between applications.
Re: An alternative to shared libraries
#3I'm afraid that VFS-based APIs will be even more fragile as the life cycle goes, and even harder to reason about.
Re: An alternative to shared libraries
#4An excellent read. Moreover, most linkers these days have the ability to strip out unused code from your executable if you are using static linkage.
That means if your tiny executable uses just some methods of a gigantic library, it can do so an it will still stay tiny. Contrast that with dynamic linkage where in practice the whole gigantic library has to ship with your executable because you can never be sure that the host will already have the right version of libGigantic installed. What a mess.
Shared libraries are the reason a lot of app installations are several GB these days. And I dare say that 99.x % of the code that's shipped with an installation package never gets used by the app installed and that most compiled app code would comfortably fit on a couple of floppy disks if static linkage was used rigorously.
Re: An alternative to shared libraries
#5An excellent read. Moreover, most linkers these days have the ability to strip out unused code from your executable if you are using static linkage. That means if your tiny executable uses just some methods of a gigantic library, it can do so an it will still stay tiny. Contrast that with dynamic linkage where in practice the whole gigantic library has to ship with your executable because you can never be sure that t…
> if your tiny executable uses just some methods of a gigantic library, it can do so an it will still stay tiny
How many other functions do the ones you explicitly call drag into the executable? How do you propose to deduplicate them and their resources between processes? I'm specifically thinking about UI code. Having 30 different copies of your UI toolkit and its resources would be silly, even if each was stripped to 1/3 the size of the original. "Just use IPC and do UI in the window server" sounds an awful lot like X11, whose architecture we were just starting to migrate away from!
> Shared libraries are the reason a lot of app installations are several GB these days.
What are you referring to? vcredist is 7MB. Direct X runtimes are ~100MB. Qt is 20MB. Not small, but not the majority of several GB. Sure it's silly to ship an app with a shared library rather than statically link it, but I think you're exaggerating the scale of the problem and understating the number of profitable examples of library sharing, especially when it comes to UI libraries.
Re: An alternative to shared libraries
#6This just illustrates that the real problem is versioning, not linking. It's nice to hand wave off the versioning problems of this approach, but in practice this isn't really different from the mechanisms used to version shared libraries and suffers all the same pitfalls. You've just replaced ld.so with a bunch of running daemons and turned it into a distributed computing problem. Don't get me wrong, I love plan 9 and its namespaces and long for them in the practical world, but this is 6 of one, half dozen of the other.
Also, filesystem namespaces in Linux are a privileged operation, so these kinds of approaches don't work at all like they do in plan 9.
Re: An alternative to shared libraries
#7An excellent read. Moreover, most linkers these days have the ability to strip out unused code from your executable if you are using static linkage. That means if your tiny executable uses just some methods of a gigantic library, it can do so an it will still stay tiny. Contrast that with dynamic linkage where in practice the whole gigantic library has to ship with your executable because you can never be sure that t…
> if your tiny executable uses just some methods of a gigantic library, it can do so an it will still stay tiny How many other functions do the ones you explicitly call drag into the executable? How do you propose to deduplicate them and their resources between processes? I'm specifically thinking about UI code. Having 30 different copies of your UI toolkit and its resources would be silly, even if each was stripped…
Don't forget, X11 is the reason why Unix got dynamic libraries.
Re: An alternative to shared libraries
#8OS X never[1] seems to require more than 8 GB, no matter how many applications you have running - I've got 28 right now, including the Office Apps. Google Earth, Various Browsers, etc... and system is ticking along nicely. I've got to believe that is a testimony to the power of shared libraries.
[1] Yes, I'm aware there is an infinite range of work loads (Video, Audio, PhotoShop, Virtualization, Oracle, etc.. that can use up as much memory as you throw at it - I'm talking about the joe average user workloads here
Re: An alternative to shared libraries
#9Abstracting away the parts that are Plan-9-implementation-specific, this article seems to be advocating replacing shared libraries with remote procedure calls / a network API, or more fundamentally, calls that can cross address spaces. It's worth nothing that this was an approach that, as I understand it, predated the advent of shared libraries on UNIX. Terminal handling (termios) and the X Window System protocol both come to mind, and we've been slowly moving away from that at least for X (libGL, Wayland/Mir, etc.). It's also strongly reminiscent of Mach's approach of message-passing between daemons, which was a decent idea, but ultimately failed because of performance.
There are definitely advantages to address-space isolation: an unintentional mistake in one component is much less likely to affect the other, the two components can pull in conflicting versions of dependencies, etc. But versioning and ABI compatibility remain issues. I think this post briefly touches on the versioning problem and assumes that providing both the old and new version of the library-daemon would solve it: that's probably technically true, but you'd need to keep every version of the library around to avoid the problem of libc introducing bugs in the process of fixing other bugs (the only concrete problem mentioned here). So yes, there's definitely more flexibility to solve problems than in the current implementations of dynamic linkers, but the problems themselves remain hard.
Meanwhile, you've also introduced the difficult constraint that libraries have to operate on copies of all your data. The hypothetical crypto library here is copying every block of ciphertext over an inter-process call, decrypting it, and copying it back to the original program. Apart from making security folks generically twitchy at all the copies of secret data running around, this is going to be awful for performance. And each side either has to trust the other side not to be trying to exploit it (which reduces the benefits of address-space isolation), or verify the data structures' integrity (which makes things even slower). It's possible that with good implementations of cross-process shared memory and low-overhead, secure message encodings (like Cap'n Proto), you could make this better, but it'll be a bit of a project.
I'm happy to admit that the implementations of dynamic linking are all less than awesome. Fundamentally, there's no reason that you can't design a shared-library system with all of the properties in this design, including the ability to load two copies of the same library that differ only by minor version, to satisfy dependencies of two different components. Even the current GNU linker (which is not my favorite dynamic linker) supports symbol versioning, so it could offer both the GLIBC_2.18 and GLIBC_2.19 versions of a function in the same library, although this facility isn't used very much.
Re: An alternative to shared libraries
#10"If you think about it, if your code is small and clean, you wouldn’t feel the need for shared libraries."
Yes you would, for any sane definition of "small and clean". Code can't be made arbitrarily small; some problem spaces are fundamentally complex.
For what it's worth I have played around some with dynamic libraries and with FUSE, and I found the former incomparably easier to work with. Maybe that speaks more to FUSE in particular than to the idea in general (or maybe it's just me being bad at FUSE), but that's been my experience.