Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The C-Shaped Hole in Package Management

nesbitt.io

Post reply on HN
Print view View original

31–40 of 77 posts

Re: The C-Shaped Hole in Package Management

#31

Post by amiga386 »

duped wrote: post #23
rwmj wrote: post #2

Please don't. C packaging in distros is working fine and doesn't need to turn into crap like the other language-specific package managers. If you don't know how to use pkgconf then that's your problem.

> C packaging in distros is working fine GLIBC_2.38 not found

If you have a distro-supplied binary that doesn't link with the distro-supplied glibc, something is very very wrong.

If you're supplying your own binaries and not compiling/linking them against the distro-supplied glibc, that's on you.

Re: The C-Shaped Hole in Package Management

#33

Post by conorbergin »

I use a lot of obscure libraries for scientific computing and engineering. If I install it from pacman or manage to get an AUR build working, my life is pretty good. If I have to use a Python library the faff becomes unbearable, make a venv, delete the venv, change python version, use conda, use uv, try and install it globally, change python path, source .venv/bin/activate. This is less true for other languages with local package management, but none of them are as frictionless as C (or Zig which I use mostly). The other issue is .venvs, node_packages and equivalents take up huge amounts of disk and make it a pain to move folders around, and no I will not be using a git repo for every throwaway test.

Re: The C-Shaped Hole in Package Management

#34

Post by pif »

duped wrote: post #24

Missing in this discussion is that package management is tightly coupled to module resolution in nearly every language. It is not enough to merely install dependencies of given versions but to do so in a way that the language toolchain and/or runtime can find and resolve them. And so when it comes to dynamic dependencies (including shared libraries) that are not resolved until runtime you hit language-level constrain…

> And if you've ever dug into the guts of glibc's loader it's 40 years of unreadable cruft.

You meant: it's 40 years of debugged and hardened run-everywhere never-fails code, I suppose.

Re: The C-Shaped Hole in Package Management

#35

Post by auxym »

conorbergin wrote: post #33

I use a lot of obscure libraries for scientific computing and engineering. If I install it from pacman or manage to get an AUR build working, my life is pretty good. If I have to use a Python library the faff becomes unbearable, make a venv, delete the venv, change python version, use conda, use uv, try and install it globally, change python path, source .venv/bin/activate. This is less true for other languages with…

uv has mostly solved the python issue. IME it's dependency resolution is fast and just works. Packages are hard linked from a global cache, which also greatly reduces storage requirements when you work with multiple projects.

Re: The C-Shaped Hole in Package Management

#36

Post by tpoacher »

zbentley wrote: post #12
hliyan wrote: post #7

Earlier quoted context omitted.

When I used to work with C many years ago, it was basically: download the headers and the binary file for your platform from the official website, place them in the header/lib paths, update the linker step in the Makefile, #include where it's needed, then use the library functions. It was a little bit more work than typing "npm install", but not so much as to cause headaches.

What do you do when the code you downloaded refers to symbols exported by libraries not already on your system? How do you figure out where those symbols should come from? What if it expects version-specific behavior and you’ve already installed a newer version of libwhatever on your system (I hope your distro package manager supports downgrades)? These are very, very common problems; not edge cases. Put another way:…

You are conflating development with distribution of binaries (a problem which interpreted languages do not have, I hasten to add).

1. The accepted solution to what you're describing in terms of development, is passing appropriate flags to `./configure`, specifying the path for the alternative versions of the libraries you want to use. This is as simple as it gets.

As for where to get these libraries from in the event that the distro doesn't provide the right version, `./configure` is basically a script. Nothing stopping you from printing a couple of ftp mirrors in the output to be used as a target to wget.

2. As for the problem of distribution of binaries and related up-to-date libraries, the appropriate solution is a distro package manager. A c package manager wouldn't come into this equation at all, unless you wanted to compile from scratch to account for your specific circumstances, in which case, goto 1.

Re: The C-Shaped Hole in Package Management

#37

Post by amluto »

auxym wrote: post #35
conorbergin wrote: post #33

I use a lot of obscure libraries for scientific computing and engineering. If I install it from pacman or manage to get an AUR build working, my life is pretty good. If I have to use a Python library the faff becomes unbearable, make a venv, delete the venv, change python version, use conda, use uv, try and install it globally, change python path, source .venv/bin/activate. This is less true for other languages with…

uv has mostly solved the python issue. IME it's dependency resolution is fast and just works. Packages are hard linked from a global cache, which also greatly reduces storage requirements when you work with multiple projects.

uv does nothing to help when you have old, crappy, barely maintained Python packages that don’t work reliably.

Re: The C-Shaped Hole in Package Management

#38

Post by duped »

pif wrote: post #34
duped wrote: post #24

Missing in this discussion is that package management is tightly coupled to module resolution in nearly every language. It is not enough to merely install dependencies of given versions but to do so in a way that the language toolchain and/or runtime can find and resolve them. And so when it comes to dynamic dependencies (including shared libraries) that are not resolved until runtime you hit language-level constrain…

> And if you've ever dug into the guts of glibc's loader it's 40 years of unreadable cruft. You meant: it's 40 years of debugged and hardened run-everywhere never-fails code, I suppose.

No, I meant 40 years of unreadable cruft. It's not hard to write a correct loader. It's very hard to understand glibc's implementation.

Re: The C-Shaped Hole in Package Management

#39

Post by duped »

amiga386 wrote: post #31
duped wrote: post #23

Earlier quoted context omitted.

> C packaging in distros is working fine GLIBC_2.38 not found

If you have a distro-supplied binary that doesn't link with the distro-supplied glibc, something is very very wrong. If you're supplying your own binaries and not compiling/linking them against the distro-supplied glibc, that's on you.

Linking against every distro-supplied glibc to distribute your own software is as unrealistic as getting distributions to distribute your software for you. The model is backwards from what users and developers expect.

But that's not the point I'm making. I'm attacking the idea that they're "working just fine" when the above is a bug that nearly everyone hits in the wild as a user and a developer shipping software on Linux. It's not the only one caused by the model, but it's certainly one of the most common.

Re: The C-Shaped Hole in Package Management

#40

Post by bengarney »

JohnFen wrote: post #17
zbentley wrote: post #12

Earlier quoted context omitted.

What do you do when the code you downloaded refers to symbols exported by libraries not already on your system? How do you figure out where those symbols should come from? What if it expects version-specific behavior and you’ve already installed a newer version of libwhatever on your system (I hope your distro package manager supports downgrades)? These are very, very common problems; not edge cases. Put another way:…

> These are very, very common problems; not edge cases. Honestly? Over the course of my career, I've only rarely encountered these sorts of problems. When I have, they've come from poorly engineered libraries anyway.

Here is a thought experiment (for devs who buy into package managers). Take the hash of a program and all its dependency. Behavior is different for every unique hash. With package managers, that hash is different on every system, including hashes in the future that are unknowable by you (ie future "compatible" versions of libraries).

That risk/QA load can be worth it, but is not always. For an OS, it helps to be able to upgrade SSL (for instance).

In my use cases, all this is a strong net negative. npm-base projects randomly break when new "compatible" version of libraries install for new devs. C/C++ projects don't build because of include/lib path issues or lack of installation of some specific version or who knows what.

If I need you to install the SDL 2.3.whatever libraries exactly, or use react 16.8.whatever to be sure the app runs, what's the point of using a complex system that will almost certainly ensure you have the wrong version? Just check it in, either by an explicit version or by committing the library's code and building it yourself.

Post reply on HN