Live data from Hacker News

The C-Shaped Hole in Package Management

nesbitt.io

41–50 of 77 posts

Re: The C-Shaped Hole in Package Management

#41
post #35

I use a lot of obscure libraries for scientific computing and engineering. If I install it from pacman or manage to get an AUR build working, my life is pretty good. If I have to use a Python library the faff becomes unbearable, make a venv, delete the venv, change python version, use conda, use uv, try and install it globally, change python path, source .venv/bin/activate. This is less true for other languages with…

uv has mostly solved the python issue. IME it's dependency resolution is fast and just works. Packages are hard linked from a global cache, which also greatly reduces storage requirements when you work with multiple projects.

uv is great for resolution, but it seems like it doesn't really address the build complexity for heavy native dependencies. If you are doing any serious work with torch or local LLMs, you still run into issues where wheels aren't available for your specific cuda/arch combination. That is usually where I lose time, not waiting for the resolver.

Re: The C-Shaped Hole in Package Management

#42
post #35

I use a lot of obscure libraries for scientific computing and engineering. If I install it from pacman or manage to get an AUR build working, my life is pretty good. If I have to use a Python library the faff becomes unbearable, make a venv, delete the venv, change python version, use conda, use uv, try and install it globally, change python path, source .venv/bin/activate. This is less true for other languages with…

uv has mostly solved the python issue. IME it's dependency resolution is fast and just works. Packages are hard linked from a global cache, which also greatly reduces storage requirements when you work with multiple projects.

You still need to compile when those libraries are not pre compiled.

Re: The C-Shaped Hole in Package Management

#43

I use a lot of obscure libraries for scientific computing and engineering. If I install it from pacman or manage to get an AUR build working, my life is pretty good. If I have to use a Python library the faff becomes unbearable, make a venv, delete the venv, change python version, use conda, use uv, try and install it globally, change python path, source .venv/bin/activate. This is less true for other languages with…

compiling an open source C project isn't time consuming?

Re: The C-Shaped Hole in Package Management

#44
post #6
post #2

Please don't. C packaging in distros is working fine and doesn't need to turn into crap like the other language-specific package managers. If you don't know how to use pkgconf then that's your problem.

I agree entirely. C doesn't need this. That I don't have to deal with such a thing has become a new and surprising advantage of the language for me.

I find this sentiment bewildering. Can you help me understand your perspective? Is this specifically C or C++? How do you manage a C/C++ project across a team without a package manager? What is your methodology for incorporating third party libraries?

I have spent the better half of 10 years navigating around C++'s deplorable dependency management story with a slurry of Docker and apt, which had better not be part of everyone's story about how C is just fine. I've now been moving our team to Conan, which is also a complete shitshow for the reasons outlined in the article: there is still an imaginary line where Conan lets go and defers to "system" dependencies, with a completely half-assed and non-functional system for communicating and resolving those dependencies which doesn't work at all once you need to cross compile.

Re: The C-Shaped Hole in Package Management

#45
post #2

Please don't. C packaging in distros is working fine and doesn't need to turn into crap like the other language-specific package managers. If you don't know how to use pkgconf then that's your problem.

Well, if you're fine with using 3-year old versions of those libraries packaged by severely overworked maintainers who at one point seriously considered blindly converting everything into Flatpaks and shipping those simply because they can't muster enough of manpower, sure. "But you can use 3rd party repositories!" Yeah, and I also can just download the library from its author's site. I mean, if I trust them enough t…

If this is a concern (which it rarely is) then you can pitch in with distro packaging. Volunteers are always welcome.

> "But you can use 3rd party repositories!"

That's not something I said.

Re: The C-Shaped Hole in Package Management

#46
post #7

Earlier quoted context omitted.

When I used to work with C many years ago, it was basically: download the headers and the binary file for your platform from the official website, place them in the header/lib paths, update the linker step in the Makefile, #include where it's needed, then use the library functions. It was a little bit more work than typing "npm install", but not so much as to cause headaches.

And then you got some minor detail different from the compiled library and boom, UB because some struct is layed out differently or the calling convention is wrong or you compiled with a different -std or …

Which is exactly why you should leave it to the distros to construct a consistent build environment. If your distro regularly gets this wrong then you do have a problem.

Re: The C-Shaped Hole in Package Management

#47
post #2

Please don't. C packaging in distros is working fine and doesn't need to turn into crap like the other language-specific package managers. If you don't know how to use pkgconf then that's your problem.

I've contemplated this quite a bit (and I personally maintain a C++ artifact that I deploy to production machines, and I generally prefer not to use containers for it), and I think I disagree.

Distributions have solved a very specific problem quite nicely: they are building what is effectively one application (the distro) with many optional pieces, it has one set of dependencies, and the users update the whole thing when they update. If the distro wants to patch a dependency, it does so. ELF programs that set DT_INTERP to /lib/ld-linux-[arch].so.1 opt in to the distro's set of dependencies. This all works remarkably well and a lot of tooling has been built around it.

But a lot of users don't work in this model. We build C/C++ programs that have their own set of dependencies. We want to try patching some of them. We want to try omitting some. We want to write programs that are hermetic in the sense that we are guaranteed to notice if we accidentally depend on something that's actually an optional distro package. The results ... are really quite bad, unless the software you are building is built within a distro's build system.

And the existing tooling is terrible. Want to write a program that opts out of the distro's library path? Too bad -- DT_INTERP really really wants an absolute path, and the one and only interpreter reliably found at an absolute path will not play along. glibc doesn't know how to opt out of the distro's library search path. There is no ELF flag to do it, nor is there an environment variable. It doesn't even really support a mode where DT_INTERP is not used but you can still do dlopen! So you can't do the C equivalent of Python venvs without a giant mess.

pkgconf does absolutely nothing to help. Sure, I can write a makefile that uses pkgconf to find the distro's libwhatever, and if I'm willing to build from source on each machine* (or I'm writing the distro itself) and if libwhatever is an acceptable version* and if the distro doesn't have a problematic patch to it, then it works. This is completely useless for people like me who want to build something remotely portable. So instead people use enormous kludges like Dockerfile to package the entire distro with the application in a distinctly non-hermetic way.

Compare to solutions that actually do work:

- Nix is somewhat all-encompassing, but it can simultaneously run multiple applications with incompatible sets of dependencies.

- Windows has a distinct set of libraries that are on the system side of the system vs ISV boundary. They spend decades doing an admirable job of maintaining the boundary. (Okay, they seem to have forgotten how to maintain anything in 2026, but that's a different story.) You can build a Windows program on one machine and run it somewhere else, and it works.

- Apple bullies everyone into only targeting a small number of distros. It works, kind of. But ask people who like software like Aperture whether it still runs...

- Linux (the syscall interface, not GNU/Linux) outdoes Microsoft in maintaining compatibility. This is part of why Docker works. Note that Docker and all its relatives basically completely throw out the distro model of interdependent packages all with the same source. OCI tries to replace it with a sort-of-tree of OCI layers that are, in theory, independent, but approximately no one actually uses it as such and instead uses Docker's build system and layer support as an incredibly poorly functioning and unreliable cache.

- The BSDs are basically the distro model except with one single distro each that includes the kernel.

I would love functioning C virtual environments. Bring it on, please.

Re: The C-Shaped Hole in Package Management

#48
post #17

Earlier quoted context omitted.

> These are very, very common problems; not edge cases. Honestly? Over the course of my career, I've only rarely encountered these sorts of problems. When I have, they've come from poorly engineered libraries anyway.

Here is a thought experiment (for devs who buy into package managers). Take the hash of a program and all its dependency. Behavior is different for every unique hash. With package managers, that hash is different on every system, including hashes in the future that are unknowable by you (ie future "compatible" versions of libraries). That risk/QA load can be worth it, but is not always. For an OS, it helps to be able…

Check it in and build it yourself using the common build system that you and the third party dependency definitely definitely share, because this is the C/C++ ecosystem?

Re: The C-Shaped Hole in Package Management

#49
post #39

Earlier quoted context omitted.

If you have a distro-supplied binary that doesn't link with the distro-supplied glibc, something is very very wrong. If you're supplying your own binaries and not compiling/linking them against the distro-supplied glibc, that's on you.

Linking against every distro-supplied glibc to distribute your own software is as unrealistic as getting distributions to distribute your software for you. The model is backwards from what users and developers expect. But that's not the point I'm making. I'm attacking the idea that they're "working just fine" when the above is a bug that nearly everyone hits in the wild as a user and a developer shipping software on…

It's hardly unrealistic - most free software has been packaged, by each distro. Very handy for the developer: just email the distro maintainers (or post on your mailing list) that the new version is out, they'll get round to packaging it. Very handy for the user, they just "apt install foo" and ta-da, Foo is installed.

That was very much the point of using a Linux distro (the clue is in the name!) Trying to work in a Windows/macOS way where the "platform" does fuck-all and the developer has to do it all themselves is the opposite of how distros work.

Re: The C-Shaped Hole in Package Management

#50
post #45

Earlier quoted context omitted.

Well, if you're fine with using 3-year old versions of those libraries packaged by severely overworked maintainers who at one point seriously considered blindly converting everything into Flatpaks and shipping those simply because they can't muster enough of manpower, sure. "But you can use 3rd party repositories!" Yeah, and I also can just download the library from its author's site. I mean, if I trust them enough t…

If this is a concern (which it rarely is) then you can pitch in with distro packaging. Volunteers are always welcome. > "But you can use 3rd party repositories!" That's not something I said.

>(which it rarely is)

You're saying it's _rare_ for developers to want to advance a dependency past the ancient version contained in is?

Speaking for the robotics and ML space, that is simply the opposite of a true statement where I work.

Also doesn't your philosophy require me to figure out the packaging story for every separate distro, too? Do you just maintain multiple entirely separate dependency graphs, one for each distro? And then say to hell with Windows and Mac? I've never practiced this "just use the system package manager" mindset so I don't understand how this actually works in practice for cross-platform development.

Post reply on HN