Building Go programs with Nix flakes
xeiaso.net
Building Go programs with Nix flakes
1–10 of 26 posts
Re: Building Go programs with Nix flakes
#2Nice to have some development in this space. However I would be very interested in some discussion why this approach should be superior to goBuildModule, which is also mentioned in the article. goBuildModule works well for us and it is even arguably simpler to use, you just have to update the vendor hash whenever you updated the dependencies.
The only real downside for us is that it doesn't handle private dependencies well.
Re: Building Go programs with Nix flakes
#3Nice to have some development in this space. However I would be very interested in some discussion why this approach should be superior to goBuildModule, which is also mentioned in the article. goBuildModule works well for us and it is even arguably simpler to use, you just have to update the vendor hash whenever you updated the dependencies. The only real downside for us is that it doesn't handle private dependencie…
The main difference is that it's easier to track what changed in the vendor closure rather than just waiting for the build to fail and updating the hash with the result that Nix gives you.
Re: Building Go programs with Nix flakes
#4Nice to have some development in this space. However I would be very interested in some discussion why this approach should be superior to goBuildModule, which is also mentioned in the article. goBuildModule works well for us and it is even arguably simpler to use, you just have to update the vendor hash whenever you updated the dependencies. The only real downside for us is that it doesn't handle private dependencie…
The main difference is that it's easier to track what changed in the vendor closure rather than just waiting for the build to fail and updating the hash with the result that Nix gives you.
but how does it react when go.mod and the toml get out of sync? For us this will just break on CI and requires then a commit with the updated vendorSha to be fixed. I would expect the same workflow from this approach?
Re: Building Go programs with Nix flakes
#5Now if only there were a reliable way to do this with Python programs with a requirements.txt.
Last time I tried (maybe 2 months ago on NixOS 22.05), I found several different tools that claimed to be able to do this and they all failed in various ways; some failed quickly, others took a long time to fail. I ended up using my brain as an SAT solver and manually copying in nix expressions from older versions of nixpkgs that matched versions of the python libs that were needed.
Re: Building Go programs with Nix flakes
#6Now if only there were a reliable way to do this with Python programs with a requirements.txt. Last time I tried (maybe 2 months ago on NixOS 22.05), I found several different tools that claimed to be able to do this and they all failed in various ways; some failed quickly, others took a long time to fail. I ended up using my brain as an SAT solver and manually copying in nix expressions from older versions of nixpkg…
In all fairness, this is impossible with or without nix.
I've had some success with Poetry. But then I ran into programs that invoke pip during startup...
Re: Building Go programs with Nix flakes
#7Now if only there were a reliable way to do this with Python programs with a requirements.txt. Last time I tried (maybe 2 months ago on NixOS 22.05), I found several different tools that claimed to be able to do this and they all failed in various ways; some failed quickly, others took a long time to fail. I ended up using my brain as an SAT solver and manually copying in nix expressions from older versions of nixpkg…
Yeah the situation with Python is kind of crap to begin with. The main problem with requirements.txt is that it doesn't lock the SHAsums of the packages being depended on.
The Python ecosystem also doesn't seem to have a sensible solution for dependency declaration in general. It's kind of remarkable that anything works at all in Python land, it's a lot of "let's hope it works often enough". To be fair it does work out a lot of the time, but when that fails you end up coming to people like me.
I should do some research and see if I can figure out the best way to do this in Python, but I don't use Python at all personally or professionally.
Re: Building Go programs with Nix flakes
#8Nice to have some development in this space. However I would be very interested in some discussion why this approach should be superior to goBuildModule, which is also mentioned in the article. goBuildModule works well for us and it is even arguably simpler to use, you just have to update the vendor hash whenever you updated the dependencies. The only real downside for us is that it doesn't handle private dependencie…
The announcement post for gomod2nix has some additional info on the problems with buildGoModule's approach to vendorization:
> The buildGoModule package is designed around fixed-output derivations, which means that a single derivation is created where all the dependencies of the package you want to build are wrapped, and only a single hash of the derivation is specified. It fetches all dependencies in the fixed output, creating a vendor directory which is used for the build.
> This has several issues, most notably there is no sharing of dependencies between packages that depend on the same Go module.
> The other notable issue is that it forces developers to remember editing the vendorSha256 attribute separately from the already existing hash/sha256 attribute on the derivation. Forgetting to do so can not only lead to incorrect builds but also be frustrating when working with larger packages that takes a long time to build, and only very late in the build notice that something was broken so you have to start over from scratch.
> Because of the lack of hash granularity the build needs to clone every dependency every time the vendorSha256 is invalidated, and cannot use the cache from previous builds.
https://www.tweag.io/blog/2021-03-04-gomod2nix/
So you get greater granularity here, and thus also greater cache reuse between builds.
Re: Building Go programs with Nix flakes
#9Now if only there were a reliable way to do this with Python programs with a requirements.txt. Last time I tried (maybe 2 months ago on NixOS 22.05), I found several different tools that claimed to be able to do this and they all failed in various ways; some failed quickly, others took a long time to fail. I ended up using my brain as an SAT solver and manually copying in nix expressions from older versions of nixpkg…
Does https://pypi.org/project/pypi2nix/ work?
Now that pip supports pinning with a sha (I say “now that”, this was added years ago but it wasn’t always thus), it seems it should be trivial to map onto the Nix model.
I have been working with poetry and playing with poetry2nix so haven’t tried raw pip requirements files with Nix.
Re: Building Go programs with Nix flakes
#10Nice to have some development in this space. However I would be very interested in some discussion why this approach should be superior to goBuildModule, which is also mentioned in the article. goBuildModule works well for us and it is even arguably simpler to use, you just have to update the vendor hash whenever you updated the dependencies. The only real downside for us is that it doesn't handle private dependencie…
The announcement post for gomod2nix has some additional info on the problems with buildGoModule's approach to vendorization: > The buildGoModule package is designed around fixed-output derivations, which means that a single derivation is created where all the dependencies of the package you want to build are wrapped, and only a single hash of the derivation is specified. It fetches all dependencies in the fixed outpu…
Is there any reason this isn't just merged into the buildGoModule package? It seems like a net positive?