Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Go programs with Nix flakes

xeiaso.net

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Building Go programs with Nix flakes

#11

Post by drpyser22 »

xena wrote: post #7
aidenn0 wrote: post #5

Now if only there were a reliable way to do this with Python programs with a requirements.txt. Last time I tried (maybe 2 months ago on NixOS 22.05), I found several different tools that claimed to be able to do this and they all failed in various ways; some failed quickly, others took a long time to fail. I ended up using my brain as an SAT solver and manually copying in nix expressions from older versions of nixpkg…

Yeah the situation with Python is kind of crap to begin with. The main problem with requirements.txt is that it doesn't lock the SHAsums of the packages being depended on. The Python ecosystem also doesn't seem to have a sensible solution for dependency declaration in general. It's kind of remarkable that anything works at all in Python land, it's a lot of "let's hope it works often enough". To be fair it does work o…

What would you consider a sensible solution? What about

* Pip-tools(requirements.in + requirements.txt)

* Poetry (pyproject.toml+poetry.lock)

* Pipenv(Pipfile+Pipfile.lock)

Those and others enable easy declaration of direct dependencies and compile (resolve) a lock file of the actual dependency graph with hashes. Those tools have their issues and idiosyncrasies, but the dependency declaration aspect seems sensible and functional.

Re: Building Go programs with Nix flakes

#12

Post by ac130kz »

aidenn0 wrote: post #5

Now if only there were a reliable way to do this with Python programs with a requirements.txt. Last time I tried (maybe 2 months ago on NixOS 22.05), I found several different tools that claimed to be able to do this and they all failed in various ways; some failed quickly, others took a long time to fail. I ended up using my brain as an SAT solver and manually copying in nix expressions from older versions of nixpkg…

PEP 518 has been around for a long time. All of the projects must be running pyproject.toml + poetry.lock. Like really, it's the new modern standard, which solves a lot of headaches. The configs, the Python version, the dependencies, are all nicely crammed together. There's even a thing poetry2nix, which seems pretty well supported.

Re: Building Go programs with Nix flakes

#14

Post by tadfisher »

flicaflow wrote: post #4
xena wrote: post #3

Earlier quoted context omitted.

The main difference is that it's easier to track what changed in the vendor closure rather than just waiting for the build to fail and updating the hash with the result that Nix gives you.

but how does it react when go.mod and the toml get out of sync? For us this will just break on CI and requires then a commit with the updated vendorSha to be fixed. I would expect the same workflow from this approach?

The only reason they would get out of sync is if you updated dependency versions without also running gomod2nix. The benefit here is that you can do that without the intermediate step of first building the project and updating the hash after the build fails.

Re: Building Go programs with Nix flakes

#15

Post by lewo »

pxc wrote: post #8
flicaflow wrote: post #2

Nice to have some development in this space. However I would be very interested in some discussion why this approach should be superior to goBuildModule, which is also mentioned in the article. goBuildModule works well for us and it is even arguably simpler to use, you just have to update the vendor hash whenever you updated the dependencies. The only real downside for us is that it doesn't handle private dependencie…

The announcement post for gomod2nix has some additional info on the problems with buildGoModule's approach to vendorization: > The buildGoModule package is designed around fixed-output derivations, which means that a single derivation is created where all the dependencies of the package you want to build are wrapped, and only a single hash of the derivation is specified. It fetches all dependencies in the fixed outpu…

Another advantage of the gomod2nix approach concerns long term build reproducibility [1]. Having each source dependency in its own derivation would make easier source code mirroring. Since Nix knows the hash of each sources, we could imagine to teach Nix how to get the source code from some other places than GitHub. For instance, Nix could use the hash of the dependency to query Software Heritage [2].

Note there is also an opened issue to restrict Nix fixed output derivation which would make impossible the buildGoModule "hack".

[1] https://github.com/NixOS/nixpkgs/issues/84826

[2] https://www.tweag.io/blog/2020-06-18-software-heritage/

Re: Building Go programs with Nix flakes

#16

Post by sidlls »

jjtheblunt wrote: post #13

"go" the übertool is so well thought out and handy, I'm missing how this doesn't essentially add complexity.

Dependency management in go is…underwhelming to say the least. Go get? Go mod download? Go install? Go tool error messages related to the same? All a mess. This issue, implicit interfaces, and a CS-3/400 class project quality implementation of generics (harsh, I know) are three reasons that the language frustrates me to no end.

Re: Building Go programs with Nix flakes

#17

Post by aidenn0 »

theptip wrote: post #9
aidenn0 wrote: post #5

Now if only there were a reliable way to do this with Python programs with a requirements.txt. Last time I tried (maybe 2 months ago on NixOS 22.05), I found several different tools that claimed to be able to do this and they all failed in various ways; some failed quickly, others took a long time to fail. I ended up using my brain as an SAT solver and manually copying in nix expressions from older versions of nixpkg…

Does https://pypi.org/project/pypi2nix/ work? Now that pip supports pinning with a sha (I say “now that”, this was added years ago but it wasn’t always thus), it seems it should be trivial to map onto the Nix model. I have been working with poetry and playing with poetry2nix so haven’t tried raw pip requirements files with Nix.

If I were a better engineer, I'd have recorded a list of all the things I tried, but I didn't. Most of them ended in the substring "2nix" but the names kind of blur together so I can't definitively say I tried it. I have some free time on the 23rd and may see if pypi2nix works with the project or not, and file a bug if it doesn't.

Re: Building Go programs with Nix flakes

#18

Post by pxc »

lewo wrote: post #15
pxc wrote: post #8

Earlier quoted context omitted.

The announcement post for gomod2nix has some additional info on the problems with buildGoModule's approach to vendorization: > The buildGoModule package is designed around fixed-output derivations, which means that a single derivation is created where all the dependencies of the package you want to build are wrapped, and only a single hash of the derivation is specified. It fetches all dependencies in the fixed outpu…

Another advantage of the gomod2nix approach concerns long term build reproducibility [1]. Having each source dependency in its own derivation would make easier source code mirroring. Since Nix knows the hash of each sources, we could imagine to teach Nix how to get the source code from some other places than GitHub. For instance, Nix could use the hash of the dependency to query Software Heritage [2]. Note there is a…

That's actually really important. Sometimes upstream sources do disappear, and the caches we have don't help very much, or only help for the exact collection used in a specific package, when they're one big chunk like with buildGoModule.

I hope that FODs in Nix itself stay flexible, though, even if a more limited form becomes preferable in Nixpkgs. New fetchers being possible to implement in Nixpkgs rather than in Nix itself seems really valuable, and clunky uses of FODs to wrap upstream vendorization are still an important fallback option for wrapping new or ill-behaved upstream package managers, until better implementations are worked out or become possible.

Re: Building Go programs with Nix flakes

#19

Post by silisili »

sidlls wrote: post #16
jjtheblunt wrote: post #13

"go" the übertool is so well thought out and handy, I'm missing how this doesn't essentially add complexity.

Dependency management in go is…underwhelming to say the least. Go get? Go mod download? Go install? Go tool error messages related to the same? All a mess. This issue, implicit interfaces, and a CS-3/400 class project quality implementation of generics (harsh, I know) are three reasons that the language frustrates me to no end.

Agreed on most points, but why the distaste for implicit interfaces? I find myself really, really liking that feature.

Re: Building Go programs with Nix flakes

#20

Post by ghuntley »

flicaflow wrote: post #2

Nice to have some development in this space. However I would be very interested in some discussion why this approach should be superior to goBuildModule, which is also mentioned in the article. goBuildModule works well for us and it is even arguably simpler to use, you just have to update the vendor hash whenever you updated the dependencies. The only real downside for us is that it doesn't handle private dependencie…

See https://code.tvl.fyi/tree/nix/buildGo as an alternative.

"Most language-specific Nix tooling outsources the build to existing language-specific build tooling, which essentially means that Nix ends up being a wrapper around all sorts of external build systems.

However, systems like Bazel take an alternative approach in which the compiler is invoked directly and the composition of programs and libraries stays within a single homogeneous build system.

Users don't need to learn per-language build systems and especially for companies with large monorepo-setups (like Google) this has huge productivity impact.

This project is an attempt to prove that Nix can be used in a similar style to build software directly, rather than shelling out to other build systems. "

Post reply on HN