Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BuildKit: Docker's Hidden Gem That Can Build Almost Anything

tuananh.net

Post reply on HN
Print view View original

71–80 of 80 posts

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#71

Post by stevenhuang »

amluto wrote: post #54
matheus-rr wrote: post #38

The --mount=type=cache for package managers is genuinely transformative once you figure it out. Before that, every pip install or apt-get in a Dockerfile was either slow (no caching) or fragile (COPY requirements.txt early and pray the layer cache holds). What nobody tells you is that the cache mount is local to the builder daemon. If you're running builds on ephemeral CI instances, those caches are gone every build…

There is something wrong with the industry in which we think that, when a production build requires SSH keys, the problem is that the keys might leak into the build artifact.

Keys leaking into the build artifact was never the concern.

It's about not having the private keys stored unknowingly in intermediate layers of a build container.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#72

Post by amluto »

stevenhuang wrote: post #71
amluto wrote: post #54

Earlier quoted context omitted.

There is something wrong with the industry in which we think that, when a production build requires SSH keys, the problem is that the keys might leak into the build artifact.

Keys leaking into the build artifact was never the concern. It's about not having the private keys stored unknowingly in intermediate layers of a build container.

Those intermediate layers are usually part of the artifact. Try exporting an image with docker save and investigate what’s inside. This is all documented in a mostly comprehensible manner in the OCI specs.

I’m afraid you’re missing my point, though. A high quality build system takes fixed inputs and produces outputs that are, to the extent possible, only a function of the inputs. If there’s a separate process that downloads the inputs (and preferably makes sure they are bitwise identical to what is expected), fine, but that step should be strictly outside the inputs to the actual thing that produces the release artifact. Think of it as:

    artifact = build_process(inputs)

    inputs = fetch(credentials, cache, hashes, etc)
Or, even better perhaps:

    inputs = …
    assert hash(inputs) == expected
(And now, unless you accidentally hash your credentials into the expected hash, you can’t leak credentials into the output!)

Once you have commingled it so that it looks like:

    final output, intermediate layers = monolithic_mess(credentials, cache, etc)
Then you completely lose track of which parts are deterministic, what lives in the intermediate layers, where the credentials go, etc.

Docker build is not a good build system, and it strongly encourages users to do this the wrong way, and there are many, many things wrong with it, and only one of those things is that the intermediate layers that you might think of as a cache are also exposed as part of the output.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#74

Post by stevenhuang »

amluto wrote: post #72
stevenhuang wrote: post #71

Earlier quoted context omitted.

Keys leaking into the build artifact was never the concern. It's about not having the private keys stored unknowingly in intermediate layers of a build container.

Those intermediate layers are usually part of the artifact. Try exporting an image with docker save and investigate what’s inside. This is all documented in a mostly comprehensible manner in the OCI specs. I’m afraid you’re missing my point, though. A high quality build system takes fixed inputs and produces outputs that are, to the extent possible, only a function of the inputs. If there’s a separate process that do…

It was confusing of you to say build artifact to refer to the container itself in this context. Sure you're not wrong because the container is also a build artifact, but in context of CI, build artifacts is the output of running the build using the container.

Hence my confusion of what you meant -- no one's saying ssh keys are in the CI build artifacts. But obviously they can be in the container as layers if people do it wrong, which is bad.

We're talking about the same thing basically. Yes fully defining your inputs to the container by passing in the keys is a good solution.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#75

Post by modulared »

cpuguy83 wrote: post #60
jasonpeacock wrote: post #53

Earlier quoted context omitted.

Flakes fixes this for Nix, it ensures builds are truly reproducible by capturing all the inputs (or blocking them).

No it doesn't. If the content of a url changes then the only way to have reproducibility is caching. You tell nix the content hash is some value and it looks up the value in the nix store. Note, it will match anything with that content hash so it is absolutely possible to tell it the wrong hash.

Not having a required input, say when you try to reproduce a previous build of a package, is a separate issue to an input silently changing when you go to rebuild it. No build system can ensure a link stays up, only that what's fetched hasn't changed. The latter is what the hash in nix is for. If it tries to fetch a file from a link and the hash doesn't match, the build fails.

Flakes, then, run in a pure evaluation mode, meaning you don't have access to stuff like the system triple, the current time, or env vars and all fetching functions require a hash.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#76

Post by cpuguy83 »

modulared wrote: post #75
cpuguy83 wrote: post #60

Earlier quoted context omitted.

No it doesn't. If the content of a url changes then the only way to have reproducibility is caching. You tell nix the content hash is some value and it looks up the value in the nix store. Note, it will match anything with that content hash so it is absolutely possible to tell it the wrong hash.

Not having a required input, say when you try to reproduce a previous build of a package, is a separate issue to an input silently changing when you go to rebuild it. No build system can ensure a link stays up, only that what's fetched hasn't changed. The latter is what the hash in nix is for. If it tries to fetch a file from a link and the hash doesn't match, the build fails. Flakes, then, run in a pure evaluation m…

Buildkit has the same caching model. That's what I'm saying. It doesn't force you to give it digests like nix functions often do but you can (and should).

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#77

Post by amluto »

stevenhuang wrote: post #74
amluto wrote: post #72

Earlier quoted context omitted.

Those intermediate layers are usually part of the artifact. Try exporting an image with docker save and investigate what’s inside. This is all documented in a mostly comprehensible manner in the OCI specs. I’m afraid you’re missing my point, though. A high quality build system takes fixed inputs and produces outputs that are, to the extent possible, only a function of the inputs. If there’s a separate process that do…

It was confusing of you to say build artifact to refer to the container itself in this context. Sure you're not wrong because the container is also a build artifact, but in context of CI, build artifacts is the output of running the build using the container. Hence my confusion of what you meant -- no one's saying ssh keys are in the CI build artifacts. But obviously they can be in the container as layers if people d…

I think there's a lot of confusing terminology in your comment.

> the container is also a build artifact

By "build artifact" I mean the data that is the output of the build and get distributed to other machines (or run locally perhaps). So a build artifact can be a tarball, an OCI image [0], etc. But calling a container a build artifact is really quite strange. A "container" is generally taken to mean the thing you might see in the output of 'docker container ls' or similar -- they're a whole pile of state including a filesystem, a bunch of volume mounts, and some running processes if they're not stopped. You don't distribute containers to other machines [1].

> in context of CI, the output of running the build using the container

I have no idea what you mean. What container? CI doesn't necessarily involve containers at all.

> no one's saying ssh keys are in the CI build artifacts. But obviously they can be in the container as layers if people do it wrong, which is bad.

If the build artifact is an image, and the keys are in the image, then the keys are in the build artifact.

> Yes fully defining your inputs to the container by passing in the keys is a good solution.

Are you suggesting doing a build by an incantation like:

    $ docker run --rm -v /input:[sources] -v "/keys:($HOME)/.ssh" my_builder:latest /input/build_my_thing
This is IMO a terrible idea. A good build system DOES NOT PROVIDE KEYS TO THE BUILD PROCESS.

Yes, I realize that almost everyone fudges this because we have lots of tools that make it easy. Even really modern stuff like uv does this.

    $ uv build
whoops, that uses optional credentials, fetches (hopefully locked-by-hash) dependencies, and builds. It's convenient for development. But for a production build, this would be much better if it was cleanly split into a fetch-the-dependencies step and a build step and the build step ran without network access or any sort of credentials.

[0] https://specs.opencontainers.org/image-spec/

[1] A build artifact could be a container snapshot, but that's different.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#78

Post by stevenhuang »

amluto wrote: post #77
stevenhuang wrote: post #74

Earlier quoted context omitted.

It was confusing of you to say build artifact to refer to the container itself in this context. Sure you're not wrong because the container is also a build artifact, but in context of CI, build artifacts is the output of running the build using the container. Hence my confusion of what you meant -- no one's saying ssh keys are in the CI build artifacts. But obviously they can be in the container as layers if people d…

I think there's a lot of confusing terminology in your comment. > the container is also a build artifact By "build artifact" I mean the data that is the output of the build and get distributed to other machines (or run locally perhaps). So a build artifact can be a tarball, an OCI image [0], etc. But calling a container a build artifact is really quite strange. A "container" is generally taken to mean the thing you m…

Container is standard terminology to refer to a running instance of an image. Yes I was being imprecise, substitute container for oci image. But you seem hung up on frivolity and not getting what I'm saying. We are agreeing with each other and just talking in circles. I can see that you don't see that but that's ok. All of this was because I misunderstood what you said initially when you referred to build artifact as the oci image when I thought you were talking about other sorts of build artifacts.

I mean using the CI system to pass in keys or creds. Yes, it's better to build the image with dependencies, but sometimes you can't do that.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#79

Post by westurner »

mikepurvis wrote: post #70
westurner wrote: post #69

Earlier quoted context omitted.

Blaze instead of make, ant, maven. But now there's cmake and ninjabuild. gn wraps ninjabuild wraps cmake these days fwiu. Blaze is/was integrated with Omega scheduler, which is not open. Bazel is open source. By the time Bazel was open sourced, Twitter had pantsbuild and Facebook had buck. OpenWRT's Makefiles are sufficient to build OpenWRT and the kernel for it. (GNU Make is still sufficient to build the Linux kerne…

The ansible-in-containers thing is very much an unsolved problem. Basically right now you have three choices: - install ansible in-band and run it against localhost (sucks because your playbook is in a final image layer; you might not want Python at all in the container) - use packer with ansible as your provisioner and a docker container export, see: https://alex.dzyoba.com/blog/packer-for-docker/ - copy a previous…

With multi-stage Dockerfiles, you only copy the final, built application artifacts from the earlier stage(s). Then, building a package as one signed file to copy is justified and easier anyway.

There's always:

  RUN dnf remove -y ansible && dnf clean all
I thought there was a native way to build container images with ansible that don't have ansible installed in the image though?

ansible/ansible-builder: https://github.com/ansible/ansible-builder

"How to Use ansible-builder to Create Execution Environments" https://oneuptime.com/blog/post/2026-02-21-how-to-use-ansibl... :

> The Build Process Explained: When you run ansible-builder build, it goes through these steps:

> Reads your `execution-environment.yml` definition, Resolves collection dependencies (including transitive dependencies), Generates a `Containerfile` in a `context/` directory, Copies dependency files into the build context, Runs the container build using Podman or Docker

How to Use Ansible to Build Podman Images > Building with Build Arguments, Multi-Stage Builds, Building with Buildah: https://oneuptime.com/#building-with-buildah :

It's possible to build images using build a commands in an ansible playbook:

  buildah from
  buildah copy
  buildah run
  buildah config
  buildah commit
.

The CLI way to traverse an ansible playbook graph, for e.g. dagger:

  ansible-playbook --list-tags 
  ansible-playbook --tags {tagname}
But where can it parallelize?

It probably shouldn't (?) parallelize because that wouldn't be a deterministic build; installing A then B is not the same as installing B then A. (Is not the same thing as installing A in one container image layer, B in another container image layer, and then trying to merge the package databases.) A given package B could conditionally install or configure according to whether or not A is already installed, and so for example package install tasks are not commutative.

.

Bootc (osbuild) builds VM and native machine images from Containerfiles:

  bootc-image-builder
  bootc upgrade
/? bootc ansible [Debian] https://www.google.com/search?q=bootc+ansible https://www.google.com/search?q=bootc+ansible+debian

"Demonstrate a debian or arch base image" bootc-dev/bootc#865 https://github.com/bootc-dev/bootc/issues/865

Is packer necessary with bootc?

To require signatures for containers and also for native containers with bootc:

  cat /etc/containers/policy.json | grep sigstoreSigned

  podman image trust show --raw
Here's this on ansible, dagger, bootc: "Public link: Dagger for Programmable CI/CD" https://gemini.google.com/share/3965633a3ff8

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#80

Post by hanwenn »

mikepurvis wrote: post #42
skrtskrt wrote: post #33

Earlier quoted context omitted.

> It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason about, or in the worst case, terrifying to touch. This is true of packaging and build systems in general. They are often the passion projects of one or a handful of people in an organization - by the time they have active outside development, those idiosyncratic concepts are…

As someone who has worked in the space for a while and been heavily exposed to nix, bazel, cmake, bake, and other systems, and also been in that "passion project" role, I think what I've found is that these kinds of systems are just plain hard to talk about. Even the common elements like DAGs cause most people's eyes to immediately glaze over. Managers and executives are happy to hear that you made the builds faster…

blaze/bazel was a big improvement over its predecessor (a set of python scripts that generated huge makefiles), but that did not make it free of accidental engineering. The google infrastructure teams were very tightly staffed, so for a long time, it was held together with proverbial duct tape and heroism.

Source: I was on part of the team that did the open sourcing, and had to clean lots of cruft in the code base that had accreted over 8 years.

Post reply on HN