Keeping Up with Docker Official Images • Hacker News Board

Re: Keeping Up with Docker Official Images

#11

Post by michaelperel » Tue, Jun 08, 2021, 7:42 PM UTC

Shameless Plug: I wrote a cli-plugin for docker, docker-lock, to solve the mutable tag problem without having to manually specify hashes - https://github.com/safe-waters/docker-lock

It creates a Lockfile (think package-lock.json) that tracks the image digests (sha256 hashes) of your base images, so you will always know exactly which images you are using even if you only specify tags. This way, you can know if a base image has changed, yet still receive important security updates that you would not receive if you hardcode the digest. It supports any registry, so is useful even if you are not using Dockerhub. It also works with Dockerfiles, docker-compose files, and Kubernetes manifests.

I hope anyone dealing with this issue finds it helpful :)

Re: Keeping Up with Docker Official Images

#12

Post by 0xbadcafebee » Tue, Jun 08, 2021, 7:44 PM UTC

slimslenders wrote: post #5

0xbadcafebee wrote: post #3

Distroless is a misnomer. Any "distribution" of files that has its own build process and unique end state, is a distribution. So, "distroless" containers are still "a distro", it just has less extraneous files. This has some useful properties, but at the same time downsides: bolting-on additional requirements is significantly harder, and development outside of a tightly controlled CI system becomes burdensome. At the…

Thank you, and yes. I love that idea of a distance metric. I think you're also pointing out that distroless images can end up increasing this developer distance, right? I was originally drawn to the idea of distroless images as a way to reduce vulnerabilities. However, staying up to date with a well maintained distro is effective too.

Yeah; specifically, if your developers don't use the distroless containers to develop their apps (or stop using them when they become burdensome to update), then the environments are becoming divergent, which will lead to divergent behavior. The solution [incl. for things like vulnerability management] is the same idea behind Shift Left: move as much of the work "left" (earlier) in the pipeline/value stream/etc as possible.

The more you shift left, the smaller the distance from dev to prod, the better the outcomes. Whatever environment the developers want to use to develop, make the production system the same; then improve the development environment in order to improve production. Over time this will need to change as complex systems are hard to replicate locally. But the closer they are, the better.

Re: Keeping Up with Docker Official Images

#13

Post by slimslenders » Tue, Jun 08, 2021, 7:49 PM UTC

itamarst wrote: post #9

Seems like the author is assuming the official images get security updates applied immediately. They don't. So you need to apply system package updates yourself, separately from updating the base image. https://pythonspeed.com/articles/security-updates-in-docker/ has an example of a security update that was missing in the official Ubuntu base image, two weeks after Ubuntu released the updated package. I've also seen…

I came across your article on security updates when I was researching this. Thanks for writing it - it was super helpful. One of the sub goals for us is to quantify the lag that you've observed here. We're also reviewing an automated check that would flag times when an apt upgrade would be material. This is very much based on your bad arguments 1 through 4.

Re: Keeping Up with Docker Official Images

#14

Post by mfontani » Tue, Jun 08, 2021, 8:04 PM UTC

I keep track of upstream image changes using https://github.com/crazy-max/diun

... but I also separately ensure all software installed in a docker image is pinned to a version, and have a process I run daily to check whether the upstream packages versions have changed, in which case I rebuild the images which then get the updated (possibly security) version.

It's fiddly, and a lot of bash and perl. I'd welcome a similarly trust-able tooling from a reputable source.

Re: Keeping Up with Docker Official Images

#15

Post by itamarst » Tue, Jun 08, 2021, 8:21 PM UTC

slimslenders wrote: post #13

itamarst wrote: post #9

Seems like the author is assuming the official images get security updates applied immediately. They don't. So you need to apply system package updates yourself, separately from updating the base image. https://pythonspeed.com/articles/security-updates-in-docker/ has an example of a security update that was missing in the official Ubuntu base image, two weeks after Ubuntu released the updated package. I've also seen…

I came across your article on security updates when I was researching this. Thanks for writing it - it was super helpful. One of the sub goals for us is to quantify the lag that you've observed here. We're also reviewing an automated check that would flag times when an apt upgrade would be material. This is very much based on your bad arguments 1 through 4.

Cool, I would love to see some numbers! I know that some official images aren't updated for literally months (e.g. `centos`, even before the CentOS/RHEL decoupling), but others are more frequent.

Re: Keeping Up with Docker Official Images

#16

Post by terom » Tue, Jun 08, 2021, 8:23 PM UTC

itamarst wrote: post #9

Seems like the author is assuming the official images get security updates applied immediately. They don't. So you need to apply system package updates yourself, separately from updating the base image. https://pythonspeed.com/articles/security-updates-in-docker/ has an example of a security update that was missing in the official Ubuntu base image, two weeks after Ubuntu released the updated package. I've also seen…

The neat way to re-run a layer daily is to use a build arg:

  FROM debian
  
  ARG APT_UPDATE_DATE
  RUN apt-get update && apt-get upgrade -y

Then `docker build --build-arg APT_UPDATE_DATE=$(date +%Y%m%d)` will always re-run the `apt-get update` for the first build each day. You probably also want `--pull`.

Re: Keeping Up with Docker Official Images

#17

Post by slimslenders » Tue, Jun 08, 2021, 8:25 PM UTC

itamarst wrote: post #15

slimslenders wrote: post #13

Earlier quoted context omitted.

I came across your article on security updates when I was researching this. Thanks for writing it - it was super helpful. One of the sub goals for us is to quantify the lag that you've observed here. We're also reviewing an automated check that would flag times when an apt upgrade would be material. This is very much based on your bad arguments 1 through 4.

Cool, I would love to see some numbers! I know that some official images aren't updated for literally months (e.g. `centos`, even before the CentOS/RHEL decoupling), but others are more frequent.

will send when we've got some data. Had been wanting to reach out to you about an idea for an automated check anyway.

Re: Keeping Up with Docker Official Images

#18

Post by jonjonsonjr » Tue, Jun 08, 2021, 8:41 PM UTC

michaelperel wrote: post #11

Shameless Plug: I wrote a cli-plugin for docker, docker-lock, to solve the mutable tag problem without having to manually specify hashes - https://github.com/safe-waters/docker-lock It creates a Lockfile (think package-lock.json) that tracks the image digests (sha256 hashes) of your base images, so you will always know exactly which images you are using even if you only specify tags. This way, you can know if a base…

Nice! I was thinking about building something similar -- just filed an issue for how you might extend this to work for migrating registries.

Pleasantly surprised to come across this PR: https://github.com/safe-waters/docker-lock/pull/73

This is a perfect application of crane :)

Re: Keeping Up with Docker Official Images

#19

Post by slimslenders » Tue, Jun 08, 2021, 9:19 PM UTC

michaelperel wrote: post #11

Shameless Plug: I wrote a cli-plugin for docker, docker-lock, to solve the mutable tag problem without having to manually specify hashes - https://github.com/safe-waters/docker-lock It creates a Lockfile (think package-lock.json) that tracks the image digests (sha256 hashes) of your base images, so you will always know exactly which images you are using even if you only specify tags. This way, you can know if a base…

Just took a look. Very cool! The generate/verify/rewrite phases are very familiar :). I guess we sort of moved rewrite/verify into a PullRequest/CheckRun. Would it make sense to run verify and rewrite as a pre-commit hook too?

Re: Keeping Up with Docker Official Images

#20

Post by slimslenders » Tue, Jun 08, 2021, 9:55 PM UTC

mfontani wrote: post #14

I keep track of upstream image changes using https://github.com/crazy-max/diun ... but I also separately ensure all software installed in a docker image is pinned to a version, and have a process I run daily to check whether the upstream packages versions have changed, in which case I rebuild the images which then get the updated (possibly security) version. It's fiddly, and a lot of bash and perl. I'd welcome a simi…

Feel free to get in touch if you want to try out our tool. The support for pinning versions in run layers is just being released but happy to send you a breakdown of how that works.