Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Unregistry – “docker push” directly to servers without a registry

github.com

Post reply on HN
Print view View original

161–170 of 178 posts

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#161

Post by psviderski »

matt_kantor wrote: post #119
selcuka wrote: post #33

Earlier quoted context omitted.

You mean ssh'ing into the remote server, then pulling image from local? That would require your local host to be accessible from the remote host, or setting up some kind of ssh tunneling.

This is what docker-pushmi-pullyu[1] does, using `ssh -R` as suggested by a sibling comment. [1]: https://github.com/mkantor/docker-pushmi-pullyu

The problem with running a registry locally is that Docker doesn't provide an API to get individual image layers to be able to build a registry API on top. You have to hook into the containerd Docker uses under the hood. You can't do this locally in many cases, for example, on macOS the VM running Docker Desktop doesn't expose the containerd socket. I guess the workaround you implemented in docker-pushmi-pullyu is an extra copy to the registry which is a bummer.

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#162

Post by psviderski »

dboreham wrote: post #115

I like the idea, but I'd want this functionality "unbundled". Being able to run a registry server over the local containerd image store is great. The details of how some other machine's containerd gets images from that registry to me is a separate concern. docker pull will work just fine provided it is given a suitable registry url and credentials. There are many ways to provide the necessary network connectivity and…

They're unbundled already. You can run unregistry as a standalone service and use your own way to push/pull from it: https://github.com/psviderski/unregistry?tab=readme-ov-file#...

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#163

Post by lloeki »

cyberax wrote: post #116
lloeki wrote: post #108

Earlier quoted context omitted.

> We have a private repo with pre-compiled binaries, and a simple Homebrew formula that downloads the utilities and installs them. Perfectly doable with Nix. Ignore the purists and do the hackiest way that works. It's too bad that tutorials get lost on concepts (which are useful to know but a real turn down) instead of focusing on some hands-on practical how-to. This should about do it and is really not that differen…

Hm. That actually sounds doable (we do have hashes for integrity). I'll try that and see how it goes. > Instantly sounds like a whole reason to use nix and capture those tools as part of the dependency set. It's tempting, and I tried that, but ran away crying. We're using Docker images instead for now. We are also using direnv that transparently execs commands inside Docker containers, this works surprisingly well.

Sure, whatever floats your boat!

I'm just sad that Nix is often dismissed as intractable, and I feel that's mostly because tutorials get too hung up on concept rabbit holing.

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#164

Post by sushidev »

I've prepared a quick one using reverse port forwarding and a local temp registry. In case anyone finds it useful:

  #!/bin/bash
  set -euo pipefail
  
  IMAGE_NAME="my-app"
  IMAGE_TAG="latest"
  
  # A temporary Docker registry that runs on your local machine during deployment.
  LOCAL_REGISTRY="localhost:5000"
  REMOTE_IMAGE_NAME="${LOCAL_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}"
  REGISTRY_CONTAINER_NAME="temp-deploy-registry"
  
  # SSH connection details.
  # The jump host is an intermediary server. Remove `-J "${JUMP_HOST}"` if not needed.
  JUMP_HOST="user@jump-host.example.com"
  PROD_HOST="user@production-server.internal"
  PROD_PORT="22" # Standard SSH port
  
  # --- Script Logic ---
  
  # Cleanup function to remove the temporary registry container on exit.
  cleanup() {
      echo "Cleaning up temporary Docker registry container..."
      docker stop "${REGISTRY_CONTAINER_NAME}" >/dev/null 2>&1 || true
      docker rm "${REGISTRY_CONTAINER_NAME}" >/dev/null 2>&1 || true
      echo "Cleanup complete."
  }
  
  # Run cleanup on any script exit.
  trap cleanup EXIT
  
  # Start the temporary Docker registry.
  echo "Starting temporary Docker registry..."
  docker run -d -p 5000:5000 --name "${REGISTRY_CONTAINER_NAME}" registry:2
  sleep 3 # Give the registry a moment to start.
  
  # Step 1: Tag and push the image to the local registry.
  echo "Tagging and pushing image to local registry..."
  docker tag "${IMAGE_NAME}:${IMAGE_TAG}" "${REMOTE_IMAGE_NAME}"
  docker push "${REMOTE_IMAGE_NAME}"
  
  # Step 2: Connect to the production server and deploy.
  # The `-R` flag creates a reverse SSH tunnel, allowing the remote host
  # to connect back to `localhost:5000` on your machine.
  echo "Executing deployment command on production server..."
  ssh -J "${JUMP_HOST}" "${PROD_HOST}" -p "${PROD_PORT}" -R 5000:localhost:5000 \
    "docker pull ${REMOTE_IMAGE_NAME} && \
     docker tag ${REMOTE_IMAGE_NAME} ${IMAGE_NAME}:${IMAGE_TAG} && \
     systemctl restart ${IMAGE_NAME} && \
     docker system prune --force"
  
  echo "Deployment finished successfully."

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#165

Post by cik »

psviderski wrote: post #160
richardc323 wrote: post #144

I naively sent the Docker developers a PR[1] to add this functionality into mainline Docker back in 2015. I was rapidly redirected into helping out in other areas - not having to use a registry undermined their business model too much I guess. [1]: https://github.com/richardcrichardc/docker2docker

You're the OG! Hats off, mate. It's a bummer docker still doesn't have an API to explore image layers. I guess their plans to eventually transition to containerd image store as the default. Once we have containerd image store both locally and remotely we will finally be able to do what you've done without the registry wrapper.

You're bang on, but you can do things with dive (https://github.com/wagoodman/dive) and use chunks of the code in other projects... That's what I've been doing. The license is MIT so it's permissive.

But yes, an API would be ideal. I've wasted far too much time on this.

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#166

Post by matt_kantor »

pbh101 wrote: post #157
akovaski wrote: post #134

Earlier quoted context omitted.

The ability to push a verified artifact is an anti-feature in most contexts? How so?

It is fine if you are just working by yourself on non-prod things and you’re happy with that. But if you are working with others on things that matter, then you’ll find you want your images to have been published from a central, documented location, where it is verified what tests they passed, the version of the CI pipeline, the environment itself, and what revision they were built on. And the image will be tagged wi…

With that sort of setup you'd run `docker pussh` from your build server, not your local machine (really though you'd probably want a non-ephemeral registry, so wouldn't use unregistry at all).

Other than "it's convenient and my use case is low-stakes enough for me to not care", I can't think of any reason why one would want to build images on their production servers.

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#167

Post by matt_kantor »

psviderski wrote: post #161
matt_kantor wrote: post #119

Earlier quoted context omitted.

This is what docker-pushmi-pullyu[1] does, using `ssh -R` as suggested by a sibling comment. [1]: https://github.com/mkantor/docker-pushmi-pullyu

The problem with running a registry locally is that Docker doesn't provide an API to get individual image layers to be able to build a registry API on top. You have to hook into the containerd Docker uses under the hood. You can't do this locally in many cases, for example, on macOS the VM running Docker Desktop doesn't expose the containerd socket. I guess the workaround you implemented in docker-pushmi-pullyu is an…

Yeah, a few years ago I remember looking into whether I could expose image layers from the engine as a volume to mount directly into the registry, but at least at the time it seemed complex, and when I write tools like this simplicity is a primary goal.

As a mitigation docker-pushmi-pullyu caches pushed layers between runs[1]. More often than not I'm only changing upper layers of previously-pushed images, so this helps a lot. Also, since everything happens locally the push phase is typically quite fast even with cache misses (especially on an SSD), especially compared to the pull phase which is usually going over the internet (or another network).

[1]: https://github.com/mkantor/docker-pushmi-pullyu/pull/19/file...

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#168

Post by matt_kantor »

matt_kantor wrote: post #117

Functionality-wise this is a lot like docker-pushmi-pullyu[1] (which I wrote), except docker-pushmi-pullyu is a single relatively-simple shell script, and uses the official registry image[2] rather than a custom server implementation. @psviderski I'm curious why you implemented your own registry for this, was it just to keep the image as small as possible? [1]: https://github.com/mkantor/docker-pushmi-pullyu [2]: htt…

> I'm curious why you implemented your own registry for this

Answering my own question: I think it's because you want to avoid the `docker pull` side of the equation (when possible) by having the registry's backing storage be the same as the engine's on the remote host.

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#169

Post by matt_kantor »

westurner wrote: post #137
matt_kantor wrote: post #117

Functionality-wise this is a lot like docker-pushmi-pullyu[1] (which I wrote), except docker-pushmi-pullyu is a single relatively-simple shell script, and uses the official registry image[2] rather than a custom server implementation. @psviderski I'm curious why you implemented your own registry for this, was it just to keep the image as small as possible? [1]: https://github.com/mkantor/docker-pushmi-pullyu [2]: htt…

Do docker-pussh or docker-pushmi-pullyu verify container image signatures and attestations? From "About Docker Content Trust (DCT)" https://docs.docker.com/engine/security/trust/ : > Image consumers can enable DCT to ensure that images they use were signed. If a consumer enables DCT, they can only pull, run, or build with trusted images. export DOCKER_CONTENT_TRUST=1 cosign > verifying containers > verify attestation…

docker-pushmi-pullyu does a vanilla `docker pull`[1] on the remote side, so you should be able to set `DOCKER_CONTENT_TRUST` in the remote environment to get whatever behavior you want (though admittedly I have not tested this).

If there's desire for an option to specify `--disable-content-trust` during push and/or pull I'll happily add it. Please file an issue if this is something you want.

[1]: https://github.com/mkantor/docker-pushmi-pullyu/blob/12d2893...

Re: Show HN: Unregistry – “docker push” directly to servers without a registry

#170

Post by remram »

psviderski wrote: post #84
remram wrote: post #20

Does it start a unregistry container on the remote/receiving end or the local/sending end? I think that runs remotely. I wonder if you could go the other way instead?

It starts an unregistry container on the remote side. I wonder, what's the use case on your mind for doing it the other way around?

I guess I feel a little dirty running the container on the prod server. My machine has all the dev tools, and it is also where I install and run this pussh tool, so I would rather have the container run there too.
Post reply on HN