Earlier quoted context omitted.
I think the "They" mentioned was Docker, not Podman. That Docker was adopting the containerd standard.
Jup that’s definitely it, not sure how I got it that wrong.
It's not just you. I interpreted it similarly
Docker 29 has changed its default image store for new installs
71–80 of 93 posts
Re: Docker 29 has changed its default image store for new installs
#72Docker already fills up my dev machines yet they decided for this insane solution: > The containerd image store uses more disk space than the legacy storage drivers for the same images. This is because containerd stores images in both compressed and uncompressed formats, while the legacy drivers stored only the uncompressed layers. Why ?
Also this doesn't just mean more disk space usage, but also longer local build times... for the app I'm working on `exporting to image` takes 71.5 seconds with containerd, without containerd it's 4.3s (the rest of the build takes ~180 seconds). And that's just a 5.76GB image.
That's almost certainly nothing to do with the change. Please report this if you can with commands used.
Buildkit isn't changing behavior here. Internally in docker there is a shim to make the legacy storage behave like containerd snapshotters (as well as it can, anyway--not perfect due to hard to resolve issues in the old storage). But it still kept both the compressed and uncompressed versions of images.
Re: Docker 29 has changed its default image store for new installs
#73Earlier quoted context omitted.
The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash
Recompressing should be guaranteed deterministic. It’s the packing/unpacking of tar archives to/from directories on disk that leads to the non-determinism (such as timestamps and ownership metadata). If the tar is left intact, both zstd and gzip should produce byte for byte identical outputs given the same compression parameters.
That is not correct. You would have to use the same compression tool (and likely version) for this to match.
Old docker discarded the compressed bits but kept some metadata about the the so it can at least recreate the tar.
It also recreated the manifest o push.
Re: Docker 29 has changed its default image store for new installs
#74Earlier quoted context omitted.
You are correct; I confused archiving with compression. However, even considering only the compression process, same compression parameters cannot be guaranteed, as it is unknown which compression parameters the image publisher used.
Thats true. And regardless of compressed vs regular tar, I think the OCI format working with opaque archives is extremely limiting. I hope the industry will eventually redesign to use content addressable storage per file and have metadata to describe the layer/disk layout instead. That would allow per file deduplication, and we can use tar for just bulk transfer over the wire, rather than using tar for the data at re…
containerd 2.3 has support for erofs which does a direct import of the layer.
It can even convert the tar based layers to erofs, faster than extracting the tar normally.
Also looking at block-based content store so that blocks can be deduped across images.
Re: Docker 29 has changed its default image store for new installs
#75Earlier quoted context omitted.
The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash
If that's the purpose, couldn't you store the hash and throw away the compressed image? (As others said, compression is deterministic for the same algorithm, parameters and input data)
The whole entire reason is compression is not deterministic across tooling.
Re: Docker 29 has changed its default image store for new installs
#76Earlier quoted context omitted.
Recompressing should be guaranteed deterministic. It’s the packing/unpacking of tar archives to/from directories on disk that leads to the non-determinism (such as timestamps and ownership metadata). If the tar is left intact, both zstd and gzip should produce byte for byte identical outputs given the same compression parameters.
That is not correct. You would have to use the same compression tool (and likely version) for this to match. Old docker discarded the compressed bits but kept some metadata about the the so it can at least recreate the tar. It also recreated the manifest o push.
Thanks for the correction. I did mean given the same tooling version/parameters, but (as you and others pointed out) preserving and recreating that state is not at all straightforward.
Re: Docker 29 has changed its default image store for new installs
#77Since people are mentioning alternatives, worth calling out colima if maybe you don't want to jump onto Red Hat's podman or Suse's Rancher. Both are open source but there are some corporate entities behind them with their own agendas that you might want to consider.
My reasoning is simply that I don't really want to swap out one overly complicated thing for another. I'm sure Podman is fine and amazing. But I'm just not in the IBM/Red Hat ecosystem and I have some reservations their generally a bit overly complex solutions. There's a reason IBM is involved, just saying. And as I'm not planning to use podman in production I see no reason to have it on my laptop.
As for Rancher, that seems to me a bit like moving the problem than solving it as it seems to be a for profit solution around an OSS core with its own complexity and potentially similar risks to Docker Desktop down the road.
With colima, it's all open source and easy to install/upgrade via homebrew. Nice simple wrapper around qemu. There's no UI, and I don't really miss having one. Lazydocker works fine as a TUI if you crave a UI and so do other generic docker UIs/IDEs. I mainly use docker and docker compose on the command line and that works fine for me. It has Kubernetes support as well if you need that but that's not something I use or need.
Re: Docker 29 has changed its default image store for new installs
#78[dead]
Re: Docker 29 has changed its default image store for new installs
#79It sounds like this breaks all Docker installs that use userns-remap? Are they really shipping a breaking change with no fix? In addition to bloating the disk? In addition to breaking all old systems that relied on mapping /var/lib/docker? I can't believe Docker finally shit the bed. Time to replace Docker with Podman.... sigh
The page is documenting current compatibility. No users are automatically migrated to an incompatible setup. If you used userns-remap then you should currently still be using the previous Docker storage stack.
Re: Docker 29 has changed its default image store for new installs
#80I'm a maintainer of the Moby project (which is used to build the Docker Engine), and saw this post got some attention, so let me try to outline some of the changes and motivation. Happy to answer questions if there's any.
First of all, some history; the Docker Engine was a monolith daemon that provided many services; this worked well when using Docker as a standalone solution, but when used as runtime for Kubernetes, this wasn't ideal; many components were not designed for this purpose, which meant they had to be replaced / overridden with hacks to make it work. The containerd project was created to provide a more modular runtime for the container ecosystem, providing separate subcomponents (a containerd runtime, image/content storage) for the container ecosystem to build on. It was created "from scratch" with lessons learned over the Years, providing a modern foundation.
While docker has used containerd as a runtime for many Years, it still used its own implementation for storing images ("graph-drivers"); this implementation started to show its age and had many limitations; graph-drivers have no native support for multi-platform ("multi-arch") images, no support for OCI Artifacts, and no reproducible images when pushing to different registries (among others).
Around 4 Years ago, we started to re-implement the image storage using containerd "snapshotters"; our initial goal was to provide a mostly seamless transition; add multi-arch support, but keep the UX as close as possible to the graph-drivers. Around 2 Years ago, Docker Desktop changed to using the containerd image storage (snapshotters) as a default for new installations, and Docker v29 made it the default for Linux installations.
While we kept most of the UX similar, there are some differences; when storing an image with graph-drivers, docker would pull the OCI image, extract the content (layers), and discard the (compressed) layers. While this reduced storage, it also made images non-reproducible as the image had to be re-constructed when pushing to a registry (which also resulted in slower pushes).
The containerd image storage uses a different design, where a copy of the compressed artifacts are preserved (by default); this requires more storage to keep these extra blobs, but reduces duplication and increases push performance. It was the decision containerd maintainers made early in their design process, and all containerd-based tools have used this model since the start of the containerd project.
We have a couple of roadmap items to improve this in future; some are outlined in this ticket; https://github.com/moby/moby/issues/51581, but there's other options that will become availeble through the containerd image store; support for erofs as an alternative to (tar) compressed image layers, as well as automatic garbage-collection (which would reduce the need for manually pruning content through `docker system prune` (and related commands).
(FWIW; docker still provides graph-drivers as an alternative https://docs.docker.com/engine/storage/drivers/select-storag...)