Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker 29 has changed its default image store for new installs

docs.docker.com

Post reply on HN
Print view View original

81–90 of 93 posts

Re: Docker 29 has changed its default image store for new installs

#81

Post by thaJeztah »

mort96 wrote: post #68
a_t48 wrote: post #64

Earlier quoted context omitted.

Zstd for example only promises determinism on the same version of the library. I've personally seen the hashes mutate between pull and export. Things like tar padding also make a difference. Really, the thing to do is to hash on the _uncompressed_ data and let compression be a transport/registry detail. That's what I've done, at least.

I didn't know that about zstd, that's a bit unfortunate. Tar isn't related here though, we're talking about compression not archival formats

Yes, compression being part of the OCI image's digest was (in hindsight) a poor decision. _Technically_ OCI images allow uncompressed layers, and the layers could be included without compression (and transport compression to be used); this would allow layers to be fully reproducible. We explored some options to do this (and made some preparations; https://github.com/containerd/containerd/pull/8166), but also discovered that various implementations of registry clients didn't handle transport-compression correctly (https://github.com/distribution/distribution/pull/3754), which could result in client either pulling the full, uncompressed, content, or image validation failing.

Re: Docker 29 has changed its default image store for new installs

#82

Post by UltraSane »

fabian2k wrote: post #9

> This difference is particularly noticeable with multiple images sharing the same base layers. With legacy storage drivers, shared base layers were stored once locally, and reused images that depended on them. With containerd, each image stores its own compressed version of shared layers, even though the uncompressed layers are still de-duplicated through snapshotters. This seems like a really weird decision. If bas…

Enterprise grade dedupe helps a lot with this. Dedupe on them has gotten very good.

Re: Docker 29 has changed its default image store for new installs

#83

Post by UltraSane »

colechristensen wrote: post #35
fpoling wrote: post #22

Earlier quoted context omitted.

On my 2 years old ThinkPad laptop SSD is faster than lz4. On a fat EC2 server lz4 is faster. So one really has to test a particular config.

Yeah, I'm not surprised the PCIe 5.0 transfer speeds matched with top tier SSD chips win that race. It still bothers me that the fastest most performant computer I have access to is almost always my laptop, and that by a considerable margin. Someone should do some lz4 vs. ssd benchmarks across hardware to make my argument more solid and the boundaries clear.

You can get AWS instances with very fast local NVMe drives.

Re: Docker 29 has changed its default image store for new installs

#84

Post by a_t48 »

thaJeztah wrote: post #81
mort96 wrote: post #68

Earlier quoted context omitted.

I didn't know that about zstd, that's a bit unfortunate. Tar isn't related here though, we're talking about compression not archival formats

Yes, compression being part of the OCI image's digest was (in hindsight) a poor decision. _Technically_ OCI images allow uncompressed layers, and the layers could be included without compression (and transport compression to be used); this would allow layers to be fully reproducible. We explored some options to do this (and made some preparations; https://github.com/containerd/containerd/pull/8166 ), but also discove…

For my registry fork/custom pull client I hash on the uncompressed content and store as compressed under the uncompressed digest. This lets me have my cake and eat it, too - compression free digests, smaller storage costs, be able to set consistent compression settings, have the ability to spend extra CPU to recompress on the backend without breaking hashes, etc. I control both pull client and registry, so it works.

Re: Docker 29 has changed its default image store for new installs

#85

Post by a_t48 »

thaJeztah wrote: post #80

I'm a maintainer of the Moby project (which is used to build the Docker Engine), and saw this post got some attention, so let me try to outline some of the changes and motivation. Happy to answer questions if there's any. First of all, some history; the Docker Engine was a monolith daemon that provided many services; this worked well when using Docker as a standalone solution, but when used as runtime for Kubernetes,…

Hey, just wanted to say I appreciate the containerd API, between the two it's much more performant to work against.

Re: Docker 29 has changed its default image store for new installs

#87

Post by colechristensen »

UltraSane wrote: post #83
colechristensen wrote: post #35

Earlier quoted context omitted.

Yeah, I'm not surprised the PCIe 5.0 transfer speeds matched with top tier SSD chips win that race. It still bothers me that the fastest most performant computer I have access to is almost always my laptop, and that by a considerable margin. Someone should do some lz4 vs. ssd benchmarks across hardware to make my argument more solid and the boundaries clear.

You can get AWS instances with very fast local NVMe drives.

If you have many cores and have the right optimizations in place the bottleneck for lz4 decompression is RAM throughput which is always going to beat whatever fancy disk setup you have.

But yes on the extreme end absolutely there's a point where lz4 stops making sense, but also most of us aren't trying to max out a 128 core postgres server or whatever.

Re: Docker 29 has changed its default image store for new installs

#88

Post by Oxodao »

johannes1234321 wrote: post #36
giobox wrote: post #29

Earlier quoted context omitted.

> https://docs.docker.com/reference/cli/docker/system/prune/ Just in case - I'm always amazed how many Docker users don't know about the prune command for cleaning up the caches and deleting unused container images and just slowly let their docker image cache eat their disk.

Prune is nice, but if you have a bunch of containers which run shirt time for a build step or similar prune would collect those, too. A filter "last used a few months ago" would be useful.

I do prune all the time, but working on a lot of different projects it fills up as quickly as it empties

Re: Docker 29 has changed its default image store for new installs

#89

Post by Oxodao »

stingraycharles wrote: post #25
newsoftheday wrote: post #17

Earlier quoted context omitted.

I had to work on a Mac M3 for a year, it sucked, it did not feel snappier than any Windows or Linux machine (including this one) that I've ever used and that is going back to the 1980's.

I suggest you judge based on benchmarks rather than vibes. If you believe the latest M3 does not perform better than machines you’ve used in the 80s, I have no idea how to even start a reasonable discussion about this.

Well, no ? That's litterally saying "trust the synthetic process, we don't care about real world usage" ?? I don't care if it works better theorically, if it feels bad in everyday usage it IS bad

Re: Docker 29 has changed its default image store for new installs

#90

Post by stingraycharles »

Oxodao wrote: post #89
stingraycharles wrote: post #25

Earlier quoted context omitted.

I suggest you judge based on benchmarks rather than vibes. If you believe the latest M3 does not perform better than machines you’ve used in the 80s, I have no idea how to even start a reasonable discussion about this.

Well, no ? That's litterally saying "trust the synthetic process, we don't care about real world usage" ?? I don't care if it works better theorically, if it feels bad in everyday usage it IS bad

Well, grab an Apple from the 80s and try running a modern app on it and see whether the M3 performs better or worse.

Or, if the point is that software became very bloated, then sure but they also do a lot more nowadays so then you’re really just comparing apples with oranges.

Post reply on HN