Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker 29 has changed its default image store for new installs

docs.docker.com

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Docker 29 has changed its default image store for new installs

#61

Post by a_t48 »

epistasis wrote: post #37
kodama-lens wrote: post #30

Earlier quoted context omitted.

I think there is an Issue/PR right now to change this. See: https://github.com/containerd/containerd/issues/13307

Oh, very glad to see this, ML applications that were mentioned in it are exactly why I was thinking this was such a disastrous change. However, the tedium of the reply chain reminds me why I tend to focus most energy on internal projects rather than external open source... Docker may have been built for a specific type of use case that most developers are familiar with (e.g. web apps backed by a DB container) but con…

Docker(Hub) just isn't built for this use case. I've built https://clipper.dev to better handle ML/large images. It consists of a registry+pull client that breaks apart layers and does content addressing of individual chunks by _uncompressed_ hash, so that content can be better shared. My pull client has better parallelization and wastes much less bandwidth. It annoys the heck out of me when I change one file in a layer and have to redownload bytes my device already has. By sharing across layers I've seen 80-90% improvements in pull times for "patches".

I'm also in the process of building a BuildKit builder, I'm seeing large improvements on the speed of exporting images. The same image that takes Docker >3 minutes to export and push takes me under a minute. https://github.com/clipper-registry/benchmarks/actions/runs/...

Re: Docker 29 has changed its default image store for new installs

#62

Post by a_t48 »

epistasis wrote: post #16
fabian2k wrote: post #9

> This difference is particularly noticeable with multiple images sharing the same base layers. With legacy storage drivers, shared base layers were stored once locally, and reused images that depended on them. With containerd, each image stores its own compressed version of shared layers, even though the uncompressed layers are still de-duplicated through snapshotters. This seems like a really weird decision. If bas…

This is hell for a lot of ML containers, that have gigabytes of CUDA and PyTorch. Before at least you could keep your code contained to a layer. But if I understand this correctly every code revision duplicates gigabytes of the same damn bloated crap.

It's even worse when you end up installing PyTorch as a separate package in some other layer. It's not shared between layers at all with regular Docker.

Re: Docker 29 has changed its default image store for new installs

#64

Post by a_t48 »

mort96 wrote: post #58
XYen0n wrote: post #50

Earlier quoted context omitted.

The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash

If that's the purpose, couldn't you store the hash and throw away the compressed image? (As others said, compression is deterministic for the same algorithm, parameters and input data)

Zstd for example only promises determinism on the same version of the library. I've personally seen the hashes mutate between pull and export. Things like tar padding also make a difference. Really, the thing to do is to hash on the _uncompressed_ data and let compression be a transport/registry detail. That's what I've done, at least.

Re: Docker 29 has changed its default image store for new installs

#65

Post by a_t48 »

epistasis wrote: post #27
spwa4 wrote: post #26

Earlier quoted context omitted.

If you have problems with 13 (I believe) GB of docker layers ... how do you deal with terabytes or petabytes of AI training data?

Petabytes of training data is only one application of PyTorch, which is going to use tens of thousands of containers, but... Inference, development cycles, any of the application domains of PyTorch that don't involve training frontier models... all of those are complicated by excessive container layers. But mostly dev really sucks with writing out an extra 10GB for a small code change.

Going to self promote one last time here - I've built a fix for this, at least for the registry/image export side, at https://clipper.dev. Docker(Hub) can't share large files between layers, but I can.

Re: Docker 29 has changed its default image store for new installs

#66

Post by tetha »

fabian2k wrote: post #9

> This difference is particularly noticeable with multiple images sharing the same base layers. With legacy storage drivers, shared base layers were stored once locally, and reused images that depended on them. With containerd, each image stores its own compressed version of shared layers, even though the uncompressed layers are still de-duplicated through snapshotters. This seems like a really weird decision. If bas…

It does. It's also very nice that this moves storage usage from /var/lib/docker over to /var/lib/containerd.

Due to that, a careless installation of a few new dev-systems under the new docker version immediately blew up storage usage on the root-disk, while happily ignoring hundreds of gigabytes on a volume on /var/lib/docker.. because that's where it needs the storage, right? A few older systems also were upgraded but didn't, which was quite confusing at first.

Sorry for being salty, but that was a pretty hectic afternoon with those new agents trashing builds, and now we have a pretty annoying migration plan to plan for the rest. And yes yes it's just a reinstallation, but we have other things to do as well.

Re: Docker 29 has changed its default image store for new installs

#67

Post by tetha »

newsoftheday wrote: post #19

The article says to regularly run prune, how regularly? Currently I run the following once per day from cron: docker system prune -a -f docker volume prune -a -f

Monitor your disks to see if they grow full, and have an idea what your storage baseline should be. Storage in /var/lib/docker/overlay2 can also leak, even if you prune regularly.

Re: Docker 29 has changed its default image store for new installs

#68

Post by mort96 »

a_t48 wrote: post #64
mort96 wrote: post #58

Earlier quoted context omitted.

If that's the purpose, couldn't you store the hash and throw away the compressed image? (As others said, compression is deterministic for the same algorithm, parameters and input data)

Zstd for example only promises determinism on the same version of the library. I've personally seen the hashes mutate between pull and export. Things like tar padding also make a difference. Really, the thing to do is to hash on the _uncompressed_ data and let compression be a transport/registry detail. That's what I've done, at least.

I didn't know that about zstd, that's a bit unfortunate.

Tar isn't related here though, we're talking about compression not archival formats

Re: Docker 29 has changed its default image store for new installs

#69

Post by nijave »

StableAlkyne wrote: post #31
spwa4 wrote: post #26

Earlier quoted context omitted.

If you have problems with 13 (I believe) GB of docker layers ... how do you deal with terabytes or petabytes of AI training data?

You don't even need MB of training data for some ML applications. AI is the sexy thing nowadays, but neural networks (Torch is a NN library) are generally useful for even small regression and clarification problems. For some problems you might even be able to get away with single digit numbers of training points (classic example of this regime being Physics-Informed Neural Networks)

Yeah, our handful of models we just commit to the git repo--usually only a few MB.

Image still ends up being like 6-8Gi tho. iirc pytorch had a hard dependency on CUDA libs which pulled in a bunch of different hardware-specific kernel binaries. The models ran on CPU and didn't even need CUDA but it was incredibly hard to remove them--there was some pytorch init code that expected the CUDA crap to exist even on CPU-only.

Re: Docker 29 has changed its default image store for new installs

#70

Post by flakes »

XYen0n wrote: post #60
flakes wrote: post #52

Earlier quoted context omitted.

Recompressing should be guaranteed deterministic. It’s the packing/unpacking of tar archives to/from directories on disk that leads to the non-determinism (such as timestamps and ownership metadata). If the tar is left intact, both zstd and gzip should produce byte for byte identical outputs given the same compression parameters.

You are correct; I confused archiving with compression. However, even considering only the compression process, same compression parameters cannot be guaranteed, as it is unknown which compression parameters the image publisher used.

Thats true. And regardless of compressed vs regular tar, I think the OCI format working with opaque archives is extremely limiting. I hope the industry will eventually redesign to use content addressable storage per file and have metadata to describe the layer/disk layout instead. That would allow per file deduplication, and we can use tar for just bulk transfer over the wire, rather than using tar for the data at rest.
Post reply on HN