Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker 29 has changed its default image store for new installs

docs.docker.com

Post reply on HN
Print view View original

41–50 of 93 posts

Re: Docker 29 has changed its default image store for new installs

#41

Post by awestroke »

ElevenLathe wrote: post #5
Oxodao wrote: post #4

Docker already fills up my dev machines yet they decided for this insane solution: > The containerd image store uses more disk space than the legacy storage drivers for the same images. This is because containerd stores images in both compressed and uncompressed formats, while the legacy drivers stored only the uncompressed layers. Why ?

Sounds like a straightforward time-space tradeoff: if you have the compressed layers sitting around when you need them, you can avoid the expense and time of compressing them.

But if it stores the uncompressed layers, why store the compressed ones too? Why both at the same time?

Re: Docker 29 has changed its default image store for new installs

#42

Post by wolttam »

newsoftheday wrote: post #19

The article says to regularly run prune, how regularly? Currently I run the following once per day from cron: docker system prune -a -f docker volume prune -a -f

This would depend entirely on how much churn your system is doing on containers/volumes/images. Once a day sounds really often for most situations.

"Regularly" = when you're running out of space because of a bunch of built up old stuff.

Re: Docker 29 has changed its default image store for new installs

#43

Post by 0cf8612b2e1e »

spwa4 wrote: post #26
epistasis wrote: post #16

Earlier quoted context omitted.

This is hell for a lot of ML containers, that have gigabytes of CUDA and PyTorch. Before at least you could keep your code contained to a layer. But if I understand this correctly every code revision duplicates gigabytes of the same damn bloated crap.

If you have problems with 13 (I believe) GB of docker layers ... how do you deal with terabytes or petabytes of AI training data?

You don’t train petabytes on your laptop.

Re: Docker 29 has changed its default image store for new installs

#44

Post by newsoftheday »

stingraycharles wrote: post #25
newsoftheday wrote: post #17

Earlier quoted context omitted.

I had to work on a Mac M3 for a year, it sucked, it did not feel snappier than any Windows or Linux machine (including this one) that I've ever used and that is going back to the 1980's.

I suggest you judge based on benchmarks rather than vibes. If you believe the latest M3 does not perform better than machines you’ve used in the 80s, I have no idea how to even start a reasonable discussion about this.

> If you believe the latest M3 does not perform better than machines you’ve used in the 80s

That wasn't what I was trying to say, I apologize, I should have been clearer. What I intended to say was that I've been using various, many computers since the 1980's so I have a wide and deep sampling of experiences with them and to that end...the M3 did NOT feel to me like it performed better. Regardless the benchmarks, I know how the machine should feel and I know M3 did not feel any better than any other machine I've used (and that is a lot of laptops).

Re: Docker 29 has changed its default image store for new installs

#45

Post by hommelix »

> Docker Engine includes an experimental feature that can automatically switch to the containerd image store under certain conditions. This feature is experimental. It's provided for those who want to test it, but starting fresh is the recommended approach.

How bad did we fall with the ship often, ship early and fix later idea? Make a major change, release it and the migration feature is experimental and not recommended.

Re: Docker 29 has changed its default image store for new installs

#48

Post by embedding-shape »

fabian2k wrote: post #9

> This difference is particularly noticeable with multiple images sharing the same base layers. With legacy storage drivers, shared base layers were stored once locally, and reused images that depended on them. With containerd, each image stores its own compressed version of shared layers, even though the uncompressed layers are still de-duplicated through snapshotters. This seems like a really weird decision. If bas…

"really weird decision" seems like an understatement, I thought the entire point of the specific storage design with the whole layering shebang was so things could be shared? If you remove that, just get rid of layers as a whole, what's the point otherwise?

Re: Docker 29 has changed its default image store for new installs

#49

Post by kevinmgranger »

johannes1234321 wrote: post #36
giobox wrote: post #29

Earlier quoted context omitted.

> https://docs.docker.com/reference/cli/docker/system/prune/ Just in case - I'm always amazed how many Docker users don't know about the prune command for cleaning up the caches and deleting unused container images and just slowly let their docker image cache eat their disk.

Prune is nice, but if you have a bunch of containers which run shirt time for a build step or similar prune would collect those, too. A filter "last used a few months ago" would be useful.

Any reason why those containers can't be run ephemerally?

Re: Docker 29 has changed its default image store for new installs

#50

Post by XYen0n »

Filligree wrote: post #10
ElevenLathe wrote: post #5

Earlier quoted context omitted.

Sounds like a straightforward time-space tradeoff: if you have the compressed layers sitting around when you need them, you can avoid the expense and time of compressing them.

Why would I need the compressed layers?

The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash
Post reply on HN