Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker 29 has changed its default image store for new installs

docs.docker.com

Post reply on HN
Print view View original

51–60 of 93 posts

Re: Docker 29 has changed its default image store for new installs

#51

Post by black3r »

Oxodao wrote: post #4

Docker already fills up my dev machines yet they decided for this insane solution: > The containerd image store uses more disk space than the legacy storage drivers for the same images. This is because containerd stores images in both compressed and uncompressed formats, while the legacy drivers stored only the uncompressed layers. Why ?

Also this doesn't just mean more disk space usage, but also longer local build times... for the app I'm working on `exporting to image` takes 71.5 seconds with containerd, without containerd it's 4.3s (the rest of the build takes ~180 seconds). And that's just a 5.76GB image.

Re: Docker 29 has changed its default image store for new installs

#52

Post by flakes »

XYen0n wrote: post #50
Filligree wrote: post #10

Earlier quoted context omitted.

Why would I need the compressed layers?

The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash

Recompressing should be guaranteed deterministic. It’s the packing/unpacking of tar archives to/from directories on disk that leads to the non-determinism (such as timestamps and ownership metadata). If the tar is left intact, both zstd and gzip should produce byte for byte identical outputs given the same compression parameters.

Re: Docker 29 has changed its default image store for new installs

#53

Post by eikenberry »

mgrandl wrote: post #38
nitinreddy88 wrote: post #8

Earlier quoted context omitted.

They are adopting to containerd standard, not sure why negative sentiment

Where did you see that? I just did a deep dive into podman/quadlets/bootc/composefs and never once seen a mention of that. A google search also didn’t bring anything like that up.

I think the "They" mentioned was Docker, not Podman. That Docker was adopting the containerd standard.

Re: Docker 29 has changed its default image store for new installs

#54

Post by [deleted] »

wolttam wrote: post #42
newsoftheday wrote: post #19

The article says to regularly run prune, how regularly? Currently I run the following once per day from cron: docker system prune -a -f docker volume prune -a -f

This would depend entirely on how much churn your system is doing on containers/volumes/images. Once a day sounds really often for most situations. "Regularly" = when you're running out of space because of a bunch of built up old stuff.

[deleted]

Re: Docker 29 has changed its default image store for new installs

#55

Post by tdemin »

0xbadcafebee wrote: post #33

It sounds like this breaks all Docker installs that use userns-remap? Are they really shipping a breaking change with no fix? In addition to bloating the disk? In addition to breaking all old systems that relied on mapping /var/lib/docker? I can't believe Docker finally shit the bed. Time to replace Docker with Podman.... sigh

> If you upgraded from an earlier version, your daemon continues using the legacy graph drivers (overlay2) until you enable the containerd image store.

Re: Docker 29 has changed its default image store for new installs

#56

Post by mgrandl »

eikenberry wrote: post #53
mgrandl wrote: post #38

Earlier quoted context omitted.

Where did you see that? I just did a deep dive into podman/quadlets/bootc/composefs and never once seen a mention of that. A google search also didn’t bring anything like that up.

I think the "They" mentioned was Docker, not Podman. That Docker was adopting the containerd standard.

Jup that’s definitely it, not sure how I got it that wrong.

Re: Docker 29 has changed its default image store for new installs

#57

Post by bravetraveler »

arnitdo wrote: post #23
newsoftheday wrote: post #19

The article says to regularly run prune, how regularly? Currently I run the following once per day from cron: docker system prune -a -f docker volume prune -a -f

From the docs, you can just run `docker system prune -a --volumes` Ref: https://docs.docker.com/reference/cli/docker/system/prune/

Personally, I'd recommend the pointed 'docker {container,image,volume} prune' commands for scheduling granularity/control. At least, filtering as you've also shown.

The 'system' context captures networks; much to my dismay, this has been a problem for no fewer than three employers. It's painfully common for things to expect the networks to persist. They don't really consume resources, so I see no reason to invite the systematic heartburn.

When? When there's disk pressure. Maybe some longer term (weekly, monthly?) to keep a lid on things. The image cache provides a benefit, no sense fighting it. At our rate, daily pruning means I might lose hours (through a week) repeatedly pulling the same images.

Re: Docker 29 has changed its default image store for new installs

#58

Post by mort96 »

XYen0n wrote: post #50
Filligree wrote: post #10

Earlier quoted context omitted.

Why would I need the compressed layers?

The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash

If that's the purpose, couldn't you store the hash and throw away the compressed image?

(As others said, compression is deterministic for the same algorithm, parameters and input data)

Re: Docker 29 has changed its default image store for new installs

#59

Post by mort96 »

NewJazz wrote: post #20
Filligree wrote: post #10

Earlier quoted context omitted.

Why would I need the compressed layers?

Pushing

What about pushing? Computers are fast enough to compress stuff as it's being transmitted, you don't need to store the compressed copy anywhere...

Re: Docker 29 has changed its default image store for new installs

#60

Post by XYen0n »

flakes wrote: post #52
XYen0n wrote: post #50

Earlier quoted context omitted.

The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash

Recompressing should be guaranteed deterministic. It’s the packing/unpacking of tar archives to/from directories on disk that leads to the non-determinism (such as timestamps and ownership metadata). If the tar is left intact, both zstd and gzip should produce byte for byte identical outputs given the same compression parameters.

You are correct; I confused archiving with compression. However, even considering only the compression process, same compression parameters cannot be guaranteed, as it is unknown which compression parameters the image publisher used.
Post reply on HN