Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced a container image from 800GB to 2GB

sealos.io

Post reply on HN
Print view View original

11–20 of 88 posts

Re: We reduced a container image from 800GB to 2GB

#11

Post by cowsandmilk »

trenchpilgrim wrote: post #8
reddozen wrote: post #6

Is it spooky that they said they looked inside a customer's image to fix this? A bunch of engineers just had access to their customer's intellectual property, security keys, git repos, ...

If you are adding security keys and git repos to your final shipped image you are doing things very wrong - a container image is literally a tarball and some metadata about how to run the executables inside. Even if you need that data to build your application you should use a multi-stage build to include only the final artifacts in the image you ship. For stuff like security keys you should typically add them as bui…

That’s nice, but you still shouldn’t be looking into your customer’s containers.

Re: We reduced a container image from 800GB to 2GB

#12

Post by adastra22 »

cowsandmilk wrote: post #11
trenchpilgrim wrote: post #8

Earlier quoted context omitted.

If you are adding security keys and git repos to your final shipped image you are doing things very wrong - a container image is literally a tarball and some metadata about how to run the executables inside. Even if you need that data to build your application you should use a multi-stage build to include only the final artifacts in the image you ship. For stuff like security keys you should typically add them as bui…

That’s nice, but you still shouldn’t be looking into your customer’s containers.

How else do they diagnose issues? Sorry to break it to you, this is absolutely standard across the entire industry.

Re: We reduced a container image from 800GB to 2GB

#14

Post by Ysx »

trenchpilgrim wrote: post #8
reddozen wrote: post #6

Is it spooky that they said they looked inside a customer's image to fix this? A bunch of engineers just had access to their customer's intellectual property, security keys, git repos, ...

If you are adding security keys and git repos to your final shipped image you are doing things very wrong - a container image is literally a tarball and some metadata about how to run the executables inside. Even if you need that data to build your application you should use a multi-stage build to include only the final artifacts in the image you ship. For stuff like security keys you should typically add them as bui…

> For stuff like security keys you should typically add them as build args, not as content in the image.

Build args are content in the image: https://docs.docker.com/reference/build-checks/secrets-used-...

Re: We reduced a container image from 800GB to 2GB

#15

Post by KronisLV »

Images don't seem to be working:

https://sealos.io/_next/image?url=.%2Fimages%2Fcontainerd-hi...

https://sealos.io/_next/image?url=.%2Fimages%2Fbloated-conta...

Either way, hope the user was communicated with or alerted to what's going on.

At the same time, someone said that 800 GB container images are a problem in of themselves no matter the circumstances and they got downvoted for saying so - yet I mostly agree.

Most of mine are about 50-250 MB at most and even if you need big ones with software that's GB in size, you will still be happier if you treat them as something largely immutable. I've never had odd issues with them thanks to this. If you really care about data persistence, then you can use volumes/bind mounts or if you don’t then just throw things into tmpfs.

I'm not sure whether treating containers as something long lived with additional commmits/layers is a great idea, but if it works for other people, then good for them. Must be a pain to run something so foundational for your clients, though, cause you'll be exposed to most of the edge cases imaginable sooner or later.

Re: We reduced a container image from 800GB to 2GB

#16

Post by BinaryIgor »

Interesting, although something about the language makes me think it was written by a LLM; I like the ending though:

"The key insight is to treat container images not as opaque black boxes, but as structured, manipulable archives. Deeply understanding the underlying technology, like the OCI image specification, allows for advanced optimization and troubleshooting that goes far beyond standard tooling. This knowledge is essential for preventing issues like Kubernetes disk space exhaustion before they start."

Re: We reduced a container image from 800GB to 2GB

#17

Post by hiddew »

trenchpilgrim wrote: post #8
reddozen wrote: post #6

Is it spooky that they said they looked inside a customer's image to fix this? A bunch of engineers just had access to their customer's intellectual property, security keys, git repos, ...

If you are adding security keys and git repos to your final shipped image you are doing things very wrong - a container image is literally a tarball and some metadata about how to run the executables inside. Even if you need that data to build your application you should use a multi-stage build to include only the final artifacts in the image you ship. For stuff like security keys you should typically add them as bui…

> For stuff like security keys you should typically add them as build args, not as content in the image.

Do not use build arguments for anything secret. The values are committed into the image layers.

Re: We reduced a container image from 800GB to 2GB

#18

Post by tecleandor »

trenchpilgrim wrote: post #8
reddozen wrote: post #6

Is it spooky that they said they looked inside a customer's image to fix this? A bunch of engineers just had access to their customer's intellectual property, security keys, git repos, ...

If you are adding security keys and git repos to your final shipped image you are doing things very wrong - a container image is literally a tarball and some metadata about how to run the executables inside. Even if you need that data to build your application you should use a multi-stage build to include only the final artifacts in the image you ship. For stuff like security keys you should typically add them as bui…

Yeah, typically, but in this case they're commiting and commiting in the container image, and saving changes from running software. Not only that, they're commiting log files into the image, which is crazy.

The thing here is they're using Docker container images like if they were VM disks and they end up with images with almost 300 layers, like in this case. I think LXC or VMs should be a better case for this (but I don't know if they've tested it or why are they using Docker)

Re: We reduced a container image from 800GB to 2GB

#20

Post by perlgeek »

The real lesson they should learn is to not rely on running images and then using "docker commit" to turn it into an image, but instead to use proper image building tools.

If you absolutely have to do it that way, be very deliberate about what you actually need. Don't run an SSH daemon, don't run cron, don't an SMTP daemon, don't run the suite of daemons that run on a typical Linux server. Only run precisely what you need to create the files that you need for a "docker commit".

Each service that you run can potentially generate log files, lock files, temp files, named pipes, unix sockets and other things you don't want in your image.

Taking a snapshot from a working, regular VM and using that as a docker image is one of the worst ways to built one.

Post reply on HN