Live data from Hacker News

We reduced a container image from 800GB to 2GB

sealos.io

51–60 of 88 posts

Re: We reduced a container image from 800GB to 2GB

#51

This seems very much like a ‘we mis configured our containers; then we realised, then we fixed it, then we blogged about it’ post of very little value.

Right? Blog posts like these makes me question competence instead of attributing it.

Competence comes from experience built on lots of screw-ups. I like when people blog their mistakes.

Re: We reduced a container image from 800GB to 2GB

#52
post #43
post #21

Earlier quoted context omitted.

Is it fascinating? Do people not know that each layer comes with its own downsides? Do people just do 272 layers and think that it’s normal? This seems like people discovering that water is wet and fire is hot.

I feel like I'm having a LLM fever dream

Seriously. Honestly this whole thing feels kinda like…using an LLM to write a blog post about debugging weird problems that only exist because the whole platform was built by an LLM in the first place. The multiple top level comments that are clearly written by an LLM are icing on the (layer) cake.

Re: We reduced a container image from 800GB to 2GB

#53
post #27

272 layers in a single image seems really unusual, is that just due to my lack of experience with containers? I've never seen an image with more than maybe a few dozen in my career...

I can't think of anything that would justify that many layers. If I have that much complexity, I would split up the container or start writing bash scripts.

The automation of containers looks simple but developers with systems experience know the actual complexity of operating systems and running applications.

People who know javascript but don't know how a file system works can build and deploy containers. They just copy and paste stuff until it runs. The automation of containers makes brute force iteration a viable option. It was a lot more difficult trying to run a Linux server, which would force you to learn something or use a platform as a service instead.

Re: We reduced a container image from 800GB to 2GB

#54
post #36

Our platform is designed to solve a very specific workflow, and the DevBox is only the first step in that process. Our users need to connect their local VS Code, Cursor, or JetBrains IDEs to the cloud environment. The industry-standard extensions for this only speak the SSH protocol. So, to give our users the tools they love, the container must run an SSHD to act as the host. We aren't just a CDE like Coder or Codesp…

Agree with other commenters that this seems like a bad idea. Why on earth should the release image contain all of the cruft of development?? Why on earth should it contain historical versions of all that cruft??

This does seem bonkers to me. All but guaranteed to have worse issues than bloated container images in the future.

Re: We reduced a container image from 800GB to 2GB

#55
post #30
post #29

Earlier quoted context omitted.

What if I need cron in my docker container? And ssh? And a text editor? And a monitoring agent? :P Thankfully LXD is here to serve this need: very lightweight containers for systems, where your app runs in a complete ecosystem, but very light on the ram usage.

>What if I need cron in my docker container? And ssh? And a text editor? And a monitoring agent? :P How are you going to orchestrate all those daemons without systemd? :P As you mentioned, a container running systemd and a suite of background services is the typical use case of LXD, not docker. But the difference seems to be cultural -- there's nothing preventing one from using systemd as the entry point of a docker…

fwiw I recently bootstrapped a small Debian image for myself, originally intended to sandbox coding agents I was evaluating. Shortly after I got annoyed by baseline vim and added my tmux & nvim dotfiles, now I find myself working inside the container regularly. It definitely works and is actually not the worst experience if your workflow is cli-focused.

Re: We reduced a container image from 800GB to 2GB

#56
post #28

I’m shocked that a company would share how amazingly bad their layer management had become. This may be a great internal blog, but I wouldn’t share it publicly.

From what I understood they provide a kind of shared platform where anyone can run things, and it was one of their clients/users performing the commits.

Re: We reduced a container image from 800GB to 2GB

#58
> image-manip squash: This is the key to reclaiming disk space and the core of our strategy to squash the image layers. The tool creates a temporary container, applies all 272 layers in sequence to an empty root filesystem, and then exports the final, merged filesystem as a single new layer. This flattens the image's bloated history into a lean, optimized final state.

Wouldn't a multistage Dockerfile have accomplished the same thing? smth like

FROM bigimage

RUN rm bigfile

FROM scratch

COPY --from=0 / /

Re: We reduced a container image from 800GB to 2GB

#59
post #47
post #28

I’m shocked that a company would share how amazingly bad their layer management had become. This may be a great internal blog, but I wouldn’t share it publicly.

I'm confused. I had the same initial reaction as you and then read further and it sounds like the image was actually provided by a client? > The problematic user image had an astonishing 272 layers, each representing a commit operation.

This sounds like a case of "We are in growth mode and will accept any garbage the customer will throw at us" without calculating the tech debt costs.

As someone who is currently there, it's very frustrating place.

Re: We reduced a container image from 800GB to 2GB

#60
post #49

The real lesson they should learn is to not rely on running images and then using "docker commit" to turn it into an image, but instead to use proper image building tools. If you absolutely have to do it that way, be very deliberate about what you actually need. Don't run an SSH daemon, don't run cron, don't an SMTP daemon, don't run the suite of daemons that run on a typical Linux server. Only run precisely what you…

My first reaction: 800GB who committed that?!? This size alone screams something is wrong. To be fair even with basic dockerfiles it’s easy to build up a lot of junk. But there should be a general size limit in any workflow that just alerts when something grows out of proportion. We had this in our shop just a few weeks ago. A docker image for some ai training etc grew too big and nobody got alerted about the image f…

Given that Jfrog bills on egress for these container images I’m sure you guys saw an eye watering bill for the privilege of distributing your bloated container
Post reply on HN