Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced a container image from 800GB to 2GB

sealos.io

Post reply on HN
Print view View original

71–80 of 88 posts

Re: We reduced a container image from 800GB to 2GB

#71

Post by otterley »

untrimmed wrote: post #2

Our platform is designed to solve a very specific workflow, and the DevBox is only the first step in that process. Our users need to connect their local VS Code, Cursor, or JetBrains IDEs to the cloud environment. The industry-standard extensions for this only speak the SSH protocol. So, to give our users the tools they love, the container must run an SSHD to act as the host. We aren't just a CDE like Coder or Codesp…

Who are you, exactly? There is practically no publicly available information about your company, other than that it appears to be held by a Chinese entity called Labring.

Re: We reduced a container image from 800GB to 2GB

#72

Post by larusso »

SOLAR_FIELDS wrote: post #60
larusso wrote: post #49

Earlier quoted context omitted.

My first reaction: 800GB who committed that?!? This size alone screams something is wrong. To be fair even with basic dockerfiles it’s easy to build up a lot of junk. But there should be a general size limit in any workflow that just alerts when something grows out of proportion. We had this in our shop just a few weeks ago. A docker image for some ai training etc grew too big and nobody got alerted about the image f…

Given that Jfrog bills on egress for these container images I’m sure you guys saw an eye watering bill for the privilege of distributing your bloated container

Yes. But fair enough that we got a warning the very next day.

Re: We reduced a container image from 800GB to 2GB

#73

Post by sally_glance »

SOLAR_FIELDS wrote: post #61
sally_glance wrote: post #55

Earlier quoted context omitted.

fwiw I recently bootstrapped a small Debian image for myself, originally intended to sandbox coding agents I was evaluating. Shortly after I got annoyed by baseline vim and added my tmux & nvim dotfiles, now I find myself working inside the container regularly. It definitely works and is actually not the worst experience if your workflow is cli-focused.

My experience is if the tooling is set up right it’s not painful, it’s the fiddling around with volume mounts folder permissions and debug points and “what’s inside the container and what isn’t” etc that is always the big pain point

Very accurate - that was one of the steps that caused me to fiddle quite a bit. Had to add an entrypoint to chown the mounts and also some Buildkit cache volumes for all the package managers.

You can skip the uid/chown stuff if you work with userns mappings, but this was my work machine so I didn't want to globally touch the docker daemon.

Re: We reduced a container image from 800GB to 2GB

#74

Post by mystifyingpoi »

gkfasdfasdf wrote: post #58

> image-manip squash: This is the key to reclaiming disk space and the core of our strategy to squash the image layers. The tool creates a temporary container, applies all 272 layers in sequence to an empty root filesystem, and then exports the final, merged filesystem as a single new layer. This flattens the image's bloated history into a lean, optimized final state. Wouldn't a multistage Dockerfile have accomplishe…

I think yep, pretty much. Maybe they didn't know this existed?

Re: We reduced a container image from 800GB to 2GB

#75

Post by adastra22 »

stackskipton wrote: post #64
adastra22 wrote: post #12

Earlier quoted context omitted.

How else do they diagnose issues? Sorry to break it to you, this is absolutely standard across the entire industry.

Evict the containers, let the customer know and get customer approval to work with their images.

You have approval in the terms of service. This is absolutely known and expected across the entire industry. It's why your employees have clauses in their contracts about respecting third party confidentiality.

Re: We reduced a container image from 800GB to 2GB

#76

Post by sally_glance »

stackskipton wrote: post #62
sally_glance wrote: post #56

Earlier quoted context omitted.

From what I understood they provide a kind of shared platform where anyone can run things, and it was one of their clients/users performing the commits.

So they don't set reasonable expectations with the customers and accept any and all garbage. As Ops person, this is a path to Ops hell as customers throw more and more garbage at you and toil dealing with customer problems becomes unbearable. This is a case of Product Team not working with customers, finding out what is reasonable and allowing system to set reasonable limits.

I would give them some leeway, sometimes you have to learn the hard way. But I was also kind of surprised didn't mention contacting the client anywhere.

Re: We reduced a container image from 800GB to 2GB

#77

Post by rekabis »

Defence № 2 and № 3 are ones I would do everywhere as a knee-jerk reaction, regardless of any justification to not bother with them. It’s just an ingrained habit at this point.

It’s № 1 which I could not have guessed at or gone for. Good write-up, love the transparency.

Re: We reduced a container image from 800GB to 2GB

#78

Post by rekabis »

cebert wrote: post #28

I’m shocked that a company would share how amazingly bad their layer management had become. This may be a great internal blog, but I wouldn’t share it publicly.

Transparency breeds trust.

Sure, it frightens away the short-sighted or particularly excitable people, but anyone who understands how unrealistic perfection is will be comforted by such transparency. Exposing the warts not only sets expectations, but it also assures people that things will (likely) not be just swept under the rug in a company culture of denialism and obfuscation.

Re: We reduced a container image from 800GB to 2GB

#79

Post by solatic »

Reliable systems require hard limits imposed by designers. When systems hit the hard limits, it's a sign somebody's assumptions are wrong: either the designer built too small, or there's some bug pushing up against the hard limit. Either you catch the bug or make an intentional decision on how to scale further. This is basic engineering and is a requisite part of any undergraduate engineering degree worth its salt.

Allowing eight hundred gigabyte containers is gross incompetence. Trying to fix it by scaling the node disk from 2 TB to 2.5 TB is further evidence of incompetence. Understanding that you need to build a hard cap, but not concluding with action items to actually build one - instead just building monitoring for image size - is a clear sign to stay away.

It boggles my mind that the author could understand copy-on-write filesystem semantics but can't imagine how to engineer actual size limits on said filesystem. How is that possible?

.... oh right, the blogpost is LLM slop. So nobody knows what the author actually learned.

Post reply on HN