Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced a container image from 800GB to 2GB

sealos.io

Post reply on HN
Print view View original

61–70 of 88 posts

Re: We reduced a container image from 800GB to 2GB

#61

Post by SOLAR_FIELDS »

sally_glance wrote: post #55
curt15 wrote: post #30

Earlier quoted context omitted.

>What if I need cron in my docker container? And ssh? And a text editor? And a monitoring agent? :P How are you going to orchestrate all those daemons without systemd? :P As you mentioned, a container running systemd and a suite of background services is the typical use case of LXD, not docker. But the difference seems to be cultural -- there's nothing preventing one from using systemd as the entry point of a docker…

fwiw I recently bootstrapped a small Debian image for myself, originally intended to sandbox coding agents I was evaluating. Shortly after I got annoyed by baseline vim and added my tmux & nvim dotfiles, now I find myself working inside the container regularly. It definitely works and is actually not the worst experience if your workflow is cli-focused.

My experience is if the tooling is set up right it’s not painful, it’s the fiddling around with volume mounts folder permissions and debug points and “what’s inside the container and what isn’t” etc that is always the big pain point

Re: We reduced a container image from 800GB to 2GB

#62

Post by stackskipton »

sally_glance wrote: post #56
cebert wrote: post #28

I’m shocked that a company would share how amazingly bad their layer management had become. This may be a great internal blog, but I wouldn’t share it publicly.

From what I understood they provide a kind of shared platform where anyone can run things, and it was one of their clients/users performing the commits.

So they don't set reasonable expectations with the customers and accept any and all garbage. As Ops person, this is a path to Ops hell as customers throw more and more garbage at you and toil dealing with customer problems becomes unbearable.

This is a case of Product Team not working with customers, finding out what is reasonable and allowing system to set reasonable limits.

Re: We reduced a container image from 800GB to 2GB

#63

Post by ndsipa_pomu »

cebert wrote: post #28

I’m shocked that a company would share how amazingly bad their layer management had become. This may be a great internal blog, but I wouldn’t share it publicly.

It's like a car repair company sharing how they dramatically improved ride comfort, speed and fuel usage by using air to fill tyres rather than concrete.

Re: We reduced a container image from 800GB to 2GB

#64

Post by stackskipton »

adastra22 wrote: post #12
cowsandmilk wrote: post #11

Earlier quoted context omitted.

That’s nice, but you still shouldn’t be looking into your customer’s containers.

How else do they diagnose issues? Sorry to break it to you, this is absolutely standard across the entire industry.

Evict the containers, let the customer know and get customer approval to work with their images.

Re: We reduced a container image from 800GB to 2GB

#65

Post by collingreen »

ndsipa_pomu wrote: post #63
cebert wrote: post #28

I’m shocked that a company would share how amazingly bad their layer management had become. This may be a great internal blog, but I wouldn’t share it publicly.

It's like a car repair company sharing how they dramatically improved ride comfort, speed and fuel usage by using air to fill tyres rather than concrete.

After asking chatgpt for suggestions and trying them all.

Re: We reduced a container image from 800GB to 2GB

#66

Post by ndsipa_pomu »

SurceBeats wrote: post #9

Fascinating deep dive into OverlayFS CoW behavior. The 11GB btmp file getting copied 271 times is a perfect storm scenario. Did they consider mounting /var/log outside the image layers? Seems like that would prevent any log file from causing this amplification. Also interested in image-manip... Does it handle metadata differently than docker export/import?

This is less of a deep dive and more an illustration of the worst way to use containers.

Having /var/log set as as a persistent volume would have worked, but ultimately they were using "docker commit" to amend/update their images which is definitely the wrong way to do it.

Re: We reduced a container image from 800GB to 2GB

#67

Post by trenchpilgrim »

stackskipton wrote: post #64
adastra22 wrote: post #12

Earlier quoted context omitted.

How else do they diagnose issues? Sorry to break it to you, this is absolutely standard across the entire industry.

Evict the containers, let the customer know and get customer approval to work with their images.

What about this case where the container was working but was consuming overhead due to an infrastructure issue? Customer hasn't done anything wrong. If you stop their containers they'll likely leave for a competitor.

Re: We reduced a container image from 800GB to 2GB

#69

Post by move-on-by »

stackskipton wrote: post #59
cedws wrote: post #47

Earlier quoted context omitted.

I'm confused. I had the same initial reaction as you and then read further and it sounds like the image was actually provided by a client? > The problematic user image had an astonishing 272 layers, each representing a commit operation.

This sounds like a case of "We are in growth mode and will accept any garbage the customer will throw at us" without calculating the tech debt costs. As someone who is currently there, it's very frustrating place.

Oh just wait till it’s time for your company to stop the ‘growth mode’ shenanigans and get serious about acceptable levels of tech debt and feature bloat. It’s where we are.

You can’t just flip a switch. There is no “Hey, that was fun, but it’s time to start designing these things with a purpose and vision”. Beyond the totally unreasonable expectations that have been set by Product and C-level- you still have the mountain of tech debt that is coming due and changes slow to a crawl or outages skyrocket or both. Plus, hiring has been based on ‘getting things done’, so you have this group of people who are actually really skilled in hacking things together and getting it out the door. It’s tough and calls for an entire culture shift. How do you stop being a reactionary startup and become vision-based and purposeful organization?

Re: We reduced a container image from 800GB to 2GB

#70

Post by otterley »

reddozen wrote: post #6

Is it spooky that they said they looked inside a customer's image to fix this? A bunch of engineers just had access to their customer's intellectual property, security keys, git repos, ...

I did a little research on this company. It’s related to (or wholly owned by) a Chinese entity called Labring. LinkedIn shows practically nobody related to the company other than its marketing team. Something smells incredibly fishy.
Post reply on HN