41–50 of 88 posts
(And indeed, the images are broken in Firefox and Edge. Is there another browser where they're not broken?)
272 layers in a single image seems really unusual, is that just due to my lack of experience with containers? I've never seen an image with more than maybe a few dozen in my career...
> Here's how the disaster unfolded:
> 1. A user's container is under a brute-force attack, and /var/log/btmp grows to 11GB.
> 2. The user performs a commit, creating a new image layer.
> 3. A single new failed login is appended to /var/log/btmp.
> 4. Because of CoW, OverlayFS doesn't just write the new line. It copies the entire 11GB file into the new, upper layer.
> 5. This process repeated 271 times.
So the user is creating hundreds of layers for unclear reasons. The article refers to this as "exponential growth", but for that to be the case those commits would need to be triggered in proportion to the number of existing layers, which seems unlikely. Assuming the commits are caused by the user for reasons unrelated to the size of the existing image, this is growth that is quadratic† (in the number of layers; it's hard to characterize as a function of time or whatever), and it'd be nice to know why there were so many layers.
† Note that while the growth is technically quadratic, I don't think that impacted them. They say that the problem occurred when one 11GB file got copied into each of 272 image layers. That would require 2,992 GB, but they also say that the image exhibiting this problem was only 800GB.
I suspect that the answer here is that only some of the layers modified (and therefore copied) the log file. Probably about 72 of the layers. This is more like growth that's linear (still technically slightly superlinear, but probably not quadratic) in the number of failed SSH login attempts. ~75% of layers aren't contributing to the problem at all.
Fascinating deep dive into OverlayFS CoW behavior. The 11GB btmp file getting copied 271 times is a perfect storm scenario. Did they consider mounting /var/log outside the image layers? Seems like that would prevent any log file from causing this amplification. Also interested in image-manip... Does it handle metadata differently than docker export/import?
Is it fascinating? Do people not know that each layer comes with its own downsides? Do people just do 272 layers and think that it’s normal? This seems like people discovering that water is wet and fire is hot.
Interesting, although something about the language makes me think it was written by a LLM; I like the ending though: "The key insight is to treat container images not as opaque black boxes, but as structured, manipulable archives. Deeply understanding the underlying technology, like the OCI image specification, allows for advanced optimization and troubleshooting that goes far beyond standard tooling. This knowledge…
Not a guaranteed tell but I noticed the word "surgically" in the opening paragraphs, and from personal experience I find that appears in ChatGPT a lot for me. One of the common phrase tropes I find is something like "Here's a set of small, surgical steps you can take to..."
I’m shocked that a company would share how amazingly bad their layer management had become. This may be a great internal blog, but I wouldn’t share it publicly.
On the other hand, I'm impressed that a company is owning up to the problem. Is it a dumb problem to have? Definitely. Are they the only ones to have it? Almost certainly not. People are going to use the tools at their disposal, and they aren't all going to learn their tools at a high level. Think of every insane misuse of Excel you've ever heard of, for instance. IT has the choice in this case to mitigate, or limit…
272 layers in a single image seems really unusual, is that just due to my lack of experience with containers? I've never seen an image with more than maybe a few dozen in my career...
That is clearly not what these people are doing, though.
I’m shocked that a company would share how amazingly bad their layer management had become. This may be a great internal blog, but I wouldn’t share it publicly.
> The problematic user image had an astonishing 272 layers, each representing a commit operation.Our platform is designed to solve a very specific workflow, and the DevBox is only the first step in that process. Our users need to connect their local VS Code, Cursor, or JetBrains IDEs to the cloud environment. The industry-standard extensions for this only speak the SSH protocol. So, to give our users the tools they love, the container must run an SSHD to act as the host. We aren't just a CDE like Coder or Codesp…
The real lesson they should learn is to not rely on running images and then using "docker commit" to turn it into an image, but instead to use proper image building tools. If you absolutely have to do it that way, be very deliberate about what you actually need. Don't run an SSH daemon, don't run cron, don't an SMTP daemon, don't run the suite of daemons that run on a typical Linux server. Only run precisely what you…
TIL of `docker commit`. What is the use case for this command? Quick debugging or something, to share with a coworker?
Same as snapshotting a vm, or as an interactive version of "docker build". But rarely useful, since most workflows don't really need statefulness.