Okay, I'll ask the dumb question: Couldn't you also reduce the number of layers per container? Sure, if you can reuse layers you should, but unless you've done something very clever like 1 package per layer I struggle to think that 50 is really useful?
Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
21–30 of 35 posts
Re: Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
#22Okay, I'll ask the dumb question: Couldn't you also reduce the number of layers per container? Sure, if you can reuse layers you should, but unless you've done something very clever like 1 package per layer I struggle to think that 50 is really useful?
It's hardly surprising that companies consider infrastructure-level solutions to be better.
Re: Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
#23Okay, I'll ask the dumb question: Couldn't you also reduce the number of layers per container? Sure, if you can reuse layers you should, but unless you've done something very clever like 1 package per layer I struggle to think that 50 is really useful?
Its not a dumb question. It seems like when it comes to these supposed high tech enterprise solutions, they spend so much churn in doing something that is very complex and impressive like investigating architecture performance when it comes to kernel level operations and figuring out the kernel specifics that are causing slowdowns. Instead they can put that talent into just writing software without containers that ca…
Re: Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
#24Why is this so badly AI written? Netflix can surely pay for writers. At this point I refuse to read any content in the AI format of: - The problem - The solution - Why it matters
Re: Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
#25Earlier quoted context omitted.
> unless you've done something very clever like 1 package per layer I struggle to think that 50 is really useful? 1 package per layer can actually be quite nice, since it means that any package updates will only affect that layer, meaning that downloading container updates will use much less network bandwidth. This is nice for things like bootc [0] that are deployed on the "edge", but less useful for things deployed…
It doesn't work this way really? It's called a layer because each layer on top depends on the layers below. If you change the package defined in the bottom most layer, all 49 above it are invalid and need re-pulled or re-built.
Docker exploits this to figure out when it can cache a layer, but building a container is different than running one because changing the underlying file system can change what a command outputs. If you're running a container, changing one deeply buried layer doesn't change the layers above it because they're already saved.
Re: Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
#26Re: Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
#27Interesting, another case of removing HT improving performance. Reminds me of doing that on Intel CPUs of a few gens ago.
Re: Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
#28Re: Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
#29Interesting, another case of removing HT improving performance. Reminds me of doing that on Intel CPUs of a few gens ago.
It's quite logical that by saturating your CPU it can only decrease performance.
I wonder if simply setting a maximum number of concurrent mounts in the code or by letting containerd think there are only half the amount of cores, would have solved the contention to the same amount.
Re: Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
#30I am not familiar with the nitty gritty of container instance building process, so maybe I'm just not the intended audience, but this is particularly unclear to me: > To avoid the costly process of untarring and shifting UIDs for every container, the new runtime uses the kernel’s idmap feature. This allows efficient UID mapping per container without copying or changing file ownership, which is why containerd performs…
This kind of id mapping works as a mount option (it can also be used on bind mounts). You give it a mapping of "id in filesystem on disk" to "id to return to filesystem APIs" and it's all translated on the fly.