11–20 of 22 posts
Somehow, they don't hit upon the solution other organizations use: having software running all the time.
I suppose if you have a lousy economic model where the cost of running your software is a large percentage of your overall costs, that's a problem. I can only advise them to move to a model where they provide more value for their clients.
The problem: "containers that take far too long to start". Somehow, they don't hit upon the solution other organizations use: having software running all the time. I suppose if you have a lousy economic model where the cost of running your software is a large percentage of your overall costs, that's a problem. I can only advise them to move to a model where they provide more value for their clients.
This is successful for CPU workloads (AWS Lambda) but AI models and images are 50x the size
Why does the model data need to be stored in the image? Download the model data on container startup using whatever method works best.
even the smaller nvidia images (like nvidia/cuda:13.1.1-cudnn-runtime-ubuntu24.04) are about 2Gb before adding any python deps and that is a problem.
if you split the image into chunks and pull on-demand, your container will start much faster.
Why does the model data need to be stored in the image? Download the model data on container startup using whatever method works best.
Super helpful.
The problem: "containers that take far too long to start". Somehow, they don't hit upon the solution other organizations use: having software running all the time. I suppose if you have a lousy economic model where the cost of running your software is a large percentage of your overall costs, that's a problem. I can only advise them to move to a model where they provide more value for their clients.
A lot of AI workloads require GPUs which are expensive so customers would waste money running idle machines 24/7 with low utilisation which kills gross margins. By loading containers quickly means, means we can scale up quickly as requests come in and you only need to pay for usage. This is successful for CPU workloads (AWS Lambda) but AI models and images are 50x the size
They say an ideal container system would download portions of layers on demand, however is seems far from ideal for many production workloads. What if your service starts, works fine for an hour, then needs to read one file that is only available over the network, but that endpoint is unreachable? What if it is reachable but it is very very slow? The current system has issues with network stuff, but in a deploy proce…
In practice these systems typically fetch data over a local, highly available network and aggressively cache anything that gets read. If that network path becomes unavailable, it usually indicates a much larger infrastructure issue since many other parts of the system rely on the same storage or registry endpoints.
So while it does introduce a different failure mode, in most production environments it ends up being a low practical risk compared to the startup latency improvements.
For us and our customers, the trade off is worth it.
I remember dealing with this BS back in 2017. It was clear to me that containers were, more than anything else, a system for turning 15MB of I/O into 15GB of I/O. So wow and new shiny though so if you told people that they would just plug their ears with their fingers.
This doesn't follow from anything in the article.
I ran into a similar issue years ago, where the base infrastructure occupied the lion's share of the container size, very similar to the sizes shown in the article: Ubuntu base ~29 MB compressed PyTorch + CUDA 7 – 13 GB NVIDIA NGC 4.5+ GB compressed The easy solution that worked for us was to bake all of these into a single base container, and force all production containers built within the company to use that base.…
Where it starts to get harder is when you have multiple base stacks (different CUDA versions, frameworks, etc.) or when you need to update them frequently. You end up with lots of slightly different multi-GB bases.
Chunked images keep the benefit you mentioned (we still cache heavily on the nodes) but the caching happens at a finer granularity. That makes it much more tolerant to small differences between images and to frequent updates, since unchanged chunks can still be reused.