11–20 of 23 posts
Earlier quoted context omitted.
How can you cut latency by more than 1x? I am no intending to be snarky, it just doesn’t fit my brain how you can reduce a measure time by more than the original starting time.
probably just AI slop and using wrong semantics, they mean speedup ratio.
I've recently been going down the rabbit hole of creating a "fast start dev env" and it's interesting to see how this article differs from other approaches (codesandbox has some fantastic blogs, the fly.io blog on sprites has interesting pointers, e2b and daytona are related open source tools). Everyone has a different solution based on their tradeoffs. I thought the memory snapshotting part in particular was clever…
There are plenty of cool advancements in reducing inference cold start when I was meeting with folks in person at FOSDEM this year. However, I still struggle to understand: why would folks care about this? Major AI Labs all have secured their own compute in the form of hardware, data center, and power generation. That means their resource pool is fixed, and they can do all sorts of tricks to pre-load, pre-allocate, e…
I've recently been going down the rabbit hole of creating a "fast start dev env" and it's interesting to see how this article differs from other approaches (codesandbox has some fantastic blogs, the fly.io blog on sprites has interesting pointers, e2b and daytona are related open source tools). Everyone has a different solution based on their tradeoffs. I thought the memory snapshotting part in particular was clever…
Our system somewhat predates ublk; at the time we wrote it, FUSE was the most reasonable option. Moving over to ublk would require re-architecting around block devices rather than filesystems, but is indeed something we're looking at!
Edit: "The solution is to disaggregate the container launcher (runc for Docker, runsc for gVisor) from the container image delivery" is exactly what I've done! I've not built a lazy FUSE on top of it (yet! except for cache mounts in BuildKit), but it's on my TODO list. I guess I'm mainly curious what stops bytes from being shared in your case.
There are plenty of cool advancements in reducing inference cold start when I was meeting with folks in person at FOSDEM this year. However, I still struggle to understand: why would folks care about this? Major AI Labs all have secured their own compute in the form of hardware, data center, and power generation. That means their resource pool is fixed, and they can do all sorts of tricks to pre-load, pre-allocate, e…
charles, amit, can you go into more about the path based caching? Particularly "shared bytes aren’t guaranteed to be in the exact same container image layer"? I've built something that solves issues around sharing data between layers, and am interested to see if it fits usecases like Modal's. Edit: "The solution is to disaggregate the container launcher (runc for Docker, runsc for gVisor) from the container image del…
FROM some/image RUN pip install torch==2.7.1
and
FROM another/image RUN pip install torch==2.7.1
will produce images with very high overlap in contents, which will be shared by a content-based cache, but those images' final layers are disjoint from the perspective of a layerwise cache.
What is "cutting by 40x" supposed to mean?
There are plenty of cool advancements in reducing inference cold start when I was meeting with folks in person at FOSDEM this year. However, I still struggle to understand: why would folks care about this? Major AI Labs all have secured their own compute in the form of hardware, data center, and power generation. That means their resource pool is fixed, and they can do all sorts of tricks to pre-load, pre-allocate, e…
Here's my 2cents: improve cold starts also means utilizing resources more effectively.
From cloud providers to end users - every ms both adds up and translates to additional waste of electricity/hardware and costs.
charles, amit, can you go into more about the path based caching? Particularly "shared bytes aren’t guaranteed to be in the exact same container image layer"? I've built something that solves issues around sharing data between layers, and am interested to see if it fits usecases like Modal's. Edit: "The solution is to disaggregate the container launcher (runc for Docker, runsc for gVisor) from the container image del…
To clarify: we do content-based hashing, and when we say "shared bytes aren’t guaranteed to be in the exact same container image layer", what we mean is that FROM some/image RUN pip install torch==2.7.1 and FROM another/image RUN pip install torch==2.7.1 will produce images with very high overlap in contents, which will be shared by a content-based cache, but those images' final layers are disjoint from the perspecti…