Viewing profile — za_mike157

Thanks for this! We did see it and it was committed long after we already implemented this functionality. However, there are a lot of edge cases that this commit doesn't deal with …

[dead]

Interesting! I didn't see they released this. Do you know what their benchmarks are? I know for cloud run they are pretty slow

Us and the team from Modal have been upstreaming things to the GVisor repo ( https://github.com/google/gvisor/pulls ) in order to make it compatible with cuda-checkpoint and other …

haha you are right that the title is a bit strange - should just be "Reduce GPU cold starts with snapshotting" I can't read good ;)

No we don't use it. CRIU is used for normal checkpoint/restore of Linux processes. Since we run GVisor for container isolation we use their checkpoint/restore support for the sandb…

There are a lot of similarities. They run their snapshot agent as a Kubernetes DaemonSet, whereas our implementation runs as part of the Cerebrium container runtime path. Under the…

Hey! Yes you are correct! We have both been upstreaming changes to the main GVisor repo. However, in order to work within our own infrastructure we had to make various changes that…

Glad you liked it!

You are correct! From our tests, storing model weights in the image actually isn't a preferred approach for model weights larger than ~1GB. We run a distributed, multi-layer cache …

A lot of AI workloads require GPUs which are expensive so customers would waste money running idle machines 24/7 with low utilisation which kills gross margins. By loading containe…

Hey! Founder of Cerebrium here. - Runpod is one of the cheapest but it comes at the price of reliability (critical for businesses) - We have more performant cold start performance …

I haven't used SkyPilot so I am unfamiliar with the experience and performance. However, some of the situations you would like to use Cerebrium over Skypilot are: - You don't want …

I think we used this UI kit: https://minimals.cc/

I guess then the next question would be how quickly can they start executing your container from cold start when a workload comes in? Typically we see companies on around 30-60s

Do you mean why the individual file names aren't quoted? You can see an example config file at the bottom of that link you attached - agreed we should probably make it more obvious…

Thanks for confirming! Our cold start, excluding model load is 2-4 seconds typically for HF models. The only time it gets much longer when companies have done a lot with very speci…

Thanks Tom! Excited to to support you and the team as you grow

Ah I see they recently cut their pricing by 40% so you are correct - sorry about that. It seems we are more expensive compared to their new pricing