Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reduce GVisor Cold Starts with GPU Snapshotting

cerebrium.ai

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Reduce GVisor Cold Starts with GPU Snapshotting

#11

Post by eperot »

Imustaskforhelp wrote: post #9

Are there any open source solutions or is cerebrium open-sourcing the technology behind it. Although I liked the read-up (side note: you might have to tone down the animations, as others have said, it was a bit dizzy) but overall it was a nice read but I still wish for more technical details as Cold starts are something that just is something that I am interested in. So are there any more resources that perhaps the t…

gVisor is open-source, and `cuda-checkpoint` is freely available.

gVisor's `runsc checkpoint` subcommand supports a `--save-restore-exec-argv` which lets you specify a program to execute before gVisor starts taking the process snapshot.

You can fill in the blanks from there.

Re: Reduce GVisor Cold Starts with GPU Snapshotting

#13

Post by za_mike157 »

eperot wrote: post #11
Imustaskforhelp wrote: post #9

Are there any open source solutions or is cerebrium open-sourcing the technology behind it. Although I liked the read-up (side note: you might have to tone down the animations, as others have said, it was a bit dizzy) but overall it was a nice read but I still wish for more technical details as Cold starts are something that just is something that I am interested in. So are there any more resources that perhaps the t…

gVisor is open-source, and `cuda-checkpoint` is freely available. gVisor's `runsc checkpoint` subcommand supports a `--save-restore-exec-argv` which lets you specify a program to execute before gVisor starts taking the process snapshot. You can fill in the blanks from there.

Us and the team from Modal have been upstreaming things to the GVisor repo (https://github.com/google/gvisor/pulls) in order to make it compatible with cuda-checkpoint and other parts of our system. While we are both contributing fixes and performance improvements we are unfortunately leaving some secret sauce on the side but hopefully it should get most folks to a successful implementation as is

Re: Reduce GVisor Cold Starts with GPU Snapshotting

#14

Post by harleyjs »

Imustaskforhelp wrote: post #9

Are there any open source solutions or is cerebrium open-sourcing the technology behind it. Although I liked the read-up (side note: you might have to tone down the animations, as others have said, it was a bit dizzy) but overall it was a nice read but I still wish for more technical details as Cold starts are something that just is something that I am interested in. So are there any more resources that perhaps the t…

This sort of technology is available on GKE

https://docs.cloud.google.com/kubernetes-engine/docs/concept...

Re: Reduce GVisor Cold Starts with GPU Snapshotting

#15

Post by za_mike157 »

harleyjs wrote: post #14
Imustaskforhelp wrote: post #9

Are there any open source solutions or is cerebrium open-sourcing the technology behind it. Although I liked the read-up (side note: you might have to tone down the animations, as others have said, it was a bit dizzy) but overall it was a nice read but I still wish for more technical details as Cold starts are something that just is something that I am interested in. So are there any more resources that perhaps the t…

This sort of technology is available on GKE https://docs.cloud.google.com/kubernetes-engine/docs/concept...

Interesting! I didn't see they released this. Do you know what their benchmarks are? I know for cloud run they are pretty slow

Re: Reduce GVisor Cold Starts with GPU Snapshotting

#18

Post by ranjanayush »

za_mike157 wrote: post #13
eperot wrote: post #11

Earlier quoted context omitted.

gVisor is open-source, and `cuda-checkpoint` is freely available. gVisor's `runsc checkpoint` subcommand supports a `--save-restore-exec-argv` which lets you specify a program to execute before gVisor starts taking the process snapshot. You can fill in the blanks from there.

Us and the team from Modal have been upstreaming things to the GVisor repo ( https://github.com/google/gvisor/pulls ) in order to make it compatible with cuda-checkpoint and other parts of our system. While we are both contributing fixes and performance improvements we are unfortunately leaving some secret sauce on the side but hopefully it should get most folks to a successful implementation as is

Recently gVisor added cuda checkpoint automation support, please consider using that: https://github.com/google/gvisor/commit/b3c184ce6b808648ddca...

Re: Reduce GVisor Cold Starts with GPU Snapshotting

#19

Post by za_mike157 »

ranjanayush wrote: post #18
za_mike157 wrote: post #13

Earlier quoted context omitted.

Us and the team from Modal have been upstreaming things to the GVisor repo ( https://github.com/google/gvisor/pulls ) in order to make it compatible with cuda-checkpoint and other parts of our system. While we are both contributing fixes and performance improvements we are unfortunately leaving some secret sauce on the side but hopefully it should get most folks to a successful implementation as is

Recently gVisor added cuda checkpoint automation support, please consider using that: https://github.com/google/gvisor/commit/b3c184ce6b808648ddca...

Thanks for this! We did see it and it was committed long after we already implemented this functionality. However, there are a lot of edge cases that this commit doesn't deal with that users are going to have to do
Post reply on HN