Heard from a Googler that the internal infrastructure (Borg) is simply not optimized for quick startup. Launching a new Borg job often takes multiple minutes before the job runs. Not surprising at all.
Borg is not used for gcp vms, though.
AWS vs. GCP reliability is wildly different
51–60 of 245 posts
Re: AWS vs. GCP reliability is wildly different
#52Earlier quoted context omitted.
Dunno about AWS, but GCP uses live migration, and will migrate your VM across physical machines as necessary. The disk volumes are all connected over the network, nothing really depends on the actual physical machine your VM is ran on.
How does migrating a vm to another physical machine work?
This blog post is pretty old (2015) but gives a good introduction.
https://cloudplatform.googleblog.com/2015/03/Google-Compute-...
Re: AWS vs. GCP reliability is wildly different
#53> In total it scaled up about 3,000 T4 GPUs per platform > why I burned $150 on GPUs How do you rent 3000 GPUs over a period of weeks for $150? Were they literally requisitioning it and releasing it immediately? Seems like this is quite a unrealistic type of usage pattern and would depend a lot on whether the cloud provider optimises to hand you back the same warm instance you just relinquished. > GCP allows you to a…
GPUs are physical but VMs are not; I expect they just move them to a host with a GPU.
Re: AWS vs. GCP reliability is wildly different
#54Re: AWS vs. GCP reliability is wildly different
#55Re: AWS vs. GCP reliability is wildly different
#56Earlier quoted context omitted.
How does migrating a vm to another physical machine work?
They pause your VM, copy everything about its state over to the new machine, and quickly start the other instance. It's pretty clever. I think there are tricks you can play with machines that have large memory footprints to copy most of it before the pause, and only copy what has changed since then during the pause. The disks are all on the network, so no need to move anything there.
In reality it sync the memory first to the other host and only pause the vm when the last state sync is small enough to be so quick the pause is barely measurable.
Re: AWS vs. GCP reliability is wildly different
#57I wish Azure was here to round it out!
Re: AWS vs. GCP reliability is wildly different
#58The link is broken?
Works for me using Firefox in Germany, although the article doesn't really match the title so maybe that's why you were confused? :p
Re: AWS vs. GCP reliability is wildly different
#59> In total it scaled up about 3,000 T4 GPUs per platform > why I burned $150 on GPUs How do you rent 3000 GPUs over a period of weeks for $150? Were they literally requisitioning it and releasing it immediately? Seems like this is quite a unrealistic type of usage pattern and would depend a lot on whether the cloud provider optimises to hand you back the same warm instance you just relinquished. > GCP allows you to a…
It probably live-migrates your VM to a physical machine that has a GPU available.
...if there are any GPUs available in the AZ that is. I had a hell of a time last year moving back and forth between regions to grab just 1 GPU to test something. The web UI didn't have a "any region" option for launching VMs so if you don't use the API you'll have to sit there for 20 minutes trying each AZ/region until you managed to grab one.
Re: AWS vs. GCP reliability is wildly different
#60Looks like the author has never heard of the word "histogram"
That graph is a pain to see.