Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS vs. GCP reliability is wildly different

freeman.vc

Post reply on HN
Print view View original

151–160 of 245 posts

Re: AWS vs. GCP reliability is wildly different

#151

Post by esrauch »

lelandfe wrote: post #90
iLoveOncall wrote: post #81

Earlier quoted context omitted.

It is clickbait, the real title should be "AWS vs. GCP on-demand provisioning of GPU resources performance is wildly different". That said, while I agree that launch time and provisioning error rate are not sufficient to define reliability, they are definitely a part of it.

> wildly different For this, I'd prefer a title that lets me draw my own conclusions. 84 errors out of 3000 doesn't sound awful to me...? But what do I know – maybe just give me the data: "1 in 3000 GPUs fail to spawn on AWS. GCP: 84" "Time to provision GPU with AWS: 11.4s. GCP: 42.6s" "GCP >4x avg. time to provision GPU than AWS" "Provisioning on GCP both slower and more error-prone than AWS"

84 of 3000 failed is only "one nine"

Re: AWS vs. GCP reliability is wildly different

#152

Post by ryukoposting »

lacker wrote: post #116

Anecdotally I tend to agree with the author. But this really isn't a great way of comparing cloud services. The fundamental problem with cloud reliability is that it depends on a lot of stuff that's out of your control, that you have no visibility into. I have had services running happily on AWS with no errors, and the next month without changing anything they fail all the time. Why? Well, we look into it and it turn…

You've just perfectly characterized why on-site infrastructure will always have its place.

Re: AWS vs. GCP reliability is wildly different

#153

Post by thayne »

dark-star wrote: post #13

I wonder why someone would equate "instance launch time" with "reliability"... I won't go as far as calling it "clickbait" but wouldn't some other noun ("startup performance is wildly different") have made more sense?

Well, I mean it is measuring how reliably you can get a GPU instance. But it certainly isn't the overall reliability. And depending on your workflow, it might not even be a very interesting measure. I would be more interested in seeing a comparison of how long regular non-GPU instances can run without having to be rebooted, and maybe how long it takes to allocate a regular VM.

Re: AWS vs. GCP reliability is wildly different

#154

Post by yolovoe »

outworlder wrote: post #103

Unclear what the article has to do with reliability. Yes, spinning up machines on GCP is incredibly fast and has always been. AWS is decent. Azure feels like I'm starting a Boeing 747 instead of a VM. However, there's one aspect where GCP is a clear winner on the reliability front. They auto-migrate instances transparently and with close to zero impact to workloads – I want to say zero impact but it's not technically…

EC2 live migrates instances too. Not sure where we are with rollout across the fleet.

The reason, from what I understand, why GCP does live migration more is because ec2 focused on live updates instead of live migration. Whereas GCP migrates instances to update servers, ec2 live updates everything down to firmware while instances are running.

Curious, what instance types are you using on EC2 that you see so much maintenance?

Re: AWS vs. GCP reliability is wildly different

#155

Post by dubcee349 »

btgeekboy wrote: post #43
remus wrote: post #19

> The offerings between the two cloud vendors are also not the same, which might relate to their differing response times. GCP allows you to attach a GPU to an arbitrary VM as a hardware accelerator - you can separately configure quantity of the CPUs as needed. AWS only provisions defined VMs that have GPUs attached - the g4dn.x series of hardware here. Each of these instances are fixed in their CPU allocation, so if…

I wouldn't be surprised if GCP has GPUs scattered throughout the datacenter. If you happen to want to attach one, it has to find one for you to use - potentially live migrating your instance or someone else's so that it can connect them. It'd explain the massive variability between launch times.

I doubt it would be setup like that. Compute is usually deployed as part of a large set of servers. The reason for that is different compute workloads require different uplink capacity.You don't need a petabyte of uplink capacity for many GPU loads but you may for compute. Just switching ASICs are much more expensive for 400G+ than 100G. That hasn't even got into the optics, NICs and other things. You don't mix and match compute across the same place in the data center traditionally.

Re: AWS vs. GCP reliability is wildly different

#156

Post by lmeyerov »

I'd say that's a weak test of capacity. Would love to see on Azure - T4s or an equiv aren't even really provided anymore!

We find reliability a diff story. Eg, our main source of downtime on Azure is they restart (live migrate?) our reserved T4s every few weeks, causing 2-10min outages per GPU per month.

Re: AWS vs. GCP reliability is wildly different

#157

Post by mikepurvis »

mikewave wrote: post #46
dark-star wrote: post #13

I wonder why someone would equate "instance launch time" with "reliability"... I won't go as far as calling it "clickbait" but wouldn't some other noun ("startup performance is wildly different") have made more sense?

Well, if your system elastically uses GPU compute and needs to be able to spin up, run compute on a GPU, and spin down in a predictable amount of time to provide reasonable UX, launch time would definitely be a factor in terms of customer-perceived reliability.

Hopefully anyone with a workload that's that latency sensitive would a have preallocated pool of warmed up instances ready to go.

Re: AWS vs. GCP reliability is wildly different

#158

Post by dilyevsky »

_msw_ wrote: post #133
ta20200710 wrote: post #124

Earlier quoted context omitted.

EC2 does support live migration, but it's not public and only for certain instance types/hypervisors. See: https://news.ycombinator.com/item?id=17815806

Here's a comment that I made in a past thread. https://news.ycombinator.com/item?id=26650082

My experience running c5/6 instances makes me very confident ec2 doesn’t do live migration for these. Fwiw gcp live migration on latency sensitive workloads is very noticeable and often time straight up causes instance crash

Re: AWS vs. GCP reliability is wildly different

#159

Post by ryukoposting »

btgeekboy wrote: post #43
remus wrote: post #19

> The offerings between the two cloud vendors are also not the same, which might relate to their differing response times. GCP allows you to attach a GPU to an arbitrary VM as a hardware accelerator - you can separately configure quantity of the CPUs as needed. AWS only provisions defined VMs that have GPUs attached - the g4dn.x series of hardware here. Each of these instances are fixed in their CPU allocation, so if…

I wouldn't be surprised if GCP has GPUs scattered throughout the datacenter. If you happen to want to attach one, it has to find one for you to use - potentially live migrating your instance or someone else's so that it can connect them. It'd explain the massive variability between launch times.

I've only ever used AWS for this stuff. When the author said that you could just "add a GPU" to an existing instance, my first reaction was "wow, that sounds like it would be really complicated behind the scenes."

Re: AWS vs. GCP reliability is wildly different

#160

Post by valleyjo »

mempko wrote: post #134
rwiggins wrote: post #127

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

What are your thoughts on the generally slower launch times with a huge variance on GCP?

Just remember this is for GPU instances. Other vm families are pretty fast to launch.
Post reply on HN