AWS vs. GCP reliability is wildly different • Hacker News Board

Re: AWS vs. GCP reliability is wildly different

#161

Post by valleyjo » Thu, Sep 22, 2022, 2:30 AM UTC

jcheng wrote: post #117

outworlder wrote: post #103

Unclear what the article has to do with reliability. Yes, spinning up machines on GCP is incredibly fast and has always been. AWS is decent. Azure feels like I'm starting a Boeing 747 instead of a VM. However, there's one aspect where GCP is a clear winner on the reliability front. They auto-migrate instances transparently and with close to zero impact to workloads – I want to say zero impact but it's not technically…

> Yes, spinning up machines on GCP is incredibly fast and has always been. AWS is decent. FWIW this article is saying the opposite--it's AWS that beats GCP in startup speed.

This article states that GPU instances are slower on GCP - it doesn’t make any claims about non-GPU instances.

Re: AWS vs. GCP reliability is wildly different

#162

Post by jhugo » Thu, Sep 22, 2022, 2:41 AM UTC

mikewave wrote: post #46

dark-star wrote: post #13

I wonder why someone would equate "instance launch time" with "reliability"... I won't go as far as calling it "clickbait" but wouldn't some other noun ("startup performance is wildly different") have made more sense?

Well, if your system elastically uses GPU compute and needs to be able to spin up, run compute on a GPU, and spin down in a predictable amount of time to provide reasonable UX, launch time would definitely be a factor in terms of customer-perceived reliability.

All the clouds are pretty upfront about availability being non-guaranteed if you don't reserve it. I wouldn't call it a reliability issue if your non-guaranteed capacity takes some tens of seconds to provision. I mean, it might be your reliability issue, because you chose not to reserve capacity, but it's not really unreliability of the cloud — they're providing exactly what they advertise.

Re: AWS vs. GCP reliability is wildly different

#163

Post by jhugo » Thu, Sep 22, 2022, 2:43 AM UTC

mempko wrote: post #134

rwiggins wrote: post #127

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

What are your thoughts on the generally slower launch times with a huge variance on GCP?

At work we run some (non-GPU) instances in every AWS region, and there's pretty big variability over time and region for on-demand launch time. I'd expect it might be even higher for GPU instances. I suspect that a more rigorous investigation might find there isn't quite as big a difference overall as this article suggests.

Re: AWS vs. GCP reliability is wildly different

#164

Post by hericium » Thu, Sep 22, 2022, 2:47 AM UTC

dark-star wrote: post #13

I wonder why someone would equate "instance launch time" with "reliability"... I won't go as far as calling it "clickbait" but wouldn't some other noun ("startup performance is wildly different") have made more sense?

Cloud reliability is not the same as a reliability of already spawned VM.

Here it's the possibility to launch new VMs to satisfy dynamic projects' needs. Cloud provider should allow you to scale-up in a predictable way. When it doesn't - it can be called unreliable.

Also, "unreliable" is basically a synonym for "Google" these days.

Re: AWS vs. GCP reliability is wildly different

#165

Post by [deleted] » Thu, Sep 22, 2022, 2:49 AM UTC

somat wrote: post #141

rco8786 wrote: post #108

Earlier quoted context omitted.

It’s still performance. If this was “AWE failed to deliver the new machines and GCP delivered”, sure, reliability. But this isn’t that. The race car that finishes first is not “more reliable” than the one in 10th. They are equally as reliable, having both finished the race. The first place car is simply faster at the task.

The one in first can more reliably win races however.

[deleted]

Re: AWS vs. GCP reliability is wildly different

#166

Post by geysersam » Thu, Sep 22, 2022, 3:14 AM UTC

zmmmmm wrote: post #44

> In total it scaled up about 3,000 T4 GPUs per platform > why I burned $150 on GPUs How do you rent 3000 GPUs over a period of weeks for $150? Were they literally requisitioning it and releasing it immediately? Seems like this is quite a unrealistic type of usage pattern and would depend a lot on whether the cloud provider optimises to hand you back the same warm instance you just relinquished. > GCP allows you to a…

> $150

Was asking myself the same question. From the pricing information on gcp it seems minimum billing time is 1 minute, making 3000 GPUs cost $50 minimum. If this is the case then $150 is reasonable for the kind of usage pattern you describe.

Re: AWS vs. GCP reliability is wildly different

#167

Post by DangitBobby » Thu, Sep 22, 2022, 3:42 AM UTC

rwalle wrote: post #60

Looks like the author has never heard of the word "histogram" That graph is a pain to see.

A histogram would take away one of the dimensions, probably time, unless they resorted to some weird stacked layout. Without time, people would complain that they don't know if it was consistent across the tested period. The graph is fine.

Re: AWS vs. GCP reliability is wildly different

#168

Post by DangitBobby » Thu, Sep 22, 2022, 3:45 AM UTC

dekhn wrote: post #45

What would you expect? AWS is an org dedicated to giving customers what they want and charging them for it, while GCP is an org dedicated to telling customers what they want and using the revenue to get slightly better cost margins on Intel servers.

They don't really tell us what we want, we just buy what we need. Might work for you.

Re: AWS vs. GCP reliability is wildly different

#169

Post by onphonenow » Thu, Sep 22, 2022, 3:45 AM UTC

somat wrote: post #86

rco8786 wrote: post #71

Earlier quoted context omitted.

Sure but not anywhere remotely near clearing the bar to simply calling that “reliability”.

It is not reliably running the machine but reliably getting the machine. Like the article said, The promise of the cloud is that you can easily get machines when you need them the cloud that sometimes does not get you that machine(or does not get you that machine in time) is a less reliable cloud than the one that does.

If you want that promise you can reserve capacity in various ways. Google has reservations. Folks use this for DR, your org can get a pool of shared ones going if you are going to have various teams leaning on GPU etc.

The promise of the cloud is that you can flexibly spin up machines if available, and easily spin down, no long term contracts or CapEx etc. They are all pretty clear that there are capacity limits under the hood (and your account likely has various limits on it as a result).

Re: AWS vs. GCP reliability is wildly different

#170

Post by campers » Thu, Sep 22, 2022, 4:00 AM UTC

irrational wrote: post #104

dark-star wrote: post #102

Earlier quoted context omitted.

I'd still consider it as "performance issue", not "reliability issue". There is no service unavailability here. It just takes your system a minute longer until the target GPU capacity is available. Until then it runs on fewer GPU resources, which makes it slower. Hence performance. The errors might be considered a reliability issue, but then again, errors are a very common thing in large distributed systems, and any…

I’d like to see a breakdown of the cost differences. If the costs are nearly equal, why would I not choose the one that has a faster startup time and fewer errors?

With GCP you can right-size the CPU and memory of the VM the GPU is attached to, unlike the fixed GPU AWS instances, so there is the potential for cost savings there.