Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS vs. GCP reliability is wildly different

freeman.vc

Post reply on HN
Print view View original

211–220 of 245 posts

Re: AWS vs. GCP reliability is wildly different

#211

Post by Terretta »

jsolson wrote: post #196
stevenguh wrote: post #193

Earlier quoted context omitted.

GCP control plane is generally not idempotent. When trying to create the same resource twice, all request should report the same status instead one failing, one succeeding. In AWS, their APIs allow you to supply a client token if the API is not idempotent by default. See https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Run_I... .

> When try to create the same resource twice, the second should report success instead of failing. Before I quibble with the idempotency point: I agree with this, entirely, but it is what it is and a lot of software has been written against the current behavior. So I'll cite Hyrum's law here: https://www.hyrumslaw.com/ > GCP control plane is generally not idempotent. The GCE API occupies an odd space here, imo. The r…

It's a lot shorter to write:

You're right, we did it wrong.

// And paradoxically makes engineers like you.

Re: AWS vs. GCP reliability is wildly different

#213

Post by philliphaydon »

jsolson wrote: post #184
smashed wrote: post #174

Earlier quoted context omitted.

Which makes 2000% sense. Why would any tenant supplied data affect anything whatsoever? As a tenant, unless you are clashing with another resource under your own name, I don't see the point of failing. aws S3 would be an exception, where they make that limitation on globally unique bucket name very clear.

Idempotency. You're inserting a VM with a specific name. If you try to create the same resource twice, the GCE control plane reports that as a conflict. What they're doing here would be roughly equivalent to supplying the time to the AWS RunInstances API as an idempotency token. (I work on GCE, and asked an industry friend at AWS about how they guarantee idempotency for RunInstances).

Sounds like AWS got it right.

Re: AWS vs. GCP reliability is wildly different

#214

Post by ayewo »

Crash0v3rid3 wrote: post #139
mempko wrote: post #134

Earlier quoted context omitted.

What are your thoughts on the generally slower launch times with a huge variance on GCP?

The author failed to mention which regions these tests were run. GPU availability can vary depending on the regions that were tested for both Cloud providers.

The author linked to the code at the end of the post.

The regions used are "us-east-1" for AWS [1] and "us-central1-b" for GCP [2].

1: https://github.com/piercefreeman/cloud-gpu-reliability/blob/...

2: https://github.com/piercefreeman/cloud-gpu-reliability/blob/...

Re: AWS vs. GCP reliability is wildly different

#215

Post by rkangel »

jrumbut wrote: post #175
rwiggins wrote: post #127

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

> (3) `time.time()` is not monotonic. I just winced in pain thinking of the ways that can bite you. I guess in a cloud/virtualized environment with many short lived instances it isn't even that obscure an issue to run into. A nice discussion on Stack Overflow: https://stackoverflow.com/questions/64497035/is-time-from-ti...

Yes. When people write `time.time()` they almost always actually want `time.monotonic()`.

Re: AWS vs. GCP reliability is wildly different

#216

Post by HenriTEL »

mikewave wrote: post #46
dark-star wrote: post #13

I wonder why someone would equate "instance launch time" with "reliability"... I won't go as far as calling it "clickbait" but wouldn't some other noun ("startup performance is wildly different") have made more sense?

Well, if your system elastically uses GPU compute and needs to be able to spin up, run compute on a GPU, and spin down in a predictable amount of time to provide reasonable UX, launch time would definitely be a factor in terms of customer-perceived reliability.

GCP provides elactic features for that. One should use them instead of manually requesting new instances.

Re: AWS vs. GCP reliability is wildly different

#217

Post by ReptileMan »

DonHopkins wrote: post #203
hericium wrote: post #164

Earlier quoted context omitted.

Cloud reliability is not the same as a reliability of already spawned VM. Here it's the possibility to launch new VMs to satisfy dynamic projects' needs. Cloud provider should allow you to scale-up in a predictable way. When it doesn't - it can be called unreliable. Also, "unreliable" is basically a synonym for "Google" these days.

Let me unreliable that for you.

To be fair their search is so crap lately, throwing the dice is not the worst option in the world to find a result that will be actually useful.

Re: AWS vs. GCP reliability is wildly different

#218

Post by rco8786 »

somat wrote: post #141
rco8786 wrote: post #108

Earlier quoted context omitted.

It’s still performance. If this was “AWE failed to deliver the new machines and GCP delivered”, sure, reliability. But this isn’t that. The race car that finishes first is not “more reliable” than the one in 10th. They are equally as reliable, having both finished the race. The first place car is simply faster at the task.

The one in first can more reliably win races however.

You cannot infer that based on the results of the race...that's literally the entire point I am making. The 1st place car might blow up in the next race, the 10th place car might finish 10th place for the next 100 races.

If the article were measuring HTTP response times and found that AWS's average response time was 50ms and GCP's was 200ms, and both returned 200s for every single request in the test, would you say AWS is more reliable than GCP based on that? Of course not, it's asinine.

Re: AWS vs. GCP reliability is wildly different

#220

Post by darkwater »

_msw_ wrote: post #138
xyzzyz wrote: post #7

Earlier quoted context omitted.

Dunno about AWS, but GCP uses live migration, and will migrate your VM across physical machines as necessary. The disk volumes are all connected over the network, nothing really depends on the actual physical machine your VM is ran on.

Disclosure: I work for Amazon, and in the past I worked directly on EC2. From the FAQ: https://aws.amazon.com/ec2/faqs/ Q: How does EC2 perform maintenance? AWS regularly performs routine hardware, power, and network maintenance with minimal disruption across all EC2 instance types. To achieve this we employ a combination of tools and methods across the entire AWS Global infrastructure, such as redundant and concurre…

> AWS regularly performs routine hardware, power, and network maintenance with minimal disruption across all EC2 instance types. To achieve this we employ a combination of tools and methods across the entire AWS Global infrastructure, such as redundant and concurrently maintainable systems, as well as live system updates and migration.

And yet, I keep getting almost every weeks emails like this:

"EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance (instance-ID: i-xxxxxxx) associated with your AWS account (AWS Account ID: NNNNN) in the eu-west-1 region. Due to this degradation your instance could already be unreachable. We will stop your instance after 2022-09-21 16:00:00 UTC"

And we don't have tens of thousands of VMs in that region, just around 1k.

Post reply on HN