Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS vs. GCP reliability is wildly different

freeman.vc

Post reply on HN
Print view View original

181–190 of 245 posts

Re: AWS vs. GCP reliability is wildly different

#181

Post by eurasiantiger »

devmunchies wrote: post #125
dheera wrote: post #85

Earlier quoted context omitted.

Using HTTP error codes for non-REST things is cringe. 503 would mean the IaaS API calls themselves are unavailable. Very different from the API working perfectly fine but the instances not being available.

What? REST is just some API philosophy, its doesn't even have to be on top of HTTP. Why would you think HTTP status codes are made for REST? They are made for HTTP to describe the response of the resource you are requesting, and the AWS API uses HTTP so it makes sense to use HTTP status codes.

No post body was provided.

Re: AWS vs. GCP reliability is wildly different

#182

Post by eurasiantiger »

sn0wf1re wrote: post #94
eurasiantiger wrote: post #55

Earlier quoted context omitted.

503 service unavailable?

That would be confusing. The HTTP response code should not be conflated with the application's state.

There will come a moment in time when you realize exactly what you have stated here and why it is not a good mental palace to live in.

Re: AWS vs. GCP reliability is wildly different

#183

Post by dwmw2 »

dilyevsky wrote: post #158
_msw_ wrote: post #133

Earlier quoted context omitted.

Here's a comment that I made in a past thread. https://news.ycombinator.com/item?id=26650082

My experience running c5/6 instances makes me very confident ec2 doesn’t do live migration for these. Fwiw gcp live migration on latency sensitive workloads is very noticeable and often time straight up causes instance crash

Intrigued by this observation. What is it about your experience that leads you to conclude that EC2 doesn't do live migration?

And could it be phrased differently as "EC2 doesn't do live migration badly"?

Re: AWS vs. GCP reliability is wildly different

#184

Post by jsolson »

smashed wrote: post #174
sitharus wrote: post #131

Earlier quoted context omitted.

This is a very good point - AWS uses tags to give instances a friendly name, so the name does not have to be unique. The same logic would not fail on AWS.

Which makes 2000% sense. Why would any tenant supplied data affect anything whatsoever? As a tenant, unless you are clashing with another resource under your own name, I don't see the point of failing. aws S3 would be an exception, where they make that limitation on globally unique bucket name very clear.

Idempotency.

You're inserting a VM with a specific name. If you try to create the same resource twice, the GCE control plane reports that as a conflict.

What they're doing here would be roughly equivalent to supplying the time to the AWS RunInstances API as an idempotency token.

(I work on GCE, and asked an industry friend at AWS about how they guarantee idempotency for RunInstances).

Re: AWS vs. GCP reliability is wildly different

#185

Post by jhugo »

deanCommie wrote: post #178
jhugo wrote: post #162

Earlier quoted context omitted.

All the clouds are pretty upfront about availability being non-guaranteed if you don't reserve it. I wouldn't call it a reliability issue if your non-guaranteed capacity takes some tens of seconds to provision. I mean, it might be your reliability issue, because you chose not to reserve capacity, but it's not really unreliability of the cloud — they're providing exactly what they advertise.

"Guaranteed" has different tiers of meaning - both theoretical and practical. In many cases, "guaranteed" just means "we'll give you a refund if we fuck up". SLAs are very much like this. IN PRACTICE, unless you're launching tens of thousands of instances of an obscure image type, reasonable customers would be able to get capacity, and promptly from the cloud. That's the entire cloud value proposition. So no, you can…

Ignoring the fact that the results are probably partially flawed due to methodology (see top-level comment from someone who works on GCE) and are not reproducible due to missing information, pointing out the lack of a guarantee is not hand-waving. The OP uses the word "reliability" to catch attention, which certainly worked, but this has nothing to do with reliability.

Re: AWS vs. GCP reliability is wildly different

#186

Post by bushbaba »

jsolson wrote: post #184
smashed wrote: post #174

Earlier quoted context omitted.

Which makes 2000% sense. Why would any tenant supplied data affect anything whatsoever? As a tenant, unless you are clashing with another resource under your own name, I don't see the point of failing. aws S3 would be an exception, where they make that limitation on globally unique bucket name very clear.

Idempotency. You're inserting a VM with a specific name. If you try to create the same resource twice, the GCE control plane reports that as a conflict. What they're doing here would be roughly equivalent to supplying the time to the AWS RunInstances API as an idempotency token. (I work on GCE, and asked an industry friend at AWS about how they guarantee idempotency for RunInstances).

Do you really need idempotency for runVM though.

Re: AWS vs. GCP reliability is wildly different

#187

Post by my123 »

master_crab wrote: post #76
btgeekboy wrote: post #43

Earlier quoted context omitted.

I wouldn't be surprised if GCP has GPUs scattered throughout the datacenter. If you happen to want to attach one, it has to find one for you to use - potentially live migrating your instance or someone else's so that it can connect them. It'd explain the massive variability between launch times.

Yeah that was my thought too when I first read the blurb. It’s neat…but like a lot of things in large scale operations, the devil is in the details. GPU-CPU communications is a low latency high bandwidth operation. Not something you can trivially do over standard TCP. GCP offering something like that without the ability to flawlessly migrate the VM or procure enough “local” GPUs means it’s just vaporware. As a side n…

Fungible is selling a GPU decoupling solution via PCIe encapsulated over Ethernet today, so it can certainly be done.

And NVIDIA's vGPU solutions do support live migration of GPUs to another host (in which case the vGPU gets moved too, to a GPU on that target).

Re: AWS vs. GCP reliability is wildly different

#189

Post by jsolson »

kccqzy wrote: post #23

Heard from a Googler that the internal infrastructure (Borg) is simply not optimized for quick startup. Launching a new Borg job often takes multiple minutes before the job runs. Not surprising at all.

This is mostly not true in cases where resources are actually available (and in GCE if they're not the API rejects the VM outright, in general). To the extent that it is true for Borg when the job schedules immediately, it's largely due to package (~container layers, ish) loading. This is less relevant today (because reasons), and also mostly doesn't apply to GCE as the relevant packages are almost universally proactively made available on relevant hosts.

The origin for the info that jobs take "minutes" likely involves jobs that were pending available resources. This is a valid state in Borg, but GCE has additional admission control mechanisms aimed at avoiding extended residency in pending.

As dekhn notes, there are many factors that contribute to VM startup time. GPUs are their own variety of special (and, yes, sometimes slow), with factors that mostly don't apply to more pedestrian VM shapes.

Re: AWS vs. GCP reliability is wildly different

#190

Post by CodesInChaos »

kazinator wrote: post #132

> This is particularly true for GPUs, which are uniquely squeezed by COVID shutdowns, POW mining, and growing deep learning models Is the POW mining part true any more? Hasn't mining moved to dedicated hardware?

Bitcoin mining has used dedicated hardware for a long time. But I believe Ethereum mining used GPUs before the very recent proof-of-stake update.
Post reply on HN