Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS vs. GCP reliability is wildly different

freeman.vc

Post reply on HN
Print view View original

191–200 of 245 posts

Re: AWS vs. GCP reliability is wildly different

#191

Post by robbintt »

deanCommie wrote: post #178
jhugo wrote: post #162

Earlier quoted context omitted.

All the clouds are pretty upfront about availability being non-guaranteed if you don't reserve it. I wouldn't call it a reliability issue if your non-guaranteed capacity takes some tens of seconds to provision. I mean, it might be your reliability issue, because you chose not to reserve capacity, but it's not really unreliability of the cloud — they're providing exactly what they advertise.

"Guaranteed" has different tiers of meaning - both theoretical and practical. In many cases, "guaranteed" just means "we'll give you a refund if we fuck up". SLAs are very much like this. IN PRACTICE, unless you're launching tens of thousands of instances of an obscure image type, reasonable customers would be able to get capacity, and promptly from the cloud. That's the entire cloud value proposition. So no, you can…

This isn't actually true, even for tiny customers. In a personal project, I used a single host of a single instance type several times per day and had to code up a fallback.

Re: AWS vs. GCP reliability is wildly different

#192

Post by dilyevsky »

dwmw2 wrote: post #183
dilyevsky wrote: post #158

Earlier quoted context omitted.

My experience running c5/6 instances makes me very confident ec2 doesn’t do live migration for these. Fwiw gcp live migration on latency sensitive workloads is very noticeable and often time straight up causes instance crash

Intrigued by this observation. What is it about your experience that leads you to conclude that EC2 doesn't do live migration? And could it be phrased differently as "EC2 doesn't do live migration badly "?

Mainly the barrage of "instance hardware degradation" emails that i get whereas on gcp those are just migrated (sometimes with a reboot/crash). Also there is no brownout. I've never used t2/3s which apparently do support migration which would make sense.

Re: AWS vs. GCP reliability is wildly different

#193

Post by stevenguh »

jsolson wrote: post #184
smashed wrote: post #174

Earlier quoted context omitted.

Which makes 2000% sense. Why would any tenant supplied data affect anything whatsoever? As a tenant, unless you are clashing with another resource under your own name, I don't see the point of failing. aws S3 would be an exception, where they make that limitation on globally unique bucket name very clear.

Idempotency. You're inserting a VM with a specific name. If you try to create the same resource twice, the GCE control plane reports that as a conflict. What they're doing here would be roughly equivalent to supplying the time to the AWS RunInstances API as an idempotency token. (I work on GCE, and asked an industry friend at AWS about how they guarantee idempotency for RunInstances).

GCP control plane is generally not idempotent.

When trying to create the same resource twice, all request should report the same status instead one failing, one succeeding.

In AWS, their APIs allow you to supply a client token if the API is not idempotent by default.

See https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Run_I....

Re: AWS vs. GCP reliability is wildly different

#194

Post by politelemon »

roomey wrote: post #26
lbhdc wrote: post #20

Earlier quoted context omitted.

How does migrating a vm to another physical machine work?

VMware has been doing this for years, it's called vmotion and there is a lot of documentation about it if you are interested (eg https://www.thegeekpub.com/8407/how-vmotion-works/ ) Essential, memory state is copied to the new host, the VM is stunned for a millisecond and the cpu states is copied and resumed on the new host (you may see a dropped ping). All the networking and storage is virtual anyway so that is "mov…

> VM is stunned for a millisecond

This conjures up hilarious mental imagery, thanks

Re: AWS vs. GCP reliability is wildly different

#195

Post by jsolson »

bushbaba wrote: post #186
jsolson wrote: post #184

Earlier quoted context omitted.

Idempotency. You're inserting a VM with a specific name. If you try to create the same resource twice, the GCE control plane reports that as a conflict. What they're doing here would be roughly equivalent to supplying the time to the AWS RunInstances API as an idempotency token. (I work on GCE, and asked an industry friend at AWS about how they guarantee idempotency for RunInstances).

Do you really need idempotency for runVM though.

I mean, it's kinda nice to know that if you reissue a request for an instance that could costs thousands of dollars per month due to a network glitch that you won't accidentally create two of them?

More practically, though, the instance name here is literally the name of the instance as it appears in the RESTful URL used for future queries about it. The 409 here is rejecting an attempt to create the same explicitly named resource twice.

Re: AWS vs. GCP reliability is wildly different

#196

Post by jsolson »

stevenguh wrote: post #193
jsolson wrote: post #184

Earlier quoted context omitted.

Idempotency. You're inserting a VM with a specific name. If you try to create the same resource twice, the GCE control plane reports that as a conflict. What they're doing here would be roughly equivalent to supplying the time to the AWS RunInstances API as an idempotency token. (I work on GCE, and asked an industry friend at AWS about how they guarantee idempotency for RunInstances).

GCP control plane is generally not idempotent. When trying to create the same resource twice, all request should report the same status instead one failing, one succeeding. In AWS, their APIs allow you to supply a client token if the API is not idempotent by default. See https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Run_I... .

> When try to create the same resource twice, the second should report success instead of failing.

Before I quibble with the idempotency point: I agree with this, entirely, but it is what it is and a lot of software has been written against the current behavior. So I'll cite Hyrum's law here: https://www.hyrumslaw.com/

> GCP control plane is generally not idempotent.

The GCE API occupies an odd space here, imo. The resource being created is, in practice, an operation to cause the named VM to exist. The operation has its own name, but the name of the VM in the insert operation is the name of the ultimate resource.

Net, the API is idempotent at a macro level in terms of the end-to-end creation or deletion of uniquely named resources. Which is a long winded way of saying that you're right, but that from a practical perspective it accomplishes enough of the goals of a truly idempotent API to be _useful_ for avoiding the same things that the AWS mechanism avoids: creation of unexpected duplicate VMs.

The more "modern" way to do this would be to have a truly idempotent description of the target state of the actual resource with a separate resource for the current live state, but we live with the sum of our past choices.

Re: AWS vs. GCP reliability is wildly different

#197

Post by my123 »

dilyevsky wrote: post #192
dwmw2 wrote: post #183

Earlier quoted context omitted.

Intrigued by this observation. What is it about your experience that leads you to conclude that EC2 doesn't do live migration? And could it be phrased differently as "EC2 doesn't do live migration badly "?

Mainly the barrage of "instance hardware degradation" emails that i get whereas on gcp those are just migrated (sometimes with a reboot/crash). Also there is no brownout. I've never used t2/3s which apparently do support migration which would make sense.

After some kinds of hardware failure, it can become impossible to do live migration safely. When a crash can ensure due to a live migration from faulty HW, I'd argue that it's much better to not attempt it.

Re: AWS vs. GCP reliability is wildly different

#198

Post by sofixa »

jiggawatts wrote: post #120
outworlder wrote: post #110

Earlier quoted context omitted.

Not really. Or at least not in the same league. AWS doesn't have live migration at all. You have to stop/start. Azure technically does, but it doesn't always work(they say 90%). 30 seconds is a long time. VMWare has live migration (and seems to be the closest to what GCP does) but it is still an inferior user experience. This is the key thing you are missing – GCP not only has live migration, but it is completely tra…

VMware's live migration is totally seamless, so I don't know what you mean by "inferior user experience". You typically see less than a second of packet loss, and a small performance hit for about a minute while the memory is "swapped" across to the new machine. Similarly, VMware has had live storage migration for years. VMware is lightyears ahead of the big clouds, but unfortunately they "missed the boat" on the pub…

So, i used to be a part time vSphere admin, worked with many others, and had to automate the hell out of it to deal as little as possible with that dumpster fire.

No, VMware didn't miss the boat, vCloud Air was announced in 2009 and made generally available in 2013. Roughly same timelines as Azure and GCP, slightly trailing AWS, and those were the early days, where the public cloud was still exotic. And VMware had the massive advantage of brand recognition in that domain and existing footprint with enterprises which could be scaled out.

Problem was, vCloud Air, like vSphere, was shit. Yeah, it did some things well, and had some very nice features - vMotion, DRS (though it doesn't really use CPU ready contention for scheduling decisions which is stupid), vSAN, hot adding resources (but not RAM, because decades ago Linux had issues if you had less than 4GB RAM and you added more, so to this day you can't do that). When they worked, because when they didn't, good luck because error messages are useless, logs are weirdly structured and uselessly verbose, so a massive pain to deal with. Oh and many of those features were either behind a Flash UI(FFS), or an abomination of an API that is inconsistent ("this object might have been deleted or hasn't been created yet") and had weird limitations like when you have an async task you can't check it's status details. And many of those features were so complex, that a random consuming user basically had to rely on a dedicated team of vExperts, which often resulted in a nice silo slowing everyone down.

Their hardware compatibility list was a joke - the Intel X710 NIC stayed on it for more than a year with a widely known terribly broken driver.

But what made VMware fail the most, IMHO, was the wrong focus, technically - VM, instead of application. A developer/ops person couldn't care less about the object of a VM. Of course they tried some things like vApp and vCloud Director etc. which are just disgusting abominations designed with a PowerPoint in mind, not a user. And pricing. Opaque and expensive, with bad usability. No wonder everyone jumped on the pay as you go, usable alternatives.

Re: AWS vs. GCP reliability is wildly different

#199

Post by fomine3 »

Crash0v3rid3 wrote: post #139
mempko wrote: post #134

Earlier quoted context omitted.

What are your thoughts on the generally slower launch times with a huge variance on GCP?

The author failed to mention which regions these tests were run. GPU availability can vary depending on the regions that were tested for both Cloud providers.

This is a big missed point.

Re: AWS vs. GCP reliability is wildly different

#200

Post by [deleted] »

esrauch wrote: post #151
lelandfe wrote: post #90

Earlier quoted context omitted.

> wildly different For this, I'd prefer a title that lets me draw my own conclusions. 84 errors out of 3000 doesn't sound awful to me...? But what do I know – maybe just give me the data: "1 in 3000 GPUs fail to spawn on AWS. GCP: 84" "Time to provision GPU with AWS: 11.4s. GCP: 42.6s" "GCP >4x avg. time to provision GPU than AWS" "Provisioning on GCP both slower and more error-prone than AWS"

84 of 3000 failed is only "one nine"

[deleted]
Post reply on HN