Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS vs. GCP reliability is wildly different

freeman.vc

Post reply on HN
Print view View original

221–230 of 245 posts

Re: AWS vs. GCP reliability is wildly different

#221

Post by flutas »

jrumbut wrote: post #175
rwiggins wrote: post #127

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

> (3) `time.time()` is not monotonic. I just winced in pain thinking of the ways that can bite you. I guess in a cloud/virtualized environment with many short lived instances it isn't even that obscure an issue to run into. A nice discussion on Stack Overflow: https://stackoverflow.com/questions/64497035/is-time-from-ti...

> I just winced in pain thinking of the ways that can bite you.

Something similar caused my favorite bug so far to track down.

We were seeing odd spikes in our video playback analytics of some devices watching multiple years worth of video in System.currenTimeMillis() in Java isn't monotonic either is my short answer for what was causing it. Tracking down _what_ was causing it was even more fun though. Devices (phones) were updating their system time from the network and jumping between timezones.

Re: AWS vs. GCP reliability is wildly different

#222

Post by pier25 »

mikewave wrote: post #46
dark-star wrote: post #13

I wonder why someone would equate "instance launch time" with "reliability"... I won't go as far as calling it "clickbait" but wouldn't some other noun ("startup performance is wildly different") have made more sense?

Well, if your system elastically uses GPU compute and needs to be able to spin up, run compute on a GPU, and spin down in a predictable amount of time to provide reasonable UX, launch time would definitely be a factor in terms of customer-perceived reliability.

Wouldn't Cloud Run be a better product for that use case?

Re: AWS vs. GCP reliability is wildly different

#223

Post by kevincox »

mempko wrote: post #134
rwiggins wrote: post #127

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

What are your thoughts on the generally slower launch times with a huge variance on GCP?

FWIW in our use case of non-GPU instances they launched way faster and more consistently on GCP than AWS. So I guess it is complicated and may depend on exactly what instance you are launching.

Re: AWS vs. GCP reliability is wildly different

#224

Post by dekhn »

fragmede wrote: post #129
dekhn wrote: post #95

Earlier quoted context omitted.

A well-configured isolated borg cluster and well-configured job can be really fast. If there's no preemption (IE, no other job that is kicked off and gets some grace period), the packages are already cached locally, and there is no undue load on the scheduler, the resources are available, and it's a job with tasks, rather than multiple jobs, it will be close to instantaneous. I spend a significant fraction of my 11+…

Psh someone's bragging about not being at batch priority.

I ran at -1

Re: AWS vs. GCP reliability is wildly different

#225

Post by kevincox »

politelemon wrote: post #194
roomey wrote: post #26

Earlier quoted context omitted.

VMware has been doing this for years, it's called vmotion and there is a lot of documentation about it if you are interested (eg https://www.thegeekpub.com/8407/how-vmotion-works/ ) Essential, memory state is copied to the new host, the VM is stunned for a millisecond and the cpu states is copied and resumed on the new host (you may see a dropped ping). All the networking and storage is virtual anyway so that is "mov…

> VM is stunned for a millisecond This conjures up hilarious mental imagery, thanks

You just bop it on the head, and move it to the new machine quickly. By the time the VM comes to it won't even realize that it is in a new home.

Re: AWS vs. GCP reliability is wildly different

#226

Post by Cthulhu_ »

rwiggins wrote: post #127

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

I've naively used millisecond precision things for a long time - not in anything critical I don't think - but I've only recently come to more of an awareness that a millisecond is a pretty long time. Recent example is that I used a timestamp to version a record in a database, but it's feasible that in a Go application, a record could feasilby be mutated multiple times a millisecond by different users / processes / requests.

Unfortunately, millisecond-precise timestamps proved to be a bit tricky in combination with sqlite.

Re: AWS vs. GCP reliability is wildly different

#227

Post by jsolson »

philliphaydon wrote: post #213
jsolson wrote: post #184

Earlier quoted context omitted.

Idempotency. You're inserting a VM with a specific name. If you try to create the same resource twice, the GCE control plane reports that as a conflict. What they're doing here would be roughly equivalent to supplying the time to the AWS RunInstances API as an idempotency token. (I work on GCE, and asked an industry friend at AWS about how they guarantee idempotency for RunInstances).

Sounds like AWS got it right.

You're entitled to that takeaway, but I disagree. I believe GCP's tendency to use caller-supplied names for resources is one of the single best features of the platform, particularly when compared against AWS's random hex identifiers.

Note that whether this creates collisions is entirely under the customer's control. There's no requirement for global uniqueness, just a requirement that you not try to create two VMs with the same name in the same project in the same zone.

Re: AWS vs. GCP reliability is wildly different

#228

Post by jsolson »

Terretta wrote: post #211
jsolson wrote: post #196

Earlier quoted context omitted.

> When try to create the same resource twice, the second should report success instead of failing. Before I quibble with the idempotency point: I agree with this, entirely, but it is what it is and a lot of software has been written against the current behavior. So I'll cite Hyrum's law here: https://www.hyrumslaw.com/ > GCP control plane is generally not idempotent. The GCE API occupies an odd space here, imo. The r…

It's a lot shorter to write: You're right, we did it wrong. // And paradoxically makes engineers like you.

Sure, except I think that at a macro level we got it more right than AWS, despite some choices that I believe we'd make differently today.

Re: AWS vs. GCP reliability is wildly different

#229

Post by dilyevsky »

deanCommie wrote: post #178
jhugo wrote: post #162

Earlier quoted context omitted.

All the clouds are pretty upfront about availability being non-guaranteed if you don't reserve it. I wouldn't call it a reliability issue if your non-guaranteed capacity takes some tens of seconds to provision. I mean, it might be your reliability issue, because you chose not to reserve capacity, but it's not really unreliability of the cloud — they're providing exactly what they advertise.

"Guaranteed" has different tiers of meaning - both theoretical and practical. In many cases, "guaranteed" just means "we'll give you a refund if we fuck up". SLAs are very much like this. IN PRACTICE, unless you're launching tens of thousands of instances of an obscure image type, reasonable customers would be able to get capacity, and promptly from the cloud. That's the entire cloud value proposition. So no, you can…

Try spinning up 32+ core instances with local ssds attached or anything not n1 family and you will find that in may regions you can only have like single digits of them

Re: AWS vs. GCP reliability is wildly different

#230

Post by [deleted] »

smashed wrote: post #174
sitharus wrote: post #131

Earlier quoted context omitted.

This is a very good point - AWS uses tags to give instances a friendly name, so the name does not have to be unique. The same logic would not fail on AWS.

Which makes 2000% sense. Why would any tenant supplied data affect anything whatsoever? As a tenant, unless you are clashing with another resource under your own name, I don't see the point of failing. aws S3 would be an exception, where they make that limitation on globally unique bucket name very clear.

[deleted]
Post reply on HN