Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS vs. GCP reliability is wildly different

freeman.vc

Post reply on HN
Print view View original

131–140 of 245 posts

Re: AWS vs. GCP reliability is wildly different

#131

Post by sitharus »

rwiggins wrote: post #127

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

This is a very good point - AWS uses tags to give instances a friendly name, so the name does not have to be unique. The same logic would not fail on AWS.

Re: AWS vs. GCP reliability is wildly different

#133

Post by _msw_ »

ta20200710 wrote: post #124
dilyevsky wrote: post #67

Earlier quoted context omitted.

Ec2 does not have live migration. On azure it’s spotty so not every maintenance can offer it.

EC2 does support live migration, but it's not public and only for certain instance types/hypervisors. See: https://news.ycombinator.com/item?id=17815806

Here's a comment that I made in a past thread.

https://news.ycombinator.com/item?id=26650082

Re: AWS vs. GCP reliability is wildly different

#134

Post by mempko »

rwiggins wrote: post #127

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

What are your thoughts on the generally slower launch times with a huge variance on GCP?

Re: AWS vs. GCP reliability is wildly different

#135

Post by Animats »

> GCP allows you to attach a GPU to an arbitrary VM as a hardware accelerator - you can separately configure quantity of the CPUs as needed.

That would seem to indicate that asking for a VM on GCP gets you a minimally configured VM on basic hardware, and then it gets migrated to something bigger if you ask for more resources. Is that correct?

That could make sense if, much of the time, users get a VM and spend a lot of time loading and initializing stuff, then migrate to bigger hardware to crunch.

Re: AWS vs. GCP reliability is wildly different

#136

Post by verdverm »

danielmarkbruce wrote: post #119
eurasiantiger wrote: post #55

Earlier quoted context omitted.

503 service unavailable?

It's not the service that's unavailable. The resource isn't available. The service is running just fine.

GCP error messages will indicate if resources were not available, if you reached your quota, or if it was some other error. Tests like OP can differentiate these situations

Re: AWS vs. GCP reliability is wildly different

#137

Post by orf »

AWS has different pools of EC2 instances depending on the customer, the size of the account and any reservations you may have.

Spawning a single GPU at varying times is nothing. Try spawning more than one, or using spot instances, and you’ll get a very different picture. We often run into capacity issues with GPU and even the new m6i instances at all times of the day.

Very few realistic company size workloads need a single GPU. I would willingly wait 30 minutes for my instances to become available if it meant all of them where available at the same time.

Re: AWS vs. GCP reliability is wildly different

#138

Post by _msw_ »

xyzzyz wrote: post #7
politelemon wrote: post #3

A few weeks ago I needed to change the volume type on an EC2 instance to gp3. Following the instructions, the change happened while the instance was running. I didn't need to reboot or stop the instance, it just changed the type. While the instance was running. I didn't understand how they were able to do this, I had thought volume types mapped to hardware clusters of some kind. And since I didn't understand, I wasn'…

Dunno about AWS, but GCP uses live migration, and will migrate your VM across physical machines as necessary. The disk volumes are all connected over the network, nothing really depends on the actual physical machine your VM is ran on.

Disclosure: I work for Amazon, and in the past I worked directly on EC2.

From the FAQ: https://aws.amazon.com/ec2/faqs/

Q: How does EC2 perform maintenance?

AWS regularly performs routine hardware, power, and network maintenance with minimal disruption across all EC2 instance types. To achieve this we employ a combination of tools and methods across the entire AWS Global infrastructure, such as redundant and concurrently maintainable systems, as well as live system updates and migration.

Re: AWS vs. GCP reliability is wildly different

#139

Post by Crash0v3rid3 »

mempko wrote: post #134
rwiggins wrote: post #127

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

What are your thoughts on the generally slower launch times with a huge variance on GCP?

The author failed to mention which regions these tests were run. GPU availability can vary depending on the regions that were tested for both Cloud providers.

Re: AWS vs. GCP reliability is wildly different

#140

Post by mr_toad »

herpderperator wrote: post #122

The author is using 'Quantile' which I hadn't heard of before, and when I did, it seems like it actually should be 'Percentile'. Percentiles are the percentages, which is what the author is referring to.

Quantiles are a generic term for percentiles, deciles, quartiles etc. Percentiles would have been a more precise term.
Post reply on HN