Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS vs. GCP reliability is wildly different

freeman.vc

Post reply on HN
Print view View original

121–130 of 245 posts

Re: AWS vs. GCP reliability is wildly different

#121

Post by endisneigh »

daneel_w wrote: post #118
endisneigh wrote: post #74

this doesn't really seem like a fair comparison, nor is it a measure of "reliability".

It seems entirely fair to me, but the term "reliability" has a few different angles. This time it's not about working or not working, but the ability to auto-scale by invoking resources on the spot, which can be a very real requirement.

unless you're willing to burn $150 a quarter doing this exact assessment, it tells you nothing other than the data center conditions at the time of running.

it would be like doing this in us-central1 when us-central1 is down for one provider, and not another, resulting in increased latency, and saying how much faster one is than the other.

unlike say a throughput test or similar, neither of these services promise particular cold-starts, and so the numbers here cannot be contexutalized against any metric given by either company and so are only useful in the sense that they can be compared, but since there are no guarantees the positions could switch anytime.

that's why I like comparisons between serverless functions where there are pretty explicit SLAs and what not given by each company for you to compare against, as well as one another.

Re: AWS vs. GCP reliability is wildly different

#123

Post by jiggawatts »

roomey wrote: post #26
lbhdc wrote: post #20

Earlier quoted context omitted.

How does migrating a vm to another physical machine work?

VMware has been doing this for years, it's called vmotion and there is a lot of documentation about it if you are interested (eg https://www.thegeekpub.com/8407/how-vmotion-works/ ) Essential, memory state is copied to the new host, the VM is stunned for a millisecond and the cpu states is copied and resumed on the new host (you may see a dropped ping). All the networking and storage is virtual anyway so that is "mov…

The clever trick here is that they'll pre-copy most of the memory without bothering to do it consistently, but mark pages that the source had written to as "dirty". The network cutover is stop-the-world, but VMware doesn't copy the dirty pages during the stop. Instead, it simply treats them as "swapped to pagefile", where the pagefile is actually the source machine memory. When computation resumes at the target, the source is used to page memory back in on-demand. This allows very fast cutovers.

Re: AWS vs. GCP reliability is wildly different

#124

Post by ta20200710 »

dilyevsky wrote: post #67
valleyjo wrote: post #24

Earlier quoted context omitted.

Azure, AWS and GCP all have live migration. VMWare has it too.

Ec2 does not have live migration. On azure it’s spotty so not every maintenance can offer it.

EC2 does support live migration, but it's not public and only for certain instance types/hypervisors.

See: https://news.ycombinator.com/item?id=17815806

Re: AWS vs. GCP reliability is wildly different

#125

Post by devmunchies »

dheera wrote: post #85
eurasiantiger wrote: post #55

Earlier quoted context omitted.

503 service unavailable?

Using HTTP error codes for non-REST things is cringe. 503 would mean the IaaS API calls themselves are unavailable. Very different from the API working perfectly fine but the instances not being available.

What? REST is just some API philosophy, its doesn't even have to be on top of HTTP.

Why would you think HTTP status codes are made for REST? They are made for HTTP to describe the response of the resource you are requesting, and the AWS API uses HTTP so it makes sense to use HTTP status codes.

Re: AWS vs. GCP reliability is wildly different

#126

Post by jiggawatts »

voidfunc wrote: post #114
outworlder wrote: post #103

Unclear what the article has to do with reliability. Yes, spinning up machines on GCP is incredibly fast and has always been. AWS is decent. Azure feels like I'm starting a Boeing 747 instead of a VM. However, there's one aspect where GCP is a clear winner on the reliability front. They auto-migrate instances transparently and with close to zero impact to workloads – I want to say zero impact but it's not technically…

> Azure feels like I'm starting a Boeing 747 instead of a VM. Huh... interesting, this has not been my experience with Azure VM launch times. I'm usually surprised how quickly they pop up.

Depends on your disks.

Premium SSD allows 30 minutes of "burst" IOPS, which can bring down boot times to about 2-5 seconds for a typical Windows VM. The provisioning time is a further 60-180 seconds on top. (The fastest I could get it is about 40 seconds using a "smalldisk" image to ephemeral storage, but then it took a further 30 seconds or so for the VM to become available.)

Standard HDD was slow enough that the boot phase alone would take minutes, and then the VM provisioning time is almost irrelevant in comparison.

Re: AWS vs. GCP reliability is wildly different

#127

Post by rwiggins »

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts?

I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was:

    instance_name = f"gpu-test-{int(time())}"
which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can have weird boundary cases, particularly because (1) it does cleanup after creation, which will have variable latency, (2) `int()` will truncate the fractional part of the second from `time()`, and (3) `time.time()` is not monotonic.

I would not ask the author to spend money to test it again, but I think the 409s would probably disappear if you replaced `int(time())` with `uuid.uuid4()`.

Disclosure: I work at Google - on Google Compute Engine. :-)

Re: AWS vs. GCP reliability is wildly different

#128

Post by lbhdc »

GauntletWizard wrote: post #113
lbhdc wrote: post #73

Earlier quoted context omitted.

When its transferring the state to the target, how does it handle memory updates that are happening at that time? Is the programs execution paused at that point?

No, but the memory accesses have hooks that say "This memory was written". Then, program execution is paused, and just the sections of memory that were written are copied again. This has memory performance characteristics - I ran a benchmark of memory read/write speed while this was happening once. It more than halved memory speed for the 30s or so it took from migration started to migration complete. The pause, too,…

Ahh I think that was the piece I was missing, thanks! I didn't realize there were hooks for tracking memory changes.

Re: AWS vs. GCP reliability is wildly different

#129

Post by fragmede »

dekhn wrote: post #95
kccqzy wrote: post #23

Heard from a Googler that the internal infrastructure (Borg) is simply not optimized for quick startup. Launching a new Borg job often takes multiple minutes before the job runs. Not surprising at all.

A well-configured isolated borg cluster and well-configured job can be really fast. If there's no preemption (IE, no other job that is kicked off and gets some grace period), the packages are already cached locally, and there is no undue load on the scheduler, the resources are available, and it's a job with tasks, rather than multiple jobs, it will be close to instantaneous. I spend a significant fraction of my 11+…

Psh someone's bragging about not being at batch priority.

Re: AWS vs. GCP reliability is wildly different

#130

Post by citizenpaul »

lacker wrote: post #116

Anecdotally I tend to agree with the author. But this really isn't a great way of comparing cloud services. The fundamental problem with cloud reliability is that it depends on a lot of stuff that's out of your control, that you have no visibility into. I have had services running happily on AWS with no errors, and the next month without changing anything they fail all the time. Why? Well, we look into it and it turn…

That was my thoughts. People are probably pummeling GCP GPU free tier right now with stable diffusion image generators. Since it seems like all the free plug and play examples use the google python notebooks.
Post reply on HN