Live data from Hacker News

AWS vs. GCP reliability is wildly different

freeman.vc

241–245 of 245 posts

Re: AWS vs. GCP reliability is wildly different

#241

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

I've naively used millisecond precision things for a long time - not in anything critical I don't think - but I've only recently come to more of an awareness that a millisecond is a pretty long time. Recent example is that I used a timestamp to version a record in a database, but it's feasible that in a Go application, a record could feasilby be mutated multiple times a millisecond by different users / processes / re…

To put this into perspective, a game (or anything) running at 60 FPS only as a bit over 16 milliseconds to render each frame. These days higher frame rates are common enough, often putting you down into single-digit milliseconds per frame. Not think about how many things there are simulated and rendered in each frame for common games. Way more than 16.

Re: AWS vs. GCP reliability is wildly different

#242
post #221

Earlier quoted context omitted.

> I just winced in pain thinking of the ways that can bite you. Something similar caused my favorite bug so far to track down. We were seeing odd spikes in our video playback analytics of some devices watching multiple years worth of video in System.currenTimeMillis() in Java isn't monotonic either is my short answer for what was causing it. Tracking down _what_ was causing it was even more fun though. Devices (phone…

That's a bad day at the office when you have to go and say "hey remember all that data we painstakingly collected and maybe even billed clients for?"

Luckily I worked for the company that made the analytics tools and consumed them!

Bosses actually came to us because our analytics team was trying to figure out who was causing it, because it had been caught by the team doing checks against the data. (a playback period should never have had > 30s of time)

Re: AWS vs. GCP reliability is wildly different

#243
post #77

Earlier quoted context omitted.

I don’t believe this reasoning is used since at least Diane

I haven't seen any real change from Google about how they approach cloud in the past decade (first as an employee and developer of cloud services there, and now as a customer). Their sales people have hollow eyes

Didn’t say they were good at making the switch

Re: AWS vs. GCP reliability is wildly different

#244

There were 84 errors for GCP, but the breakdown says 74 409s and 5 timeouts. Maybe it was 79 409s? Or 10 timeouts? I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was: instance_name = f"gpu-test-{int(time())}" which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can…

> (3) `time.time()` is not monotonic. I just winced in pain thinking of the ways that can bite you. I guess in a cloud/virtualized environment with many short lived instances it isn't even that obscure an issue to run into. A nice discussion on Stack Overflow: https://stackoverflow.com/questions/64497035/is-time-from-ti...

A lot of the places you use time.time and would be bitten by non-monotonicity you probably want something like time.perf_counter, which is useless for measuring absolute time but perfect for calculating time elapsed.

Re: AWS vs. GCP reliability is wildly different

#245

Unclear what the article has to do with reliability. Yes, spinning up machines on GCP is incredibly fast and has always been. AWS is decent. Azure feels like I'm starting a Boeing 747 instead of a VM. However, there's one aspect where GCP is a clear winner on the reliability front. They auto-migrate instances transparently and with close to zero impact to workloads – I want to say zero impact but it's not technically…

EC2 live migrates instances too. Not sure where we are with rollout across the fleet. The reason, from what I understand, why GCP does live migration more is because ec2 focused on live updates instead of live migration. Whereas GCP migrates instances to update servers, ec2 live updates everything down to firmware while instances are running. Curious, what instance types are you using on EC2 that you see so much main…

> Curious, what instance types are you using on EC2 that you see so much maintenance

We use a bunch of different types. M5 and R5 (different sizes) are the most commonly used types but we use many different families. I haven't done an analysis to figure out which types are hotspots.

This is across thousands of instances over many regions worldwide. The percentage is low, but that still translates to daily maintenance alerts.

Post reply on HN