Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

381–390 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#381

Earlier quoted context omitted.

Each platform has features the other platform doesn't, even though AWS has more. Some of GCP's unique compelling features include live VM migration that makes it less relevant when a host has to reboot, the new life that has recently been put into Google App Engine (both flexible environment and the second generation standard environment runtimes), the global load balancer with a single IP and no pre-warming, and Clo…

Redshift is not a direct competitor with BigQuery. It's a relational data warehouse. BigQuery more directly competes with Athena, which is a managed version of Apache Presto, and my personal opinion is that Athena is way better than BigQuery because I can query data that is in S3 (object storage) without having to import it into BigQuery first. Disk and network performance is extremely consistent with AWS so long as…

The comments I made about Redshift vs BigQuery and about disk/network/etc reflect current opinions of colleagues who use AWS currently (or recently in some cases) and extensively, not 5-6 year old opinions. Even my own last use of AWS was maybe 2-3 years ago, when Redshift was AWS's closest competitor to BigQuery and when I saw disk/network issues directly.

You're right that Athena seems like the current competitor to BigQuery. This is one of those things that are easy to overlook when people made the comparison as recently as a couple of years ago (before Athena was introduced) and Redshift vs BigQuery is still often the comparison people make. This is where Amazon's branding is confusing to the customer: so many similar but slightly different product niches, filled at different times by entirely different products with entirely unrelated names.

When adding features, GCP would usually fill adjacent niches like "serverless Redshift" by adding a serverless mode to Redshift, or something like that, and behavior would be mostly similar. Harder to overlook and less risky to try.

Meanwhile, when Athena was introduced, people who had compared Redshift and BigQuery and ruled out the former as too much hassle said "ah, GCP made Amazon introduce a serverless Redshift. But it's built on totally different technology. I wonder if it will be one of the good AWS products instead of the bad ones." (Yes, bad ones exist. Amazon WorkMail is under the AWS umbrella but basically ignored, to give one example.)

And then they go back to the rest of their day, since moving products (whether from Redshift or BigQuery) to Athena would not be worth the transition cost, and forget about Athena entirely.

On the disk/network question, no I didn't see performance problems with provisioned IOPS volume types, but that doesn't matter: for GCE's equivalent of EBS magnetic storage, they do indeed give what they promise, at way less cost than their premium disk types. There's no reason it isn't a fair comparison.

And for the "instance" part of my EC2 performance comment, I was referring to a noisy neighbor problem where sometimes a newly created instance would have much worse CPU performance than promised and so sometimes delete and recreate was the solution. GCE does a much better job at ensuring the promised CPUs.

I'm glad AWS and GCP have lots of features, improve all the time, and copy each other when warranted. But I don't think the general thrust of my comparison has gone invalid, even if my recent data is more skewed toward GCP and my AWS data is skewed toward 2-3 years old. Only the specifics have changed (and the feature gap narrowed with respect to important features).

Re: Google Kubernetes Engine's third consecutive day of service disruption

#382

Earlier quoted context omitted.

> You could go multi cloud though. Multi cloud is almost always more pain than gain. You’d spend time and effort abstracting away the value that a cloud provider brings in canned services. Hell, multi region is often more than many workloads need.

My entire infrastructure is on k8s which should make multi-cloud easy... Nope.

So you’re not using any managed services?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#383

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

When everything works, GCP is the best. Stable, fast, simple, reliable. When things stop working, GCP is the worst. Slow communications and they require way too much work before escalating issues or attempting to find a solution. They already have the tools and access so most issues should take minutes for them to gather diagnostics, but instead they keep sending tickets back for "more info", inevitably followed by a…

Heh, your "test" reminds me of an old Hanselman article:

https://www.hanselman.com/blog/FizzBinTheTechnicalSupportSec...

Re: Google Kubernetes Engine's third consecutive day of service disruption

#384
post #379

Earlier quoted context omitted.

Very similar thing at our office. Considering the scale of which we run things, any outage could be a potential loss of millions _every minute_. Sure, we use support tickets with vendors for small things. Console button bugging out, etc. But for large incidents, every vendor has a representative within an hour driving distance and will be called into a room with our engineers to fix the problem. This kind of outage,…

Going single cloud on that scale is simply irresponsible though. You need failovers to different providers and hopefully also have your hardware for general workloads And suddenly the CEO doesn't care anymore if one of your potential failovers is behaving flaky in specific circumstances Not saying it's good as it is.. communication as a saas provider is - as you said- one is the most important things... But this spec…

Agree, if we are really talking about millions per minute (woah), then you can afford to failover to AWS.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#385

Earlier quoted context omitted.

My entire infrastructure is on k8s which should make multi-cloud easy... Nope.

So you’re not using any managed services?

Other than kubernetes and object storage, no.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#386

Earlier quoted context omitted.

GCP has a few features that set it apart from other cloud providers. GKE is head and shoulders above the other offerings from AWS and Azure. GCP can be a fair bit cheaper than AWS and Azure for certain workloads. Raw compute/memory is about the same. Storage can make a big difference. GCP persistent SSD costs a bit more than AWS GP2 with much better performance and way cheaper than IO2. Local SSD is also way, way che…

I have a more detailed post above, but if you are large enough, you’re not paying the listed price for AWS. But even if you are, prices change all of the time. From a completely selfish standpoint, is the price difference worth the cost to bet your reputation on if you are the one that made the final decision? Even if statistically the same could happen with AWS, no one would blame you for choosing AWS. However, I co…

Discounts included, my previous statements about cost still stand. That said, I don't think the cost differences should incentivize anyone to move unless they're operating at a scale that would make a substantial difference. Disk IO in AWS can be prohibitively expensive, for example. Any business that relied heavily on that would benefit by looking at local SSD on GCP.

There are lots of little things to like about GCP that are superior to AWS. Network IO, some of the bigdata products. Not having to deal with IAM. In the end it would be some combination of those things that should drive the decision. Basic enterprise IT shops moving to "cloud" should choose AWS 90% of the time.

Anyone starting from scratch on kubernetes or considering shifting all of their infrastructure to it should absolutely choose GKE. Anyone currently in EKS or AKS should sign up for GCP today and evaluate the differences to see what they're missing.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#387

Earlier quoted context omitted.

Your response is from a geek’s viewpoint. No insult, intended, I’m first and foremost a 30 year computer geek myself - started programming in 65C02 assembly in 6th grade and still mostly hands on. But, whether it is right or not, as an architect/manager, etc, you have to think about what’s not just best technically. You also have to manage your reputational risks if things go south and less selfishly, how quickly can…

> The equivalent would be spot instances on AWS. They're equivalent in the sense that you have nodes that can die anytime, but it's much more complicated. You could technically have a much lower cost on AWS by aggressively bidding low but we've had a few instances where the node only lived a few minutes. Preemptibles nodes are max 24h, and from our stats, they really live around that amount of time. I think the lowes…

Not much to add here other than this reflects my experience as well.

Moved for bizdev reasons, and really appreciated the improved quality of life.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#388
post #250

Earlier quoted context omitted.

- Native integration with G-Suite as an identity provider. Unified permissions modeling from the IDP, to work apps like email/Drive, to cloud resources, all the way into Kubernetes IAM. - Security posture. Project Zero is class leading, and there's absolutely a "fear-based" component there, with the open question of when Project Zero discovers a new exploit, who will they share it with before going public? The upcomi…

Your response is from a geek’s viewpoint. No insult, intended, I’m first and foremost a 30 year computer geek myself - started programming in 65C02 assembly in 6th grade and still mostly hands on. But, whether it is right or not, as an architect/manager, etc, you have to think about what’s not just best technically. You also have to manage your reputational risks if things go south and less selfishly, how quickly can…

Your response is from an IT consumer's viewpoint.

"Cloud" is not a thing one buys and one's reputation has nothing to do with the reliability of the services consumed, but the reliability of the services provided.

To put it more succinctly, "you own your availability".

In the end, "cloud" is a commodity and all cloud providers are trying to get vendor lock-in. My goal as a manager is not to couple my business revenue linearly to any particular product or service.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#389

Earlier quoted context omitted.

That shouldn't, but they do. S3 goes down [1]. The AWS global console goes down, right after Prime Day outages [2]. Lots of Google Cloud services go down [3, current thread]. Tens of Azure services go down hard [4]. Are software development and release processes improving to mitigate these outages? We don't know. You have to trust the marketing. Will regions ever be fully isolated? We don't know. Will AWS IAM and con…

Can't speak for Google, but Facebook and Salesforce chose Cells for HA. http://highscalability.com/blog/2012/5/9/cell-architectures....

Doesn't look like it was all that helpful to Facebook (as of 1542038976). Facebook.com errors out currently.

> Facebook Platform Appears to be down

> A check of https://developers.facebook.com/status/dashboard/ returns an error and I'm unable to login with facebook to some of my mobile apps.

https://news.ycombinator.com/item?id=18434262

Re: Google Kubernetes Engine's third consecutive day of service disruption

#390

Earlier quoted context omitted.

So you’re not using any managed services?

Other than kubernetes and object storage, no.

It could work then. We had the same setup. Also, we store everything on S3 but there is a lambda function that pushes new objects to Google Cloud storage too. We figured, storage is cheap so we can duplicate things and it will make moving to GCP a lot easier.

This was a while back though. Now we depend on a lot more AWS stuff.

Post reply on HN