[deleted]
Google Kubernetes Engine's third consecutive day of service disruption
371–380 of 419 posts
Re: Google Kubernetes Engine's third consecutive day of service disruption
#372Earlier quoted context omitted.
"Support costs" calculation often doesn't include the costs of not having support. When I worked at GoDaddy, there were around 2/3 of the company was customer support. At the current company I'm at, a cryptocurrency exchange, our support agents frequently hear they prefer our service over others because of our fast support response times (crypto exchanges are notorious for really poor support). All of my interactions…
Google hasn't learned this lesson. They have though; they've just drawn the conclusion that they'd rather put massive amounts of effort in to building services that users can use without needing support. This approach works well once the problems have been ironed out, but it's horrible until that's the case. Google's mature products like Ads, Docs, GMail, etc are amazing. Their new products ... aren't.
I've seen Google projects where the project lead explained (actually responded) that they don't want to provide support, end of story. Google puts folks in charge but does not give them enough in the way accountability objectives.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#373Earlier quoted context omitted.
If all you are doing with your cloud provider are a bunch of VMs, you’re wasting money. There are much cheaper and simpler ways just to host a bunch of VMs than to use one of the major cloud providers. Are you not using any of thier managed services and are you maintaining your own on VMs? If so, you have the worse of both worlds. You’re spending more on hosting and you’re not saving money on letting someone else do…
Going to the cloud thinking you are going to save money is silly. It is more expensive to run things in the cloud, what you are paying for is the ability to scale up and down without a logistics problem.
It’s only silly if you’re not changing your processes. If you’re keeping the same people doing everything manually instead of automating, or honestly if you’re keeping the same headcount and if you’re not developing and depending on services provided by your provider and instead you’re self hosting services on VMs, of course you’re going to spend more.
But I blame most of the cost overruns when using cloud providers on “consultants” who think they are “moving to the cloud” when all they really know is how to setup a little networking infrastructure and know nothing about how to use the developer, Devops, or other hosted solutions.
Most ”consultants” I’ve run across only know how to do a lift and shift and do a one to one mapping of the on prem VMs and networking infrastructure to the cloud. They know nothing about automation, transforming Devops practices, or transforming development practices and architecture.
A lift and shift should only be the first phase.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#374Earlier quoted context omitted.
Assume I’m not fully informed here. What does “written properly” mean? Sure I can move over route53 to cloud DNS easily but Firehose to PubSub? Lambda to Cloud Functions? DynamoDB to BigTable and moving the data? The syntax for provisioning these doesn’t work that well for some find and replace to work. Are you using a templater to generate cloud-specific HCL from a template or something? Sounds like a pretty big pro…
Quit using vendor lock in resources then asking why its hard to leave that vendor.
You’re always “locked in” to your infrastructure. People have this brilliant strategy of thinking they have this brilliant strategy of avoiding “lock in” and again they get the worse of both worlds. They end up spending more to avoid lock-in and they aren’t taking advantage of what thier provider has to offer.
At the end of the day, changing your underlying infrastructure is so risky and usually not worth the cost benefit analysis, it’s rarely done.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#375Earlier quoted context omitted.
> instead they keep sending tickets back for "more info" Isn't that the case with basically every support request, no matter the company or severity? The first couple of emails from 1st & even 2nd level support are mostly about answering the same questions about the environment over and over again. We've had this ping-pong situation with production outages (which we eventually analysed and worked around by ourselves)…
I've definitely had interactions with smaller companies where you can effectively bypass first and second line by demonstrating you know what you're doing, mostly just saying the right things for them to accept that you've done basic troubleshooting steps already and really do need to talk to someone beyond that point.
> I've definitely had interactions with smaller companies where you can effectively bypass first and second line by demonstrating you know what you're doing, mostly just saying the right things for them to accept that you've done basic troubleshooting steps already and really do need to talk to someone beyond that point.
"Shibboleet" https://www.xkcd.com/806/
Re: Google Kubernetes Engine's third consecutive day of service disruption
#376Right now we're experiencing an issue where a small percentage of end users on our GKE site are getting super slow speeds. The issue is ISP related as they can switch to a 4G hot spot in the same location and get normal speeds... and inside our system the timing looks normal. So there's a slowdown either TO the load balancer or FROM the load balancer. Took a week to convince Google's support contractor to even believe it wasn't an issue with our site and their advice is generally along the lines of Turn it off and Turn it back on again (which might actually fix the problem) though that's easier said than done in GCP.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#377A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?
Infrastructure as code. Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services. This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work
And this is the reason that people end up spending so much more in the cloud - treating AWS like a glorified colo.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#378Earlier quoted context omitted.
I think you're going to have to back up a claim like this with some facts. GKE being the exception, since it was launched a couple years before EKS. AWS clearly has way more services, and the features are way deeper than GCP. Just compare virtual machines and managed databases, AWS has about 2-3x more types of VMs (VMs with more than 4TB of RAM, FPGAs, AMD Epyc, etc.), and in databases, more than just MySQL and Postg…
Each platform has features the other platform doesn't, even though AWS has more. Some of GCP's unique compelling features include live VM migration that makes it less relevant when a host has to reboot, the new life that has recently been put into Google App Engine (both flexible environment and the second generation standard environment runtimes), the global load balancer with a single IP and no pre-warming, and Clo…
Redshift is not a direct competitor with BigQuery. It's a relational data warehouse. BigQuery more directly competes with Athena, which is a managed version of Apache Presto, and my personal opinion is that Athena is way better than BigQuery because I can query data that is in S3 (object storage) without having to import it into BigQuery first.
Disk and network performance is extremely consistent with AWS so long as you use newer instance types and storage types. You can't reasonably compare the old EBS magnetic storage to the newer general purpose SSD and provisioned IOPS volume types, and likewise, newer instances get consistent non-blocking 25gbps network performance.
I'm not so sure I would praise our documentation; it is one of the areas that I wish we were better at. Some of the less used services and features don't have excellent documentation, and in some cases you really have to figure it out on your own.
GCP is a pretty nice system overall, but most of the time when I see comparisons, when GCP looks better its because the person making the comparison is comparing the AWS they remember from 5-6 years ago with the GCP of today, which would be like comparing GAE from 2012 with today.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#379Earlier quoted context omitted.
As someone who works for Government and Enterprise - all I care about sometimes is how a company behaves when everything goes wrong. The issue with outages for the Government organizations I have dealt with is rarely the outage itself - but strong communication about what is occurring and realistic approximate ETAs, or options around mitigation. Being able to tell the Directors/Senior managers that issues have been "…
Very similar thing at our office. Considering the scale of which we run things, any outage could be a potential loss of millions _every minute_. Sure, we use support tickets with vendors for small things. Console button bugging out, etc. But for large incidents, every vendor has a representative within an hour driving distance and will be called into a room with our engineers to fix the problem. This kind of outage,…
Going single cloud on that scale is simply irresponsible though.
You need failovers to different providers and hopefully also have your hardware for general workloads
And suddenly the CEO doesn't care anymore if one of your potential failovers is behaving flaky in specific circumstances
Not saying it's good as it is.. communication as a saas provider is - as you said- one is the most important things... But this specific issue was not as bad as some people insinuate in this thread
Re: Google Kubernetes Engine's third consecutive day of service disruption
#380Earlier quoted context omitted.
Their approach to things like GCP is very different than their approach to those other areas of Google. But they don't separate their branding or unify their deprecation attitudes enough to avoid cross-org-chart reputation damage like this.
So basically they screw you over on either medium or hard, and according to you the real problem is them not telling us clearly whether or not we receive the medium or hard screw over option. By the way that GCP is so full of loopholes where Google can get out of its obligations its laughable. So it's not even that clear cut that the GCP is really a better alternative. And even when it turns out to be legally sound,…
Oh, courts routinely give binding weight to words like Google's deprecation policy uses, and any large megacorp who is sufficiently badly impacted by a legalese violation (though SLA issues and deprecation issues are two separate things) wouldn't be scared away from a lawsuit by Google being big. I can imagine EU regulatory action or a class action lawsuit as other possible mechanisms.
But as I say in another comment, the contract is less important than both trust and reality. Keep in mind nobody focuses on how AWS doesn't even have a public deprecation policy.
I'm right there with many people in this thread in agreeing that Google has a trust problem, due mostly to real perception issues stemming from Google's habits outside GCP, which can and do impact people's perceptions of what they'll do with GCP.
The reality of what Google has done and will do with GCP, though, is pretty good. Sure they do sometimes deprecate things in ways Amazon never would. But not nearly as often or as abruptly as they do on the consumer side - that would be commercial suicide - and they do other things better than Amazon. Tradeoffs.