Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes a black hole of unpredictable spend, according to new report

theregister.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: Kubernetes a black hole of unpredictable spend, according to new report

#61

Post by frompdx »

alongub wrote: post #46

I wrote a tool that helps estimate K8s costs by simulating K8s clusters. You write your pods in a simple DSL and it runs kube-scheduler without actual nodes behind the scenes. It's still really basic but I'd love to hear your feedback! https://github.com/aporia-ai/kubesurvival

I think this is a great idea. I can see the DSL being very useful for situations where you want to think about hypotheticals. My main piece of feedback is that you should consider supporting the ability to feed in your existing k8s manifests as input to this tool instead in addition to the DSL. I think that would make this tool very appealing and very easy to onboard users with existing clusters who are looking to reduce their costs.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#62

Post by bogota »

Jochim wrote: post #50
bogota wrote: post #36

Earlier quoted context omitted.

Unfortunately AWS is the new oracle. No one ever gets in trouble for picking it and its a great way to make it look like you as a high up exec provide value. Look how fast we are iterating now with my decision. It almost always ends in a mess of unmaintainable unthought out services that someone else has to come and clean up or move to the next proprietary service. The last 5 years for me has been soul crushing as so…

The problem with running your own DC is growing past your planned capacity. There's often a huge delay between developers having to put up with the VM infrastructure having to put up with under-resourced machines and more capacity being approved. As a developer I've put up with over-subscribed VMware clouds and I vastly prefer the Azure/AWS option.

Their are a lot of bad ways to run your own hardware. Limiting your infra to EC2 for burst capacity provides an easy escape hatch if you need. However i have never run into the issue of not calculating CPU capacity properly and also would never use VM ware that sounds like IT is running your DC. I could see this being the case for some tiny startup who just owns a small amount of rack space though.

Of course you can pay the cloud providers to deal with your companies bad planning. Thats what most do.

I also would never advocate for DC for everything. As a startup it likely makes no sense to run your own hardware and also likely doesnt make sense to run k8s also however one of those is completely acceptable. Once you get to more predictable growth owning your own hardware starts to look more attractive but most don’t know how to calculate it properly and finance likes to make it merky with capex and opex buckets.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#63

Post by orf »

cobookman wrote: post #49
sleepybrett wrote: post #30

Earlier quoted context omitted.

I just had a meeting with a 'finops' manager where i showed him how kubernetes has a similar tagging structure (labels) and how we can break down per team pricing based on cpu/memory utilization. It's not hard, you just need the tools (kubecost, etc)

pod A uses 2 cores, pod B uses 1 core. Machine has 4 cores and all remaining unscheduled pods require 2 cores. How do you attribute the partial usage of the node? Is it 2 cores billed to pod A, 1 core billed to pod B, and 1 core billed to some random team? Or do you have 2/3 of Node billed to pod A and 1/3 of Node billed to Pod B. Now deal with this permutation across all the various variables.

You do it roughly based on the deployment requests and the average HPA values throughout a time slice.

Most k8s workloads run on a homogenous set of node types, so you can have an hourly cost per gb and per vcpu without digging too much into it.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#64

Post by bogota »

thanatos519 wrote: post #54
bogota wrote: post #36

Earlier quoted context omitted.

Unfortunately AWS is the new oracle. No one ever gets in trouble for picking it and its a great way to make it look like you as a high up exec provide value. Look how fast we are iterating now with my decision. It almost always ends in a mess of unmaintainable unthought out services that someone else has to come and clean up or move to the next proprietary service. The last 5 years for me has been soul crushing as so…

I recently experienced this firsthand, in a company which owned no computers beyond employee laptops. The product was entirely built of AWS services created by a pile of Terraform spaghetti. It was only really understood by someone whose superpower was the ability to keep an apparently unlimited number of levels of indirection in his head. I hear they might need to move it all to Azure soon!

Terraform just does too much and is abused. Its great for simple config but quickly turns into custom modules to make things easier. Eventually those modules need to change and stuff breaks but you never know until someone trys to run it again. Then you just pray no one trys to manage their database with it.

Additionally as its put together over time if you actually had to re create the environment it would never work. I spent time automating recreating an environment and quickly stopped. Terraform is the illusion of infra as code and fails miserably at any scale.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#65

Post by bogota »

Retric wrote: post #52
bogota wrote: post #36

Earlier quoted context omitted.

Unfortunately AWS is the new oracle. No one ever gets in trouble for picking it and its a great way to make it look like you as a high up exec provide value. Look how fast we are iterating now with my decision. It almost always ends in a mess of unmaintainable unthought out services that someone else has to come and clean up or move to the next proprietary service. The last 5 years for me has been soul crushing as so…

Many people have been fired after AWS migrations resulted in massive cost increases. But, few people want to talk about failed projects at hospitals etc where the cloud is a poor fit. IT has always had issues with people cargo cutting solutions without understanding the details, and the cloud is no different.

I agree its just the current one that affects me. We did have someone get fired from an AWS migration at my first startup but they just turned around and tried to do again. It took one year for a single application that only lived on a single server and we never did anything more advanced before we sold.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#66

Post by 908B64B197 »

WYepQ4dNnG wrote: post #42

I have experienced first hand several cases of k8s gone wrong. In the end I have come to the conclusion that most companies don't really need the complexity of k8s. Seriously, most k8s projects I have been involved with required so much effort to bootstrap and keep it going, it just blew me away! The experience for the average developer was just frustrating and infuriating: AWS ECS to the rescue! Some will argue: ven…

> I have seen companies failing because investing so much in building infrastructure, supposly vendor lock in free (or so they thought) that they lost sight and did not invest enough building the actual product: no revenue -> party is over.

How was the stock comp for engineers at these companies?

Re: Kubernetes a black hole of unpredictable spend, according to new report

#67

Post by WYepQ4dNnG »

908B64B197 wrote: post #66
WYepQ4dNnG wrote: post #42

I have experienced first hand several cases of k8s gone wrong. In the end I have come to the conclusion that most companies don't really need the complexity of k8s. Seriously, most k8s projects I have been involved with required so much effort to bootstrap and keep it going, it just blew me away! The experience for the average developer was just frustrating and infuriating: AWS ECS to the rescue! Some will argue: ven…

> I have seen companies failing because investing so much in building infrastructure, supposly vendor lock in free (or so they thought) that they lost sight and did not invest enough building the actual product: no revenue -> party is over. How was the stock comp for engineers at these companies?

late stage "startup" who had struggled with growth for years, they thought they needed to invest in building a more complex system and infrastructure, and they hired tons of engs (who actually had literally nothing to do) ...

instead what they really needed was to focus on their customers and build them a useful product that simply worked. They spent months building and deploying their own k8s cluster: EKS was "vendor lock in" so that wasn't a good choice for them, but guess what, all their infrastructure was already running in AWS anyway and their product was already vendor locked in: RDS, S3, etc ...

also ... to make things even more complex, they thought they needed to go all-in super distributed micro services: it took literally months to get new "services" up and running in production. It was a s*t show!

One of the many story of let's break the monolith, embrace microservice, thus k8s ... gone horribly wrong.

Eventually most engs left ....

Re: Kubernetes a black hole of unpredictable spend, according to new report

#68

Post by 908B64B197 »

WYepQ4dNnG wrote: post #67
908B64B197 wrote: post #66

Earlier quoted context omitted.

> I have seen companies failing because investing so much in building infrastructure, supposly vendor lock in free (or so they thought) that they lost sight and did not invest enough building the actual product: no revenue -> party is over. How was the stock comp for engineers at these companies?

late stage "startup" who had struggled with growth for years, they thought they needed to invest in building a more complex system and infrastructure, and they hired tons of engs (who actually had literally nothing to do) ... instead what they really needed was to focus on their customers and build them a useful product that simply worked. They spent months building and deploying their own k8s cluster: EKS was "vendo…

> but guess what, all their infrastructure was already running in AWS anyway and their product was already vendor locked in: RDS, S3, etc ...

I laughed, but your response doesn't answer my question regarding engineers comp.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#69

Post by rantwasp »

flowerlad wrote: post #53
WYepQ4dNnG wrote: post #42

I have experienced first hand several cases of k8s gone wrong. In the end I have come to the conclusion that most companies don't really need the complexity of k8s. Seriously, most k8s projects I have been involved with required so much effort to bootstrap and keep it going, it just blew me away! The experience for the average developer was just frustrating and infuriating: AWS ECS to the rescue! Some will argue: ven…

> most companies don't really need the complexity of k8s If you want to run a complex service consisting of multiple microservices, auto scaling and so on, nothing beats Kubernetes. But you're right, most small businesses just need a simple web site, and for them, an Amazon Lightsail VM might suffice.

yeah, nothing beats K8s except you know... maybe serverless. or maybe cloud primitives that are already there and are handled by someone else.

IMHO, unless you have a huge fleet of bare metal machines K8s is an overhead you don't need

Re: Kubernetes a black hole of unpredictable spend, according to new report

#70

Post by rantwasp »

WYepQ4dNnG wrote: post #67
908B64B197 wrote: post #66

Earlier quoted context omitted.

> I have seen companies failing because investing so much in building infrastructure, supposly vendor lock in free (or so they thought) that they lost sight and did not invest enough building the actual product: no revenue -> party is over. How was the stock comp for engineers at these companies?

late stage "startup" who had struggled with growth for years, they thought they needed to invest in building a more complex system and infrastructure, and they hired tons of engs (who actually had literally nothing to do) ... instead what they really needed was to focus on their customers and build them a useful product that simply worked. They spent months building and deploying their own k8s cluster: EKS was "vendo…

> Eventually most engs left ....

and now have k8s on their resume. Bazinga!

Post reply on HN