Kubernetes a black hole of unpredictable spend, according to new report • Hacker News Board

Re: Kubernetes a black hole of unpredictable spend, according to new report

#81

Post by me_me_me » Wed, Jun 30, 2021, 9:50 AM UTC

WYepQ4dNnG wrote: post #42

I have experienced first hand several cases of k8s gone wrong. In the end I have come to the conclusion that most companies don't really need the complexity of k8s. Seriously, most k8s projects I have been involved with required so much effort to bootstrap and keep it going, it just blew me away! The experience for the average developer was just frustrating and infuriating: AWS ECS to the rescue! Some will argue: ven…

Thank you. At least I am not alone. I just want to write some code and not waste a week redeploying K8s shit again... send help...

Re: Kubernetes a black hole of unpredictable spend, according to new report

#82

Post by alongub » Wed, Jun 30, 2021, 10:47 AM UTC

frompdx wrote: post #61

alongub wrote: post #46

I wrote a tool that helps estimate K8s costs by simulating K8s clusters. You write your pods in a simple DSL and it runs kube-scheduler without actual nodes behind the scenes. It's still really basic but I'd love to hear your feedback! https://github.com/aporia-ai/kubesurvival

I think this is a great idea. I can see the DSL being very useful for situations where you want to think about hypotheticals. My main piece of feedback is that you should consider supporting the ability to feed in your existing k8s manifests as input to this tool instead in addition to the DSL. I think that would make this tool very appealing and very easy to onboard users with existing clusters who are looking to re…

For sure, this is definitely planned!

Re: Kubernetes a black hole of unpredictable spend, according to new report

#83

Post by jakupovic » Wed, Jun 30, 2021, 11:07 AM UTC

rantwasp wrote: post #69

flowerlad wrote: post #53

Earlier quoted context omitted.

> most companies don't really need the complexity of k8s If you want to run a complex service consisting of multiple microservices, auto scaling and so on, nothing beats Kubernetes. But you're right, most small businesses just need a simple web site, and for them, an Amazon Lightsail VM might suffice.

yeah, nothing beats K8s except you know... maybe serverless. or maybe cloud primitives that are already there and are handled by someone else. IMHO, unless you have a huge fleet of bare metal machines K8s is an overhead you don't need

Thanks for your HO but you're wrong. K8s is the future and the new JVM.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#84

Post by bogota » Wed, Jun 30, 2021, 3:21 PM UTC

williamxd3 wrote: post #78

bogota wrote: post #64

Earlier quoted context omitted.

Terraform just does too much and is abused. Its great for simple config but quickly turns into custom modules to make things easier. Eventually those modules need to change and stuff breaks but you never know until someone trys to run it again. Then you just pray no one trys to manage their database with it. Additionally as its put together over time if you actually had to re create the environment it would never wor…

You know you can lock modules versions, right? And of course the environment recreation needs to be tested periodically if you have a disaster recovery plan.

Locked modules only works if all of your modules are pulled in through the registry. that isn't an option at some places.

Testing environment recreation is impossible at a certain scale. Try it when you have hundreds of people adding terraform code most of who only know how to copy and paste.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#85

Post by jokethrowaway » Wed, Jun 30, 2021, 8:33 PM UTC

Agingcoder wrote: post #56

dan_quixote wrote: post #48

Earlier quoted context omitted.

Are those of us happy/comfortable with current major cloud offerings simply not speaking up here? I can't fathom running a data center any longer for a company of nearly any size. Given my team's responsibilities, this would require 2-3x more headcount with significantly worse SLA/SLO if we still ran our own datacenter. Maybe it's not such a big deal for places with constant demand? Or is this just a case of observer…

It depends on what you do. Cloud boxes are insanely expensive (easily 10x the price of the equivalent in house box, taking hosting, power, cooling, hw into account). To make this work, you need a combination of variable demand, and only paying for partial salaries (your cloud boxes are mutualized with other people's boxes). If you're a reasonably big company (tens of thousands of servers) , with fairly stable demand,…

I think you can find better cloud prices (outside of aws) that make sense. Running a datacenter is hard, running your own servers (software only) is much easier.

I can get this dedicated server in Germany (courtesy of hetzner.com) for 40 eur / 47 usd per month (albeit usd is relatively weak right now): CPU: Intel® Core™ i7-6700 Quad-Core RAM: 64 GB DDR4 Drives: 2x 512 GB NVMe SSD

Building an equivalent computer would cost me around 1300 eur / 1540 usd without VAT (assuming the RAM is ECC).

Let's assume a 12.5 eur / 15 usd of electricity per month and the computer will pay itself in 2 years. It's hard to price maintenance, issues and hardware failures, but I feel like this computer could last 4-5 years, making the price roughly 2x.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#86

Post by jokethrowaway » Wed, Jun 30, 2021, 8:33 PM UTC

908B64B197 wrote: post #66

WYepQ4dNnG wrote: post #42

I have experienced first hand several cases of k8s gone wrong. In the end I have come to the conclusion that most companies don't really need the complexity of k8s. Seriously, most k8s projects I have been involved with required so much effort to bootstrap and keep it going, it just blew me away! The experience for the average developer was just frustrating and infuriating: AWS ECS to the rescue! Some will argue: ven…

> I have seen companies failing because investing so much in building infrastructure, supposly vendor lock in free (or so they thought) that they lost sight and did not invest enough building the actual product: no revenue -> party is over. How was the stock comp for engineers at these companies?

I was the tech lead / main engineer at a startup with a huge investor investing 1mln in a seed round. I refused a salary with a little bit of equity (single digit percentage) and kept contracting because I didn't trust the business side and apparently I wasn't cool enough to become an exec.

Everything was fine at the beginning, we built very fast a CRUD app in node.js hosted on K8s on Google Cloud, doing something useful for our target market. We demoed with the client and everything was fine.

Then they hired a tech lead from another failed startup as a CTO. He started bringing in all his friends (all from the same country, so the company started having a french crowd and everyone else) from his previous employer as minor exec titles. Stock compensation for the exec was pretty good and they even had a 50k bonus or so I heard.

Slowly but surely they started pushing for our backend to be rewritten in Go for no reason - which I resisted - and then we decided to move to AWS + nomad. This was 2015 and tooling was even more minimal than what Hashicorp offers nowadays - so they basically started building a K8S on top of nomad to replicate the capabilities we had before.

I complained saying that we didn't have time for this with the impending demos and I said it was a stupid decision.

I was promptly fired for that - and without me on the way they threw away our node.js application and started rewriting everything in Go.

They never made another demo in time and the investor eventually pulled the plug from investing in the startup and acqui-hired the company for little money. The founders were pissed and called me later on to update me on how it went and to apologise. Fin.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#87

Post by frompdx » Thu, Jul 01, 2021, 9:40 PM UTC

alongub wrote: post #82

frompdx wrote: post #61

Earlier quoted context omitted.

I think this is a great idea. I can see the DSL being very useful for situations where you want to think about hypotheticals. My main piece of feedback is that you should consider supporting the ability to feed in your existing k8s manifests as input to this tool instead in addition to the DSL. I think that would make this tool very appealing and very easy to onboard users with existing clusters who are looking to re…

For sure, this is definitely planned!

Awesome, I'm definitely keeping an eye on this project.

Re: Kubernetes a black hole of unpredictable spend, according to new report

#88

Post by gyoza » Fri, Jul 02, 2021, 4:06 PM UTC

rantwasp wrote: post #69

flowerlad wrote: post #53

Earlier quoted context omitted.

> most companies don't really need the complexity of k8s If you want to run a complex service consisting of multiple microservices, auto scaling and so on, nothing beats Kubernetes. But you're right, most small businesses just need a simple web site, and for them, an Amazon Lightsail VM might suffice.

yeah, nothing beats K8s except you know... maybe serverless. or maybe cloud primitives that are already there and are handled by someone else. IMHO, unless you have a huge fleet of bare metal machines K8s is an overhead you don't need

i run k8s on my home network, its nice to not have to log into my linsux box to see logs and restart things.

I like having the ability to just nuke my thing at home and it comes up the same way it did before,... Not sure why it too me so long to adopt work mentality at home but its great. Plex acting funny? kill the pod. Sonar needs an update? Kill the pod.