I remember trying out docker sometime back in late 2013. Something about it never fully stuck with me. I always felt like the final boundary for a piece software should be the process, not the computer. Plopping an entire VM into a zipfile and saying "here's your software" felt like lazy engineering to many of us at the time (and still today). For our current stack, the answer has been to make the entire business app…
Google admits Kubernetes container tech is too complex
421–430 of 449 posts
Re: Google admits Kubernetes container tech is too complex
#422I work on a 3-person DevOps team that just finished migrating ~20 services from GCE vms running docker-compose to GKE. It's taken us a little over a year. Partly because K8s has a steep learning curve, but also because safely transitioning services without disrupting product teams adds a lot of overhead. The investment is already yielding great returns. Developers are happy. Actual quote: "Kubernetes is the biggest q…
You spent more than 3 person-years migrating twenty services from one container orchestration system to another and you think it was worth it? Not sure that seems worth it to me...
If you're determined to be skeptical of the value-add from Kubernetes, my random post on HN ain't gonna convince you :) The developer experience improvement is obvious to people actually interacting with K8s.
I do want to clarify that this isn't the only thing we accomplished in the past year. Just the thing we shipped that was our biggest priority and had (by far) the biggest impact.
Re: Google admits Kubernetes container tech is too complex
#423Earlier quoted context omitted.
Helm is so bad it raises my blood pressure just hearing the name. Helm tries to apply the old way of doing things (like as you said Puppet) and makes it worse than ever. K8s yaml config is simple and elegant, don't try hiding it under templates. Kustomize is the proper way of working with K8s yaml. Helm fights against it.
Helm isn't great (god, Go templates, shudder), but I love being able to bundle all the manifests for an application together with well-documented parameters/values... If you want do something like conditionally switch from a LoadBalancer service to Ingress in a test environment... I have no idea how you'd handle that in Kustomize, but it's straightforward in Helm. Ultimately it seems like you end up with the same com…
Re: Google admits Kubernetes container tech is too complex
#424Earlier quoted context omitted.
When library versions start to cause issues: like V3.5 having a bug, so you need to roll back to V3.4... that's when ./configure && make starts to have issues. Yeah, it happens with .so files, .dlls ("dll hell"), package managers and more. But that's where things like containers come in to help: "I tested Library Foo version V3.4 and that's what you get in the docker". No issues with Foo V3.5 or V3.6 causing issues..…
There are basically two options; maintain up-to-date dependencies carefully (engineer around dll-hell with lots of automated testing and be well-versed in the changelogs of dependencies) or compile a bunch of CVEs into production software. There really isn't any middle ground (except to not use third-party libraries at all).
That makes sense for a hobbyist community but not so much for production.
In a former job we needed to fork and maintain patches ourselves, keeping an eye on the CVE databases and mailinglists and applying only security patches as needed rather than upgrading versions. We managed to be proactive and avoid 90% of the patches by turning stuff off or ripping it out of the build entirely. For example with openSSH we ripped out PAM, built it without LDAP support, no kerberos support etc. And kept patching it when vulns came out. You'd be amazed at how many vulns don't affect you if you turn off 90% of the functionality and only use what you need.
We needed to do this as we were selling embedded software that had stability requirements and was supported (by us).
It drove people nuts as they would run a Nessus scan and do a version check, then look in a database and conclude our software was vulnerable. To shut up the scanners we changed the banners but still people would do fingerprinting, at which point we started putting messages like X-custom-build into our banners and explained to pentesters that they need to actually pentest to verify vulns rather than fingerprinting and doing vuln db lookups.
Point being, at some point you need to maintain stuff and have stable APIs if you want long lasting code that runs well and addresses known vulns. You don't do that by constantly changing your dependencies, you do it by removing complexity, assigning long terms owners, and spending money to maintain your dependencies.
So either you pay the library vendor to make LTS versions, or you pay in house staff to do that, or you push the risk onto the customer.
Re: Google admits Kubernetes container tech is too complex
#425Earlier quoted context omitted.
Reality is too negative for social media.
El Reg's snark is what makes it fun to speak with as they do cut to the heart of it. GKE Autopilot is still fundamentally Kubernetes and supports the k8s APIs. GKE has simplified a lot of Kubernetes tasks that can be a pain, such as upgrades & scaling, but as many people said on other threads Kubernetes definitely is not for all workloads. For workloads that actually do need to scale then Autopilot removes a lot of t…
Interested in having someone who has had to set up 1000s of containers in k8s and hated all of it?
Re: Google admits Kubernetes container tech is too complex
#426Earlier quoted context omitted.
Completely agree. The whole compile chain for most software and reliance on linked libraries, implicit dependencies like locale settings changing behavior, basically decades of weird accidents and hacks to get around memory and disk size limits, can be a nightmare to deal with. If using slow dynamic languages, or modern frontend bunglers, all the implicit c extension compilations and dependencies can still be a pain.…
I don’t know. I cant take anyone seriously who says it’s hard to type ./configure make At a prompt, then install missing libs. Unless you have to maintain updates regularly, “It’s just so hard” seems like a damn meme.
You lost me there already. Why should there be missing libs, and why would you not have to maintain updates regularly in production environment?
So let me see if I got this right, it's basically:
1. ./configure 2. make 3. ??? 4. profit!
Doesn't sound like a perfectly good solution to me.
Re: Google admits Kubernetes container tech is too complex
#427Earlier quoted context omitted.
Completely agree. The whole compile chain for most software and reliance on linked libraries, implicit dependencies like locale settings changing behavior, basically decades of weird accidents and hacks to get around memory and disk size limits, can be a nightmare to deal with. If using slow dynamic languages, or modern frontend bunglers, all the implicit c extension compilations and dependencies can still be a pain.…
Aren't we conflating compile complexities with runtime complexities here? There are plenty of open-source applications that offer pre-compiled binaries.
Re: Google admits Kubernetes container tech is too complex
#428Earlier quoted context omitted.
Long time ago I worked on HA setups for telecom (Wimax/LTE) equipment. Kubernetes is complicated but has nothing on those systems. Just to give you some idea - https://www.metaswitch.com/hs-fs/hubfs/Blogs/3gpp-ts-23-228-... (doesn’t even cover everything)
The very term "HA" still gives me nightmares. It can be very hard to get HA to work correctly. Many years ago, I worked in a startup and one of our main offerings was an HA network device. It was unbelievably finicky to get it to work in the first place and even harder to update the software on an HA cluster.
Re: Google admits Kubernetes container tech is too complex
#429Earlier quoted context omitted.
I once misconfigured iptables and locked myself out of our buildserver. Had to call lab support in a different country. Is Linux too complicated? Joyent famously took down their whole region by rebooting wrong nodes. It’s almost like running distributed networks of supercomputers at scale is hard or something...
For two systems of equal functionality, the one that allows or encourages fewer footbullets is the better design. Not all complexity is essential.
Re: Google admits Kubernetes container tech is too complex
#430Earlier quoted context omitted.
Netflix in fact 'run on servers' https://papers.freebsd.org/2019/fosdem/looney-netflix_and_fr...
Ofcourse, everything runs on servers. Question is, who owns and maintains the hardware. The link you shared just says they manage their OS layer, ofcourse they do. Everyone running on AWS VMs is responsible for their own OS layer. Wether they want precise control over their OS doesn't change their preference for who owns and manages the hardware..