I learned not to use it and wait until the k8s fad goes away.
Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…
What we learned after a year on Kubernetes
151–160 of 162 posts
Re: What we learned after a year on Kubernetes
#152Earlier quoted context omitted.
It's so worth it, though. You may not realize it, but your Healthchecks don't work unless you are using target-type IP. In the standard mode, AWS healthchecks each kubernetes node, but not your pods themselves. It will keep sending traffic to pods that are unhealthy (such as in state: terminating) if it already has connections open through kube-proxy to them, and stop sending traffic to healthy pods if the connection…
> It's so worth it, though. It was until it caused a production outage, then it became very not worth it.
Re: What we learned after a year on Kubernetes
#153Earlier quoted context omitted.
It must not fit your use case and that is ok. Kubernetes is made for the people that have to run your systems and keep them operational (ops/devops). It makes our lives a billion times easier on two fronts. Disclaimer, I am referring to a managed solution like GKE, I can imagine running k8s on your own hardware would be harder. Uptime & Stability: autoscaling, health checks, replacing nodes, restarting containers. Th…
I'd like to hear more on troubleshooting with a managed k8s platform and how you diagnose such issues. My organization is playing with EKS and while perhaps it is different from GKE I feel like I have minimal visibility into if the control plane is healthy. I've had several instances where all of my node groups have become unhealthy for various reasons and EKS failed to spin up new nodes to resolve the issue.
Observability starts with logging and metrics. K8S already puts out lots of events on what's happening which you can use to debug, and there are many 3rd-party monitoring solutions if you need more detail.
If you don't have this with EKS then use something like LogDNA and NetData to see the logs and metrics yourself.
Re: What we learned after a year on Kubernetes
#154Earlier quoted context omitted.
I dunno, because of writeups like GitLabs? We're lucky GitLab is so transparent. It lets us see that their claims don't appear to always match the actual reality. Here's an example. Blog post says: "After transitioning each service, we enjoyed many benefits of using Kubernetes in production, including much faster and safer deploys of the application, scaling, and more efficient resource allocation" Wrong. Actual anal…
Its really weird that your argument is based on just the cost. I mean you mentioned yourself you worked at google right? Perhaps you just haven't actually experienced the issues kubernetes is solving? How often have you seen that certificates expired? I have seen that. Plenty of times. Its not an issue creating that lets encrypt cronjob, its still something you need to do right. Security Updates? Have you seen how ma…
Borg is/was great for running huge numbers of services at truly massive scale, when those services were developed entirely in house and done in the exact way it wanted services to be done. It was a terrible cost the moment you wanted to run anything third party or which wasn't written in that exact Google way, and the costs were especially high if you didn't need to handle huge traffic or data sizes.
Borg and Kubernetes don't auto-magically do sysadmin work. A ton of people work in infrastructure at Google. Software updates still need to be applied etc. In Kubernetes that means rebuilding Docker images (which in reality doesn't happen as the tools for this are poor, so you just have a lot of downlevel Ubuntu images floating around).
And worse, the whole K8s/Docker paradigm is totally backwards incompatible. At Google this didn't matter because the software stack evolved in parallel with the cluster management. Programs there expected local disk to be volatile, expected to be killed at a moment's notice, expected datacenters to appear and disappear like moles. They were written that way from scratch. But that came with a terrible price: it was basically impossible to use ordinary open source apps. You could import libraries and (carefully!) incorporate them into Borg-ized projects, but that was about it.
When this tech was reimplemented and thrown over the wall to the community, the cultural expectations that came with it didn't come with it. So I've seen situations like, "whoops, we deleted our companies private key because it was stored to local disk and then the Docker container was shut down, help!". This was even reported as a bug in the software! No, the bug is that your computer is randomly deleting critical files for no good reason, and normal software does not expect that to happen. How about the way in a Dockerfile you have to write 'apt-get update && apt-get upgrade' if you want a secure OS image when the container is rebuilt? If you put the two commands on separate lines it will appear to be working, right up until you start getting weird errors about missing files from Debian mirrors, because Docker assumes every single command you run is a pure function! And then we get to security.
Screwups seem to follow Kubernetes/Docker around like flies. The tech is complex and violates basic expectations programs have about how POSIX works. When it goes wrong, it leads to mistakes in production.
Now there have been a few trends over time:
1. Hardware has got a lot more powerful. It has been outpacing growth in the internet and economy. Many more businesses fit in a smaller number of machines than when Borg was designed (an era where 4-core systems were considered high end).
2. Cheap colo providers have been driving the cost of powerful VMs down to the ground.
3. Linux has got easier to administer.
These days setting up a bunch of Linux VMs that self-upgrade, run some services via systemd etc isn't difficult, and properly tuned such a setup should be able to serve a monster amount of traffic (watch out though for Azure, which seems to overcommit capacity pretty drastically and their VMs have very unstable performance).
Most businesses that are deploying Kubernetes today quite simply do not need a million machines. Even companies that give away complex services like GitLab, as we can see from this thread, they don't really need huge scalability. It's just nice to imagine that the business will experience explosive growth and that growth is now automated, but ironically, the effort to automate business infrastructure scaling takes away from the sort of efforts that actually grow the business.
As for the other benefits, you can write a systemd unit that is much simpler than Kubernetes configuration that will give you auto-restart, including if memory limits are hit, you can view the activity of a cluster easily using plain old SSH or something like Cockpit, it will handle log rotation for you out of the box, sandboxing likewise, and so on. And of course the unattended-upgrades package has existed for a while.
I agree you need someone to do admin work, whatever path you choose. Having used Kubernetes, and the system it is based on, and plain old Linux, my intuition is that the base cost of Kubernetes is too high for almost all its users. Too many ways to screw it up, too many ways for it to go wrong, too much time spent screwing around, and too expensive. If you become another Google or Facebook then sure, go for it. Otherwise, better avoided.
Re: What we learned after a year on Kubernetes
#155Earlier quoted context omitted.
Its really weird that your argument is based on just the cost. I mean you mentioned yourself you worked at google right? Perhaps you just haven't actually experienced the issues kubernetes is solving? How often have you seen that certificates expired? I have seen that. Plenty of times. Its not an issue creating that lets encrypt cronjob, its still something you need to do right. Security Updates? Have you seen how ma…
I've been running my own Linux servers for about twenty years, so I'm not entirely unexperienced with these things. Borg is/was great for running huge numbers of services at truly massive scale, when those services were developed entirely in house and done in the exact way it wanted services to be done. It was a terrible cost the moment you wanted to run anything third party or which wasn't written in that exact Goog…
But you reach quickly enough an size where a central managed kubernetes cluster is very versatile for your whole company.
1-2 People take care of the cluster, the other teams then use that kubernetes cluster for your build, hosting of staging envs etc.
And then you have a very small infra team which is much better able to provide those services to others internally much easier and safer.
Re: What we learned after a year on Kubernetes
#156Earlier quoted context omitted.
So you are saying that Ansible, Chef and puppet are fads? That solve no real problems? OpenMosix, Bewolf and Grid computing were mostly confined on research circles. I never saw them took off like K8s did. So comparing K8s with OpenMosix is a bit unfair...
I am saying that apparently they are not good enough for the Kubernetes generation. Eventually there will come the post-Kubernetes generation and the cycle does yet another turn.
Do you now think progress is bad?
Ansible and co did and still do a great job.
kubernetes is not just a thing, its also a paradigma shift.
It sounds like something new is just bad because its new?
Re: What we learned after a year on Kubernetes
#157Earlier quoted context omitted.
Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people wi…
Technologies don't last long. Another new tech comes out that does the job better or more cheaply, and the old tech either dies or competes. I think that K8s is much too complicated for medium-scale workloads, so as soon as something makes it easier than K8s to scale and manage workloads at that scale, K8s will be out of that space.
It doesn't make much sense to migrate to k8s if you don't have an issue. And it doesn't make sense to wait for the next thing to happen when you have an issue you need to fix now.
Re: What we learned after a year on Kubernetes
#158Earlier quoted context omitted.
The features i mentioned, are solved 'out of the box'. I'm not seeing a huge stretch here at all. Its costly and risky to run 100 VMs, maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed. It is cost ineffective to have security audits on 100 VMs, maintaining access to them, auditing whats happening on them. It is a ton easier to allow someone only access to one nam…
> Its costly and risky to run 100 VMs... A tiny fraction of companies need to run 100+ VMs. It takes a lot of traffic to require that scale, unless you're just throwing money at infra to avoid optimizing your code (which can be cost effective). A large chunk of companies should just be on something like SquareSpace or Shopify, another large chunk should be using off-the-shelf services like Azure's App Containers or A…
I did not say that everyone should migrate to kubernetes as far as i rmemeber but kubernetes to me is not a fad and it fixes real issues companies have.
Re: What we learned after a year on Kubernetes
#159Earlier quoted context omitted.
Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug. Kustomize, too, doesn't get at the root of all the needs I've found myself having. A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommen…
For internal k8s config at our org we built an config DSL using Starlark. The golang Starlark interpreter is super easy to use and extend. Starlark is familiar to every developer in our org because we are a Python shop. The tooling then spits out k8s YAML. Essentially the config language implementation would be the same logic that a helm chart would do, but you’re writing the logic in Go versus a text templating engi…
Re: What we learned after a year on Kubernetes
#160Earlier quoted context omitted.
I am saying that apparently they are not good enough for the Kubernetes generation. Eventually there will come the post-Kubernetes generation and the cycle does yet another turn.
'Not good enough'? Do you now think progress is bad? Ansible and co did and still do a great job. kubernetes is not just a thing, its also a paradigma shift. It sounds like something new is just bad because its new?
My first containers were called HP-UX Vaults in 1999.
The cycle continues its reboot.