Kubernetes works great for stateless workloads. For anything stateful, monolithic, or that doesn't require autoscaling, I find LXC more appropriate: - it can be clusterized (LXD/Incus), like K8S but unlike Compose - it exposes some tooling to the data plane, especially a load balancer, like K8S - it offers system instances with a complete distribution and a init system, like a VM but unlike a Docker container - it ca…
We're Leaving Kubernetes
121–130 of 348 posts
Re: We're Leaving Kubernetes
#122Re: We're Leaving Kubernetes
#123Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…
> Personally - just let the developer own the machine they use for development. I wonder if Microsoft's approach for Dev Box is the right one.
Re: We're Leaving Kubernetes
#124Sounds more to me like they need a new CTO. And that they're desperate to tell customers that they've fixed their problems. Kubernetes is absolutely the wrong tool for this use case, and I argue that this should be obvious to someone in a CTO-level position, or their immediate advisors. Kubernetes excels as a microservices platform, running reasonably trustworthy workloads. The key features of Kubernetes are rollout…
I agree on the cloud thing. Don't agree that "high performance disks, ballooning memory, live migrations, and isolated workloads" preclude from using k8s - you can still run it as base layer. You get some central configuration storage, machine management and some other niceties for free and you can push your VM-specific features into your application pod. In fact, that's how Google Cloud is designed (except they use…
Are you aware of the limits? It must run as root and privileged?
Re: We're Leaving Kubernetes
#125Sounds more to me like they need a new CTO. And that they're desperate to tell customers that they've fixed their problems. Kubernetes is absolutely the wrong tool for this use case, and I argue that this should be obvious to someone in a CTO-level position, or their immediate advisors. Kubernetes excels as a microservices platform, running reasonably trustworthy workloads. The key features of Kubernetes are rollout…
Why would you say that performance is bad on public cloud infrastructure?
Example: What performance do you get out of your NVMe disks? Because these days you can build storage that delivers 100-200 GB/s.
https://www.graidtech.com/wp-content/uploads/2023/04/Results...
I bet few public cloud customers are seeing that kind of performance.
Re: We're Leaving Kubernetes
#126> SSD RAID 0 > A simpler version of this setup is to use a single SSD attached to the node. This approach provides lower IOPS and bandwidth, and still binds the data to individual nodes. Are you sure SSD is that slow? NVMe devices are so fast that I hardly believe there's any need for RAID 0.
Re: We're Leaving Kubernetes
#127Earlier quoted context omitted.
In my last role as a director of engineering at a startup, I found that a project `flake.nix` file (coupled with simply asking people to use https://determinate.systems/posts/determinate-nix-installer/ to install Nix) led to the fastest "new-hire-to-able-to-contribute" time of anything I've seen. Unfortunately, after a few hires (hand-picked by me), this is what happened: 1) People didn't want to learn Nix, neither d…
I think if you take about 80% of your comment and replace "Nix" with "Haskell/Lisp" and a few other techs, you'd basically have the same thing. Especially point #1.
Worse is better, sadly.
Re: We're Leaving Kubernetes
#128A heterogeneous architecture with multi-tenancy poses some unique challenges because, as mentioned in the article, you get highly inconsistent usage patterns across different services. Also, arbitrary code execution (with sandboxing) can present a signifiant challenge. For security, you ideally need full isolation between services which belong to different users; this isolation wasn't a primary design goal of Kubernetes.
That said, you can probably still use K8s, but in a different way. For smaller customers, you could co-locate on the same cluster, but for larger customers which have high scalability requirements, you could have a separate K8s cluster for each one. Surely for such customers, it's worth the extra effort.
So in conclusion, I don't think the problems which were identified necessarily warrant abandoning K8s entirely, but maybe just a rethinking of how K8s is used. K8s still provides a lot of value in treating a whole cluster of computers as a single machine, especially if all your architecture is already set up for it. In addition to scheduling/orchestration, K8s offers a lot of very nice-to-have features like performance monitoring, dashboards, aggregated logs, ingress, health checks, ...
Re: We're Leaving Kubernetes
#129Earlier quoted context omitted.
I agree on the cloud thing. Don't agree that "high performance disks, ballooning memory, live migrations, and isolated workloads" preclude from using k8s - you can still run it as base layer. You get some central configuration storage, machine management and some other niceties for free and you can push your VM-specific features into your application pod. In fact, that's how Google Cloud is designed (except they use…
True! I love the idea of using K8s to orchestrate the running of VMs. With graceful shutdown and distributed storage, it makes it even more trivial to semi-live migrate VMs. Are you aware of the limits? It must run as root and privileged?
Re: We're Leaving Kubernetes
#130Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…
Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine, regardless of how big the machine is. And having a different development machine than production leads to completely predictable and unavoidable problems. Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of thei…
Isn't this problem solved by CICD? When the developer is ready to test, they make a commit, and the pipeline deploys the code to a dev/test environment. That's how my teams have been doing it.