Overall it is a great tool and I completely do not get the hate for it, but I did have some issues with AWS EKS - we made two mistakes in our project by using k8s API instead of DNS for discovery and environmental variables instead of config maps and this ended up overloading master nodes, which started throttling sporadically, eapecially with load spikes. It seemed like AWS EKS support team were really puzzled by this and it took us weeks to get to the root cause, even with their support. This might be considered as more of an AWS issue than k8s one.
Ask HN: What is your Kubernetes nightmare?
201–210 of 263 posts
Re: Ask HN: What is your Kubernetes nightmare?
#202Earlier quoted context omitted.
We use GKE. 1. I don't know about any of this; we don't seem to have problems. 2. This sounds like an architecture issue, not a k8s issue. 3. Our entire GKE infrastructure costs less than $50 a month. 4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases. 5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we di…
Agreed. We've had only one autoscaling issue and it was on Google's end (datacenter ran out of nodes of a particular type and thus failed to scale up). Our GKE infrastructure costs a lot more, but we do a lot of heavy compute.
Most of our 'heavy compute' happens in BigQuery so we pay for it there, instead, but we make sure the rest of our analytics infrastructure needs are engineered to be as light as possible to keep costs low.
We are an ESOP so, as employee owners, it behooves us to be as cost-conscious as possible.
Re: Ask HN: What is your Kubernetes nightmare?
#203Earlier quoted context omitted.
> Kubernetes on bare metal is actually pretty easy. I would not call it easy at all. Last time I tried that a year ago you still needed a special load balancer to get it going ( https://metallb.universe.tf ). Has this changed?
MetalLB is pretty simply to configure.
That's just not true, especially if you compare it to the LoadBalancer you get on a cloud platform which usually involves zero clicks. I'm not saying it's impossible but it's definitely not "easy".
Configuration instructions: https://metallb.universe.tf/configuration/
Hint: You better know what all of these are in your environment:
For a basic configuration featuring one BGP router and one IP address range, you need 4 pieces of information:
The router IP address that MetalLB should connect to,
The router’s AS number,
The AS number MetalLB should use,
An IP address range expressed as a CIDR prefix.Re: Ask HN: What is your Kubernetes nightmare?
#204It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…
Thank you for your insight. Really solid advice.
I have a question thought.
> My personal Kubernetes nightmare is having to build a cluster from scratch on bare metal.
Can you share a few details about which distribution you used and how did you handled ingress?
Re: Ask HN: What is your Kubernetes nightmare?
#205Earlier quoted context omitted.
And don't expose workloads to the internet unless it is a prod app. Can you recommend me a good intro book to read cover to cover (hopefully not too thick).
"Kubernetes in Action" by Marko (Manning publishing) is my recommendation. Took me from someone who knows docker/docker-compose to someone who can handle Azure/AWS Kubernetes, understand the terms and design apps. Very good book.
Thanks
Re: Ask HN: What is your Kubernetes nightmare?
#206More distributed system than Kubernetes, but quite fun : We deployed a MongoDB cluster on our Kubernetes Clusters. Our application was a having a chat feature that stored the messages into the MongoDB cluster. After some months, we realized that we got some weird issues, some messages was arriving in the wrong orders, like : 1. A : Hi ! 2. B : Bye ! See you next time ! 3. A : Great and you ? 4. B : Hello ! How are yo…
> due to a rule in an random firewall
The eternal practice of middleboxing your network. This didn't work well at the time LANs were completely isolated, break much better nowadays when LANs are just a convention over WANs, and fails for virtual LANs on a single host too.
Yet people just do it, every single time. Because setting the security in a single place is expected to be easier than setting at the endpoints (I blame Windows for that culture). What is kinda understandable, but here we are, talking about Kubernets, and having that same culture.
Re: Ask HN: What is your Kubernetes nightmare?
#207It's odd, but I actually really enjoy using Kubernetes in production. We have a few rules: 1. Read a good intro book cover-to-cover before trying to understand it. 2. Pay a cloud vendor to supply a working, managed Kubernetes cluster. 3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters. 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possi…
>My personal Kubernetes nightmare is having to build a cluster from scratch on bare metal. One's heaven is another one's nightmare, i like building it from scratch because then i know every single knob, and doing a excellent job in documentation makes sure that others have that knowledge too. But hey since "administrators" is a forgotten art, you are probably better of just buying some black-boxes with terrible perfo…
Depending on your individual circumstances buying "some black-boxes with terrible performance" might be a worthwhile tradeoff
Re: Ask HN: What is your Kubernetes nightmare?
#208Earlier quoted context omitted.
MetalLB is pretty simply to configure.
That's just not true, especially if you compare it to the LoadBalancer you get on a cloud platform which usually involves zero clicks. I'm not saying it's impossible but it's definitely not "easy". Configuration instructions: https://metallb.universe.tf/configuration/ Hint: You better know what all of these are in your environment: For a basic configuration featuring one BGP router and one IP address range, you need…
Did you miss the part about layer 2 configuration, where you don't need BGP at all? https://metallb.universe.tf/configuration/#layer-2-configura...
Re: Ask HN: What is your Kubernetes nightmare?
#209Having to use it. Same for Docker.
Re: Ask HN: What is your Kubernetes nightmare?
#210Slow performance. So we have a few Spring Boot based webapps which were running (along with PgSQL) on a shared AWS t2.medium instance, we migrated these to a GKE cluster with a node pool of e2-standard-2 instances. The nodes are on a private network and don't have public IPs. The services are exposed via Load Balancer based Ingress (with SSL). Even after allocating one core to PgSQL and 2GB RAM, the API calls from th…
Are you setting cpu requests in your pod spec? This influences the cgroup cpu.shares for the containers and (unless this has been fixed) leaving cpu requests unset results in cpu.shares=2, which jvm interprets badly.
Thanks for replying. Yes we are setting the CPU requests in our pod spec. I will experiment with this further and see if solves the issue.