51–60 of 86 posts
Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.
That was compounded by the metadata requests using DNS and the metadata IP, and until recently Kubernetes didn't have any built-in local DNS cache[3] (GKE still doesn't), which in turn overloaded kube-dns, making other DNS requests fail.
We worked around the issues by disabling metadata concealment, and added metadata to /etc/hosts using pod hostAliases:
hostAliases:
- ip: "169.254.169.254"
hostnames:
- "metadata.google.internal"
- "metadata"
[2] https://github.com/GoogleCloudPlatform/k8s-metadata-proxy
[3] https://kubernetes.io/docs/tasks/administer-cluster/nodeloca...
"Once this change was applied, requests started being served without involving the AWS Metadata service and returned to an even lower latency than in EC2." Title should be: My configuration made my latency 10x higher.
We solved our configuration problems, and most rewardingly, we managed better performance than the original EC2 baseline.
Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.
"Once this change was applied, requests started being served without involving the AWS Metadata service and returned to an even lower latency than in EC2." Title should be: My configuration made my latency 10x higher.
The problem wasn't in Kubernetes at all! The problem was in KIAM and the AWS Java SDK. It would be more accurate to criticize AWS's Kubernetes support. Both KIAM and the AWS Java SDK are specific to AWS.
> Kubernetes made my latency 10x higher The title is a bit misleading, kub didn't cause the 10x latency - also latency was lower after they fixed their issues TL;DR version - Migrate from EC2 to Kub; due to some default settings in Kiam & AWS Java SDK, latency of application increased, fixed after reconfiguration and kub latency lower than EC2
It is relevant though as k8s makes everything more complicated so you have to deal with stuff like this. Also if it was a brand new app theyd maybe not notice the problem in the first place.
More complicated than what? Without a baseline for the comparison it's not that useful. In our case we transitioned over the last four years from running hundreds of VMs provisioned with puppet and jenkins to running K8S workloads (on a lot fewer nodes than we had VMs) provisioned with helm/kustomize and using gitlab ci/cd pipelines. In my opinion the current platform is much less complex to understand and manage than the old one was. Yeah there are layers of complexity there that didn't exist in the previous platform, i.e the k8s control plane, the whole service/pod indirection, new kinds of code resources to manage, but it's all pretty consistent and works logically, and isn't really any harder to internalize than any other platform-specific complexity we've had to deal with in the past. And in terms of day-to-day development, deployment and operations k8s has been completely transformative for us.
Earlier quoted context omitted.
In my opinion DevOps shouldn't exist at all.
So should developers be doing all their own ops work in a half-assed, ignorant way, or should we go back to a world of throw it over the wall systems where the ops team doesn't understand the code or have a working non-adversarial relationship with the developers? Because I've lived with both, and to hell with both of them.
It's a delicate balance to strike, especially considering having a product with a ton of features but lackluster stability doesn't get you anywhere, and neither does having a super stable product that lacks basic features.
"move fast and break things" doesn't work in a lot of sectors. It might work for an internet startup, but good luck applying that principle to fields like large infrastructures and datacenters.
Earlier quoted context omitted.
Okay. So based on 20+ years of experience, I can say that most developers have no interest in automating deployment, configuration, monitoring, performance, logging, etc... Who should do this work?
Yeah, operations-focused engineers will continue to have a niche carved out for them because too many devs black-box infrastructure. Companies can either choose to have their devs take on ops responsibilities or continue having dedicated ops jobs. In either case, whether or not dedicated ops jobs exist, ops responsibilities always will. I'll be there to pick up the slack because designing and maintaining systems is a…
1) Developers don't code on prod
2) Prod needs to be protected
3) Therefore, developers need to be restricted from prod
Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.
I don't know. k8s is pretty complicated. How many small/medium apps need more than this nginx/Terraform/Docker example? This would be a lot more difficult to set up in k8s (pods, ingress, etc.)
Terraform:
resource "docker_container" "app" { count = 2
name = "app-${count.index}"
hostname = "app-${count.index}"
image = "app:${var.app_version}"
restart = "always"
env = [
...
]
ports {
internal = parseint("${var.app_port}", 10) + count.index
external = parseint("${var.app_port}", 10) + count.index
}
networks_advanced {
name = "${docker_network.private_network.name}"
aliases = ["app-${count.index}"]
}
depends_on = [
...
]Nginx:
http { ...
upstream app {
server app-0:3000;
server app-1:3001;
}
server {
listen 80;
root /usr/share/nginx/html;
location ~* ^/api/ {
rewrite ^/api/(.*) /$1 break;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_pass http://app;
}
}
}