Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What job interviews taught me about Kubernetes

notnotp.com

Post reply on HN
Print view View original

201–210 of 233 posts

Re: What job interviews taught me about Kubernetes

#201

Post by mschuster91 »

JohnMakin wrote: post #190
mschuster91 wrote: post #178

Earlier quoted context omitted.

> Sure, it feels unified, but the cost of unification is incredibly not worth it. That's the cost I was talking about. It is indeed annoying and time-consuming to get it set-up once, but once it works... it is amazing for developers to have the ability to spin up a completely identical to prod environment for a hotfix branch to test stuff out, with no involvement from ops or anyone else. And also, it's much easier IM…

Yea, I used to believe this too, and still sort of agree - I got so tired of the argument in maintaining k8s infra in terraform I gave up and wrote what is essentially a terraform wrapper module around helm. The charts break terraform quite a bit sometimes, so you have to keep it simple, and god help you if you want to use CRD's, hashicorp providers have the notion no one actually needs those. I had dismal hopes of i…

> I think if you run EKS resources like node groups, autoscalers, LB type of resources in the same state file as helm deployments you're going to have a very bad time though.

There's no alternative to that anyway... otherwise even a terraform apply -refresh=false will quickly take well over 10 minutes.

Re: What job interviews taught me about Kubernetes

#202

Post by himata4113 »

englishspot wrote: post #192
makeitdouble wrote: post #57

Earlier quoted context omitted.

I think parent would wish for something close to what heroku represented (what would it be ?) K8s is easier at smaller scales (I understand k3s as a packaged version ?), but you still need one or two people in your team to properly understands all of the concepts and inner workings of k8s, and be able to neck deep into if/when shit hits the fan. For a small team that's a lot of commitment for something that is usuall…

even with just k3s and a few plugins/operators, it still takes someone dedicated to babying it. I've been running a k3s cluster at home for years and I dread upgrading all the things running on it, and all the things needed to keep it running. and more to that last point, we haven't talked about maintaining the actual nodes themselves yet.

if you don't use alpha or beta annotations you rarely have to worry about updates, kubernetes has a very strong "do not break it" policy on non beta/alpha annotations.

Re: What job interviews taught me about Kubernetes

#203

Post by nat8265639392 »

InvertedRhodium wrote: post #125
simoncion wrote: post #58

Earlier quoted context omitted.

> And tfstate is hard to share between developers... Really? For years and years we put our tfstate files into private S3 buckets at $DAYJOB and it seemed to work just fine. We didn't even take pains to ensure that everyone was on the very same version of the Terraform CLI. What problems did you guys run into?

One person upgrading terraform can break the state format for backwards compatibility. We use direnv with asdf to ensure everyone is in the same version now.

Run the apply via a pipeline. Pr merged in. Plan runs. Approval gate for apply.

Re: What job interviews taught me about Kubernetes

#204

Post by mikeocool »

himata4113 wrote: post #43
mikeocool wrote: post #14

I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article. The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so. But k8s is such a p…

I don't know... running a startup sized kubernetes is relatively easy and pain free these days (k3s). Especially when it comes to scaling up. CNPG is an absolute monster (in a good way). cert-manager is easier than the docker alternative, calico has never failed me (except in bgp mode which has some footguns like not being able to come back from a dead state since it has a chicken and an egg problem unless you point…

I’d take a cronjob running cert-bot and some monitoring to ensure a domain’s cert isn’t about to expire over cert-manager any day.

IIRC cert-manager has about three layers of custom resources to conv through when figuring out why a cert isn’t renewing.

Re: What job interviews taught me about Kubernetes

#205

Post by mikeocool »

vbezhenar wrote: post #141
mikeocool wrote: post #14

I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article. The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so. But k8s is such a p…

Docker swarm is that simple solution you're looking for. But people don't need simple solutions. They want scalable solutions and Kubernetes fits this niche perfectly. You can deploy it on single server today and scale to 100 servers managed cluster tomorrow. Just to provide a similar example. Linux system is insanely complicated. Kernel alone has thousands of options. Distos have tens of thousands of packages. Where…

People think they want infinitely scalable solutions. But I think what they fail realize is that by the time they actually hit a scale where it matter they’re going to have to change so much about their infra that prematurely using the scalable solution didn’t really but them much except a bunch of headaches when they didn’t have scale.

Re: What job interviews taught me about Kubernetes

#206

Post by mikeocool »

vbezhenar wrote: post #147
singron wrote: post #99

Earlier quoted context omitted.

I really wish there was an 80% kubernetes. I think you could get there with some changes: 1. No overlay networks. 1 IP per machine. pods use dynamically allocated ports, and the kubelet enforces pods listen only on their assigned ports using seccomp. 2. No kube-proxy or equivalent Layer-4 "load-balancer". It's not good, but it's often used. You should use some kind of Layer-7 load balancing instead. Also you need to…

1. You can't force third-party software to do that. There are programs with hardcored ports. There are programs which require XML modifications and container rebuilds to change port number. If your platform does not support launching of unmodified containers, it is severely restricted and not suitable for general use. All my programs always use port 8080 for HTTP, I don't make it configurable because I have no reason…

Even with all its complexity, k8s doesn’t solve every problem — good luck running an FTP server or anything that needs to dynamically allocate a large range ports on k8s.

I would absolutely trade flexibility for complexity. Particularly for edge cases like hard coded ports.

Re: What job interviews taught me about Kubernetes

#207

Post by wlonkly »

linuxftw wrote: post #165
singron wrote: post #99

Earlier quoted context omitted.

I really wish there was an 80% kubernetes. I think you could get there with some changes: 1. No overlay networks. 1 IP per machine. pods use dynamically allocated ports, and the kubelet enforces pods listen only on their assigned ports using seccomp. 2. No kube-proxy or equivalent Layer-4 "load-balancer". It's not good, but it's often used. You should use some kind of Layer-7 load balancing instead. Also you need to…

If these ideas served some useful purpose, they would already be implemented in kubernetes. The platform is quite extensible.

This reminds me of the joke about the economists who spot a $100 bill on the sidewalk.

Re: What job interviews taught me about Kubernetes

#208

Post by wlonkly »

vbezhenar wrote: post #141
mikeocool wrote: post #14

I made this decision at a startup (albeit when the eng team was ~30 people, and we had a monolith with ~10 supporting services). I wouldn’t do it again, even for the reasons stated in the article. The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so. But k8s is such a p…

Docker swarm is that simple solution you're looking for. But people don't need simple solutions. They want scalable solutions and Kubernetes fits this niche perfectly. You can deploy it on single server today and scale to 100 servers managed cluster tomorrow. Just to provide a similar example. Linux system is insanely complicated. Kernel alone has thousands of options. Distos have tens of thousands of packages. Where…

Kubernetes zeitgest aside, the naming confusion between the old and new things called Docker Swarm doomed the new one.

Re: What job interviews taught me about Kubernetes

#209

Post by mikeocool »

himata4113 wrote: post #202
englishspot wrote: post #192

Earlier quoted context omitted.

even with just k3s and a few plugins/operators, it still takes someone dedicated to babying it. I've been running a k3s cluster at home for years and I dread upgrading all the things running on it, and all the things needed to keep it running. and more to that last point, we haven't talked about maintaining the actual nodes themselves yet.

if you don't use alpha or beta annotations you rarely have to worry about updates, kubernetes has a very strong "do not break it" policy on non beta/alpha annotations.

Not using beta ingress was probably a non-starter for a lot of people, since it was the only option for 4 years.

Then there was an upgrade process that required a fair amount coordination between when you changed your manifests, when you upgraded your cluster and when you upgraded your ingress controller.

PodSecurityPolicies also gained a lot of traction and didn’t really have an alternative before it was deprecated.

Also, custom operators don’t all subscribe to the don’t break non-beta resources in the same way core does.

Re: What job interviews taught me about Kubernetes

#210

Post by corpoposter »

sshine wrote: post #27
SamuelAdams wrote: post #16

This seems to be less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated across the board. It is nice to be able to have a consistent deployment pattern, with traceability, rollback support, and production approval checks. It’s nice to not have some archaic something stuck in someone’s head. It’s also nice to be able to se…

> less about K8’s and more about the infra as code movement. It doesn’t matter if you use K8, CDK, or terraform - you get the same benefits the OP stated I’d like to gently push back on that. ;-D Terraform, when committed to git, provides organisational memory. But less so uniformity, since all providers are different (and you should expect different things when applying). No tracing besides git. And tfstate is hard…

IaC (capturing desired state declaratively) is a no-brainer for things like cloud resources IMO. Most use IaC for non-compute resources (e.x. DBs) and K8s for containerized compute workloads.

> But less so uniformity, since all providers are different

People sometimes misinterpret tools like Terraform supporting different vendors/hyperscalers as it providing a unified abstraction layer above them. As you note, it does not.

I simply fail to understand why automatic drift correction is considered important in this space. Cloud resources do not magically change themselves. Folks often cite rogue engineers making changes, but I prefer to deal with this scenario by whacking people with a stick and/or limiting access. Automatic drift correction can actually complicate making legitimate emergency changes to managed infrastructure.

Post reply on HN