Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

news.ycombinator.com

Post reply on HN
Print view View original

401–410 of 433 posts

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#401

Post by janosd »

Kubernetes is insanely complex and modular. Just yesterday I was looking at the source code and the code part I knew was replaced by yet another pluggable system. Instead of consolidating into a well-understoof set of features, Kubernetes is exploding with complexity, so it's almost impossible to "build it yourself" for a production environment.

However, there are plenty of companies that will sell you a system, including varying levels of support. You then, of course, have to hire your own DevOps engineers that will deal with the areas the support doesn't cover, which, given the complexity, is still an awful lot. Or you do everything in-house, which means hiring even more people.

TL;DR DevOps engineers won't be out of the job anytime soon. Same for Kubernetes developers.

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#402

Post by KaiserPro »

arunc wrote: post #378
KaiserPro wrote: post #311

Earlier quoted context omitted.

> Kubernetes is a Google scale solution The problem is that its _not_ a google scale solution. Its something that _looks_ like a google scale solution, but is like a movie set compared to the real thing. for example: https://kubernetes.io/docs/setup/best-practices/cluster-larg... no more than 5k nodes. Its extra ordinarily chatty at that scale, which means it;ll cost on inter-vpc traffic. I also strongly suspect that…

Curious how you ran 36k node cluster. Did it involve elastic scaling, etc? What are the alternatives for k8s?

Home grown: https://www.semanticscholar.org/paper/Robust-large-scale-ren...

off the shelf: https://hradec.com/ebooks/CGI/RMS_1.0/rfm/User_Interface/Alf...

although that was with something like 6-10k nodes because there was an upper limit to how many dispatches alfred could do because it was single threaded, from the early 90s and not really designed to scale that high

https://renderman.pixar.com/tractor is probably what they use now, or https://www.opencue.io/

but any grid engine style dispatcher/manager will do what you want. It'll give you the primitives to manage wildly larger scale than k8s.

These clusters were on real steel, as elastic clusters were horrendously expensive, and the storage was/is nowhere near fast enough.

Nowadays, I'd use AWS batch, or at a push airflow.

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#403

Post by raffraffraff »

I don't think there are "so many devops" jobs. Everywhere I've worked in the last 15 years, the number of people managing everything from hardware up to developer tools and CI/CD was tiny compared with the number of developers. Some start-ups dont bother with those people at all to begin with and regret it later. Then they hire a tiny team after years of neglecting these areas, and then expect the "wings on the plane to be swapped out during flight". Those devops people are casually expected to be experts in process (incident/problem management), cloud infra / infra as code, db config / replication, networking, security (IAM, SSO, network, OS), release / deployment, monitoring / metrics / alerting / tracing (not just deploying them but working with devs to implement observability in their code), dev tooling (code/artifact repos, every brand of CICD pipelines & runners) ... basically anything that isn't software development. They're also expected to be oncall for other teams in many companies.

Many years ago I worked at a large, old tech company where all of these areas had dedicated teams.

PS: how many people at your company "really" know Kubernetes inside out? And if it misbehaves, who do you expect to have the answer?

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#404

Post by aitchnyu »

etruong42 wrote: post #10
jmillikin wrote: post #6

> 1) internal users: mainly developers by providing CI/CD > 2) external users: end users > > Nowadays we call people that do 1) DevOps and people that do > 2) SREs (so one could argue that the role of sys admins just > got more specialized). Both are called sysadmins. SRE is a specialized software engineering role -- you'd hire SREs if you wanted to create something like Kubernetes in-house, or do extensive customiza…

> so there are more niches where hiring someone to babysit a few hundred VMs is profitable This makes a lot of sense. The same thing happened in the past with new technology, such as the electronic spreadsheet: "since 1980, right around the time the electronic spreadsheet came out, 400,000 bookkeeping and accounting clerk jobs have gone away. But 600,000 accounting jobs have been added." Episode 606: Spreadsheets!, M…

ATMs caused a decrease in teller transactions, but led to an explosion of bank branches.

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#405

Post by turkey99 »

jmillikin wrote: post #112
throwaway787544 wrote: post #93

Earlier quoted context omitted.

> but instead that operational load of running in-house software should be borne primarily by the developers of that software Go back and read a few DevOps books and blogs by the founders of it. We will always need separate disciplines for dev and ops, just like we need mechanical engineers and mechanics/racecar drivers. But we need them to work together and communicate better to solve problems better and not throw d…

My second job in large-scale software was at Google, which used the "DevOps model" since before DevOps was named. I have no need to read a blog on it. You want the person who designs the car to know what a car is , and to be able to diagnose basic issues like "the fuel gauge says 'empty' and engine won't start". And there's no analogy to an Indy car driver in software, every distributed system is self-driving. The mo…

Did Google really use the “DevOps model”? Observing from the outside it looked like SRE was their alternative

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#406

Post by raffraffraff »

rahkiin wrote: post #144
nokya wrote: post #76

Earlier quoted context omitted.

Exactly. I am consulting with a startup right now that chose to go everything docker/k8s. The CTO is half-shocked/half-depressed by the complexity of our architecture meetings, although he used to be a banking software architect in his previous assignments. Every question I ask ends up in a long 15 minutes monologue by the guy who architected all of it, even the most simple questions. They are soon launching a mobile…

60 containers sounds like an architecture problem, not a Kubernetes problem. Kubernetes does not stop you from running 1 container in 1 pod receiving ingress and talking to a database.

Yep, that's microservices gone mad

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#407

Post by raffraffraff »

AaronM wrote: post #276
throwaway7865 wrote: post #49

Earlier quoted context omitted.

We’ve moved a small-scale business to Kubernetes and it made our lives much easier. Anywhere I’ve worked business always prioritizes high availability and close to zero downtime. No one sees a random delivered feature. But if a node fails at night - everybody knows it. Clients first of all. We’ve achieved it all almost out of the box with EKS. Setup with Fargate nodes was literally a one-liner of eksctl. Multiple env…

We don't have an issue with that last point, lots of EC2 EKS nodes and they don't fail randomly. Were you using resources and limits correctly? EKS nodes can fall over randomly if you don't reserve resources on the nodes for system processes, and your workloads eat up all the resources. That's probably not well documented either.

Setting limits is important, but it always has been. Kubernetes nodes typically don't have a swap so without setting container limits, some critical process can OOM. With swap enabled, memory grows > pathological swapping ensures => caches get dropped making disk performance suck, and all the while your system is shuffling pages between memory and disk. So of course load hits 50+ and the machine turns into a 'black hole'. I've even seen a single VM do that, and cause so much disk IO that it took out the whole hypervisor (which had a single RAID volume)

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#408

Post by kuschku »

015a wrote: post #373
KaiserPro wrote: post #311

Earlier quoted context omitted.

> Kubernetes is a Google scale solution The problem is that its _not_ a google scale solution. Its something that _looks_ like a google scale solution, but is like a movie set compared to the real thing. for example: https://kubernetes.io/docs/setup/best-practices/cluster-larg... no more than 5k nodes. Its extra ordinarily chatty at that scale, which means it;ll cost on inter-vpc traffic. I also strongly suspect that…

Right; and I don't feel that's a knock against K8s; its just the trade-offs it decided to make. A true Google-scale solution would be far worse to use, you can be sure of that. K8s is a "middle 80%" scale solution. Its not made to run Google (though Google uses it a ton internally). Its also not made for your average four person startup (though, if you've got that experience internally, its not a bad choice; its not…

I'm using k8s even for my personal 2 node cluster. It's just so convenient to be able to use all the automation tools.

I can leave the cluster alone for weeks, and it'll take care of itself, my CI will build new docker containers, tooling will start rolling them out across the cluster, if deployments fail they get rolled back, and I get an email, etc.

At some point I was hands-off with the cluster for 6 months and everything kept itself up to date and running just fine.

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#409

Post by [deleted] »

OJFord wrote: post #271
Sohcahtoa82 wrote: post #239

Earlier quoted context omitted.

As a security engineer, I always cringe when anything involving containers is referred to as "serverless". I always thought that one of the advantages of going serverless was that you didn't have to worry about keeping the underlying operating system up-to-date. No needing to do a monthly "sudo apt update && apt upgrade" or whatever. But containers throw that all away when container images enter the world. Instead of…

Is anyone's goal of 'serverless' that they no longer have to deal with updating the OS? Most would say even a server-ful system (k8s, or whatever) should be considered 'cattle not pets' with immutable nodes replaced as needed anyway. No update, just replace. Just like building a new image and having new pods (or serverless whatevers) pull it.

[deleted]

Re: Ask HN: If Kubernetes is the solution, why are there so many DevOps jobs?

#410

Post by mastazi »

subsection1h wrote: post #362
mastazi wrote: post #337

Earlier quoted context omitted.

In my particular market sector it seems everyone is using some form of cloud functions (SAM, Serverless Framework etc), and migrating away from K8S/containers. Regarding PaaS stuff like Heroku, the only people I know that are still using that are solo hackers.

That's interesting. In the latest Who Is Hiring discussion, there seems to be more than 10 times as many references to Kubernetes compared to serverless. https://news.ycombinator.com/item?id=31582796

Interesting, you're right, I see 3 occurrences for "lambda" and 26 for "kubernetes". It's probably just the paritcular "bubble" I live in.
Post reply on HN