Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Big DevOps Misunderstanding

wolfoliver.medium.com

Post reply on HN
Print view View original

111–120 of 321 posts

Re: The Big DevOps Misunderstanding

#111

Post by thamer »

StreamBright wrote: post #70
manish_gill wrote: post #54

I had to basically give up my full time role as a Data Engineer who is supposed to solve "big data" problems (Storage, Streaming, Machine Learning pipelines) etc, and had to solely focus on DevOps because there was no other expert in the team. We (a colleague and I) basically learned Terraform, then Kubernetes, and then Helm in an extremely stressful environment. I wouldn't recommend that to most people. (Also, I can…

You are on the right track. I have exactly the same story, data engineer, working on automation a lot. YAML, JSON and HCL should be purged from IaC asap. Luckily Dhall can be integrated everywhere we need YAML otherwise we would go nuts. The bigges failure of k8s is to understand that YAML does not scale to teams and beyond 100 lines, especially not good to nest it (hello ArgoCD). A type safe not accidentally Turing…

Dhall or other typed languages like Cue definitely help a lot. It seems to me that the only acceptable amount of YAML you could have in deployment config is for a setup small enough that it doesn't warrant using Kubernetes, so any use case where Kubernetes is a good fit quickly becomes a major pain to maintain.

That said, you wrote:

> drop k8s like its hot [...] We have phased it out everywhere we could.

Could you elaborate on what you've replaced Kubernetes with, and for what kind of deployment?

Re: The Big DevOps Misunderstanding

#112

Post by lillecarl »

I disagree, DevOps people are writing code and "code" (YAML, HCL, JSON) that's used to provision infrastructure. When developers push to master it'll build and deploy. They're still deploying, they're just getting help with running the stuff. Can we really expect everyone to understand how solid infrastructure should be built? I don't think every JS frontend dev should have this know-how, yet their code is deployed on S3 behind Cloudfront (or whatever in front of whatever, static file hosting feels like kind of a solved problem).

Re: The Big DevOps Misunderstanding

#114

Post by kazen44 »

topochico3 wrote: post #90
lukevp wrote: post #64

Earlier quoted context omitted.

If you haven’t heard about cdktf, check it out! It lets you do infrastructure as code but in full programming languages like Python, C# and TypeScript. It’s by hashicorp, and under the covers it runs Terraform, so you have the whole set of terraform providers at your disposal.

I don’t understand the appeal of CDK, it’s the most difficult to understand overly complex part of IaaC I have ever come across. I want simple yaml/hcl that is easy to understand and write. I don’t need to feel smart writing TF, I need it to be easy to understand to reduce MTTR when things go wrong.

the issue with yaml is that is breaks down under real life complexities in larger/complexer environments.

Re: The Big DevOps Misunderstanding

#115

Post by arielb1 »

The one system I've seen work is when each medium team (of say, 20-36 people) has a "devops team" of say 4-6 people that is responsible for all the infrastructure stuff, while the rest of the people can focus on product work. It's just that keeping track of infrastructure is much more efficient as a full-time job rather than a 10%-time job.

It's important to do the org engineering right so that the infrastructure people and developers feel responsible for each-other and work well together - ideally both teams should be able to open PRs against each other's code if needed and get help any time.

If the org engineering isn't done right, devs write code that pages the ops and the ops don't respond to devs quickly enough for things to work well ("traditional ops").

Re: The Big DevOps Misunderstanding

#116

Post by kazen44 »

hermanradtke wrote: post #73
recursivedoubts wrote: post #18

I'm frankly astounded at the complexity around modern development due to dev ops and how much worse developer experience is because of it. I used to develop in a great IDE with debugging, right click re-run failed tests, I could follow the console right there in a nice, clean window integrated with my IDE, click on stack traces, etc. Now I'm running my app via multiple docker images, trying to get a buggy remote debu…

What you just described is not really related to devops. I agree that a complicated dev environment is bad. However, that isn’t a devops thing. You can have devops with a monolith that is statically compiled into a single binary. I’ve built it and done it. It is quite nice.

we currently do this at our company.

compiled code gets (autodeployed) with SFTP in some cases even.

deployment has become massively complex because of the need to keep production and test environments roughly the same. Deployment to production should not be complex and basically boils down to the following:

- think about method of getting the binaries installed on the system in a secure way. - think about sending some state back that the binary has been installed succesfully.

- add node/member/app/whathaveyou to the production membership pool/loadbalancer/BGP Route reflector/whathaveyou.

Re: The Big DevOps Misunderstanding

#117

Post by manish_gill »

flas9sd wrote: post #107
manish_gill wrote: post #54

I had to basically give up my full time role as a Data Engineer who is supposed to solve "big data" problems (Storage, Streaming, Machine Learning pipelines) etc, and had to solely focus on DevOps because there was no other expert in the team. We (a colleague and I) basically learned Terraform, then Kubernetes, and then Helm in an extremely stressful environment. I wouldn't recommend that to most people. (Also, I can…

could you give me a hint why you didn't like jsonnet? I dreaded k8s yaml when needing to (write) and use helm packages and think of indenting levels, but jsonnet via grafanas tanka gave me the option to be concise, writing functions and creating reusable bits. Of course then you're not far away from using a real language "sdk style". As you so aptly put, k8s learning curve is beginner hostile and needs system knowled…

Yeah, I think the difference is that I was

a) Learning K8s and Helm Concepts (both different things)

b) Reading/Debugging K8s and Helm YAML (both different things)

c) Ending up at Helm Chart Configs (another config)

d) Figuring out how to set all this to helm me deploy the solution I needed and customize it acc to my needs

...all within the span of a few weeks.

So by the time I got to some GitHub repos which were using Jsonnet, I was basically done. I'm sure it has its uses and given enough time and experience, I can see how to utilize it more effectively.

Re: The Big DevOps Misunderstanding

#118

Post by thedougd »

cortesoft wrote: post #17

As someone who has lead a devops team, and still has a devops team reporting to me, I have also struggled with the idea of having a team called devops, since it does go against the devops philosophy of devs owning their software in production. However, in our case (and probably many others), our devops team is responsible for creating and maintaining the tools and processes that allow the dev teams to manage their sy…

Would you say it's a platform team?

Re: The Big DevOps Misunderstanding

#119

Post by kazen44 »

streetcat1 wrote: post #113

I think that the op is missing a side. The article is about dev -> ops. The other side is ops->dev. I.e. since infrastructure is software, ops become dev (and not just dev become ops).

has ops ever not been dev?

Atleast in my experience (ISP/DC world) most operational people where expected to do troubleshooting which at some points, results in one having to program to get to the nitty gritty efficiently. At a certain scale, issues start to creep up that are hard if not impossible to troubleshoot without atleast some programming and CS experience.

Are operational people developers? Not in the traditional sense, but the development of specific tools is required to efficiently keep the lights running.

A good example of this is troubleshooting packet loss. In a large network, this can become very difficult to pinpoint because more then a few dozen hops exist, and packets drop seem to occur randomly.

Without tooling to gather this data from different points in the network and doing statistics on it, it becomes nearly impossible to get a clear picture and troubleshoot efficiently.

Re: The Big DevOps Misunderstanding

#120

Post by manish_gill »

benlivengood wrote: post #93
manish_gill wrote: post #54

I had to basically give up my full time role as a Data Engineer who is supposed to solve "big data" problems (Storage, Streaming, Machine Learning pipelines) etc, and had to solely focus on DevOps because there was no other expert in the team. We (a colleague and I) basically learned Terraform, then Kubernetes, and then Helm in an extremely stressful environment. I wouldn't recommend that to most people. (Also, I can…

I've been in two production environments where IaC was done with Turing-complete languages (Google and Facebook). It's really not better (it just lets the truly impossible/unmanageable become possible) because they were both dynamically typed. If anything, Terraform comes the closest to being a good IaC language because providers form an API to production resources as opposed to unstructured code/config like Helm or…

One can write good and bad code in any language. It just so happens that our daily driver - procedural or functional languages, happen to be ones where a Software Engineer's mental model + Existing tooling is evolved enough to make understanding it _easier_ (as opposed to just easy).

The problem with DSLs and Config languages is that someone will inevitably try to use them in a way they are not designed to be used, and then circumvent the way you're "supposed to" do things. Not only do you lose all the advances in tooling, but people will end up reinventing things that already exist in the other languages.

PS: Of course, I'll pick Static Typing every time.

Post reply on HN