A Eulogy for DevOps • Hacker News Board

Re: A Eulogy for DevOps

#101

Post by Whitespace » Sat, Jun 29, 2024, 3:38 AM UTC

weitendorf wrote: post #89

llama052 wrote: post #62

Earlier quoted context omitted.

Can you clarify what you mean by "leaks implementation details everywhere"? I like to think of kubernetes as a big orchestration platform that you can choose to use what you need. If an ingress and pods work then use that, otherwise extend an throw an operator up for what you need (it likely already exists). Cilium for instance is great for that, so is Istio and the like. They aren't hard you just have to understand…

You shouldn’t have to think about all the implementation details of your deployment target. There shouldn’t be platform engineers or Kubernetes experts. Nobody should be writing YAML, or getting paid to set up Istio. Nobody should have to learn the Kubernetes architecture or know about the kubelet or EBPF. The tools should be simple enough with good defaults that application developers just click a button and have th…

Is it fair to say that you are in favor of Heroku, fly.io, ECS, etc?

Re: A Eulogy for DevOps

#102

Post by osigurdson » Sat, Jun 29, 2024, 4:04 AM UTC

stackskipton wrote: post #98

weitendorf wrote: post #89

Earlier quoted context omitted.

You shouldn’t have to think about all the implementation details of your deployment target. There shouldn’t be platform engineers or Kubernetes experts. Nobody should be writing YAML, or getting paid to set up Istio. Nobody should have to learn the Kubernetes architecture or know about the kubelet or EBPF. The tools should be simple enough with good defaults that application developers just click a button and have th…

Ops here, In a simple Kubernetes environment, you don't have to know networking. However, few environments are simple and abstracting X away becomes extremely difficult job once business requirements collide with abstraction. Obstacle is big most of time because most applications are not easy to run. Most DevOps mostly came around because Devs flinging balls of mud over the wall and landing on the Ops side with a spl…

I think that devs really should have a decent understanding of Kubernetes. It is essentially the operating system for any app that needs more than one computer.

Re: A Eulogy for DevOps

#103

Post by llama052 » Sat, Jun 29, 2024, 4:06 AM UTC

weitendorf wrote: post #89

llama052 wrote: post #62

Earlier quoted context omitted.

Can you clarify what you mean by "leaks implementation details everywhere"? I like to think of kubernetes as a big orchestration platform that you can choose to use what you need. If an ingress and pods work then use that, otherwise extend an throw an operator up for what you need (it likely already exists). Cilium for instance is great for that, so is Istio and the like. They aren't hard you just have to understand…

You shouldn’t have to think about all the implementation details of your deployment target. There shouldn’t be platform engineers or Kubernetes experts. Nobody should be writing YAML, or getting paid to set up Istio. Nobody should have to learn the Kubernetes architecture or know about the kubelet or EBPF. The tools should be simple enough with good defaults that application developers just click a button and have th…

So there should be magic on every layer except for the application?

That will never happen, the only thing you can do is pay someone like heroku to take care of that for you. Or if your project is small and plain enough you run “serverless” which is just routing to another platform team, as I’m sure you’re familiar.

It’s complicated because there’s a lot that goes on, kubernetes or not.

Re: A Eulogy for DevOps

#104

Post by deafpolygon » Sat, Jun 29, 2024, 4:11 AM UTC

So much of DevOps is being folded back into traditional roles now that the tooling has stabilized, and people are becoming disillusioned with the build, test, deploy loop.

It doesn't scale very well: the larger the codebase/team, the more burden on each individual to make this work.

Re: A Eulogy for DevOps

#105

Post by hitsurume » Sat, Jun 29, 2024, 4:13 AM UTC

llama052 wrote: post #54

I'll start by saying that I think knowledge for knowing layers underneath the application is fading in some circles, and that makes me sad. Having been a frontend guy some 10+ years ago, into a network engineer, then infrastructure engineering and now SRE. The amount of people on both sides of the developer circle and operations circle that do not want to understand what's going on is mind boggling. I was around when…

I'm interested in knowing more about how you guys implemented the operator model and decided on those tools. Was there a book or anything that was helpful in all of this?

Re: A Eulogy for DevOps

#106

Post by signal11 » Sat, Jun 29, 2024, 4:32 AM UTC

Sn0wCoder wrote: post #29

wokwokwok wrote: post #19

There’s so much truth in this. It really cuts to the heart of it when you looking at the “devops cycle” diagram with “build, test, deploy” …and yeah, those other ones… I remember being in a meeting where our engineering lead was explaining our “devops transformation strategy”. From memory that diagram turned up in the slides with a circle on “deploy”; the operational goal was “deploy multiple times a day”. It was abo…

DevOps and releasing multiple times per day does not always mean to PROD and in most industries that is impossible. Continuous Integration and Continuous Deployment should and do mean to the development branch and container. Locally I can write code and skip all the tests, skip the linter, skip prettier, etc.…. When I do a commit prettier is run. When I PR into develop other dev’s review the code. When its approved a…

It’s possible in many even heavily regulated domains, because there’s enough data now for regulators that it reduces risk.

Releasing multiple times a day in itself isn’t the point though.

The point is — adopting good practices to ensure trunk based development is pain-free. It’s about ensuring the release process is as automated as possible (if you’re releasing once a month, you are exercising your deployment process 12 times a year. When you’re exercising your deployment process 12 times a day, it really encourages you to work the kinks out.)

It’s also ensuring that your monitoring and alerting can keep up. Essentially it’s a signal to the entire org that “We will move fast. Deal with it.”

There’s data out there from even large non-FAANG orgs now about how this approach reduces sevs instead of increasing them.

Re: A Eulogy for DevOps

#107

Post by whoknowsidont » Sat, Jun 29, 2024, 4:41 AM UTC

zer00eyz wrote: post #43

osigurdson wrote: post #37

>> abandon technology like Kubernetes I think a lot of Kubernetes hate is misplaced. It is a great piece of software engineering, well supported and runs everywhere. You certainly don't always need it but don't create a bunch of random bash scripts running all of the place instead of learning how to use it.

Its not. The whole ecosystem around it is an example of Conways law and and a Google product. None of the people using it are google. Google, also, runs its own hardware. Shockingly it is a great product if you rent hardware, autoscaling is autospending. No one knows what a feature costs any more because its all just a big bucket your pouring money into for amazon to have 30 percent margin on. We need operations peop…

>The whole ecosystem around it is an example of Conways law

This is such an inaccurate take.

>we need to stop using containers as bags

Containers and container orchestration are a NEEDED and REQUIRED piece of the technology stack in the current reality. That doesn't mean people need to be ignorant of the details that make them work.

As someone who's been around for a while we are in a better place than when we had "dedicated operations people" that just gatekept everything.

What you hate are teams just parroting what other people are doing with the tech, not the tech itself. But let me tell you, if you're going to have teams of people pretend to know what they're doing wrapping it in a standard "bag" sure does make it a hell of a lot fucking easier to unfuck when things go wrong.

Re: A Eulogy for DevOps

#108

Post by signal11 » Sat, Jun 29, 2024, 4:44 AM UTC

Sn0wCoder wrote: post #56

lolinder wrote: post #51

Earlier quoted context omitted.

The big advantage to deploying to prod constantly is trunk-based development. Any system where you maintain a separate development branch is one where you're invariably going to be asked to cherry pick feature B to production but not feature A. This is a problem because you tested everything on a version of the code where B follows A, but now B needs to stand on its own. Can it? Maybe. But you didn't test that. With…

Congratulations on moving fast! If possible ‘trunk’ is best. Long lived branches are a problem if you are doing hot fixes to prod. The goal is the process avoids hot fixes and critical bugs. Some industries would not allow software to be released more than every few months. SQA needs to sign off. Users need to UAT. 2-week notice, 15-minute notice to log off. Downtime needs to be communicated to multiple time zones. S…

> Some industries would not allow software to be released more than every few months. SQA needs to sign off. Users need to UAT. 2-week notice, 15-minute notice to log off. Downtime needs to be communicated to multiple time zones. Software versions need to be auditable for regulatory.

As a person who works in a heavily regulated field ($$$), all the audit / traceability requirements apply. We still move fast (multiple releases a day) because we have demonstrated that it reduces risk and promotes stability — something regulators are very interested in. In concrete terms, faster release rates are correlated with lower sevs (or incidents). But the underlying cause is really that faster release rates cause teams to adopt more reliable delivery practices, such as automated testing, gradual deployments, and so on.

Re: A Eulogy for DevOps

#109

Post by wokwokwok » Sat, Jun 29, 2024, 4:47 AM UTC

smusamashah wrote: post #60

wokwokwok wrote: post #19

There’s so much truth in this. It really cuts to the heart of it when you looking at the “devops cycle” diagram with “build, test, deploy” …and yeah, those other ones… I remember being in a meeting where our engineering lead was explaining our “devops transformation strategy”. From memory that diagram turned up in the slides with a circle on “deploy”; the operational goal was “deploy multiple times a day”. It was abo…

I bumped into a DevOps job at Teradata, did it for two years, then left for where I belong, in gaming as a backend developer. It was a corporate with good pays and perks including international travel etc. Some friends are still there doing DevOps very happily. Talking to them it never feels a like a dying field. I hated my DevOps role so pardon me for only reading the headings of the article, but talking about it in…

The deep deep irony of hiring someone specifically to do "devops".

That's not devops.

Devops is when you have developers who are empowered.

When you hire someone specifically to do devops you are hiring an ops team and calling it devops.

So, yeah. That still happens, sure. However, I do think it's changing these days:

linkedin > connections -> control-F -> 'devops' => 0 hits

"Site Reliability Engineer" -> 10 hits

I think hiring SRE makes sense. I think hiring 'platform engineers' makes sense. People still do that, and will, as far as I know, far into to the future.

...but I think hiring devops means you never understood, even vaguely, what devops was; you're just using a buzz word in your job ad.

Re: A Eulogy for DevOps

#110

Post by signal11 » Sat, Jun 29, 2024, 4:52 AM UTC

fragmede wrote: post #85

Sn0wCoder wrote: post #56

Earlier quoted context omitted.

Congratulations on moving fast! If possible ‘trunk’ is best. Long lived branches are a problem if you are doing hot fixes to prod. The goal is the process avoids hot fixes and critical bugs. Some industries would not allow software to be released more than every few months. SQA needs to sign off. Users need to UAT. 2-week notice, 15-minute notice to log off. Downtime needs to be communicated to multiple time zones. S…

that doesn't work at scale though. the difference between dev, staging, and prod when there's a handful of services is fine. when there's 300 of them, and 200 of them are broken in the dev environment at any given time, that means you can't actually use the dev environment to do development in because the 300 other teams are also trying to do development in that same environment so their stuff is just as broken as yo…

We have environments with thousands of services and it scales fine. Why would dev (or your lowest integration environment) be broken most of the time?

> except that means the Kafka team

That (a Kafka team, or a DB2 team) is a bit of a red flag for me. Many (but not all) “tech” teams like that are part of the problem. Cross functional delivery teams work much better, because running tech X in isolation is often not valuable.

The one exception to this is “x as a service” teams. Eg DBaaS teams, Or even a team that offers a pub/sub service. But in those cases such teams are literally measured by uptime SLOs, so if a pub/sub team can’t deliver uptime, they’d need to improve.