System Initiative: Second Wave DevOps • Hacker News Board

Re: System Initiative: Second Wave DevOps

#21

Post by jrott » Wed, Jun 21, 2023, 5:01 PM UTC

pmoriarty wrote: post #14

hinkley wrote: post #10

Earlier quoted context omitted.

Fundamentally, I think the “why” is still the fact that DevEx started with devs, and DevOps started as a collaboration with Ops people, who already were not speaking the same language of robustness that we do. Which is a little weird, and probably part of the friction between the groups. They historically took on the reliability role, if nobody else did, but they were implementing reliability on top of a house of car…

"They historically took on the reliability role, of nobody else did, but they were implementing reliability on top of a house of cards, which is a kind of hypocrisy that makes even mediocre devs bristle. Don’t lecture me on robust software, boyo. Your tools are made of string cheese and staples." I don't know why you'd blame ops for the crappyness of the tools they have at their disposal. Yes, Ansible, Salt, Puppet,…

No no no Kubernetes or Serverless or ChatGPT is going to save us this time.

More seriously it always going to be complicated and annoying. It's really past time we started dealing with the fundamental complexity of everything we are trying to do with software.

Re: System Initiative: Second Wave DevOps

#22

Post by zsoltkacsandi » Wed, Jun 21, 2023, 5:04 PM UTC

hinkley wrote: post #10

aftbit wrote: post #8

>Doing “DevOps work” is unquestionably the worst part of building a modern application. It’s full of tiny papercuts, indignities we suffer in our toolchains, our feedback loops, and our software. It’s a city of brutalist buildings filled with sharp-edged couches pretending to be comfortable. Think of all the advances in how we interact with tools in other domains - then take a look at the way you build, deploy, and o…

Fundamentally, I think the “why” is still the fact that DevEx started with devs, and DevOps started as a collaboration with Ops people, who already were not speaking the same language of robustness that we do. Which is a little weird, and probably part of the friction between the groups. They historically took on the reliability role, if nobody else did, but they were implementing reliability on top of a house of car…

My experience is (as a former dev, current ops) that the problem isn’t that dev and ops people aren’t speaking the same language, nor the tooling or processes.

The problem comes from the management/business side. They hire devs and tell them that ship features as fast as you can. Also they hire ops guys and tell them that I want this whole thing super reliable, we can’t afford a minute downtime.

In my opinion this is why DevOps is mostly pointless. We are trying to fix with tooling, processes, new tech, and fancy roles the fact that business people don’t want to make compromises or choose between the pace of delivery and reliability.

Re: System Initiative: Second Wave DevOps

#23

Post by Spivak » Wed, Jun 21, 2023, 5:04 PM UTC

hinkley wrote: post #10

aftbit wrote: post #8

>Doing “DevOps work” is unquestionably the worst part of building a modern application. It’s full of tiny papercuts, indignities we suffer in our toolchains, our feedback loops, and our software. It’s a city of brutalist buildings filled with sharp-edged couches pretending to be comfortable. Think of all the advances in how we interact with tools in other domains - then take a look at the way you build, deploy, and o…

Fundamentally, I think the “why” is still the fact that DevEx started with devs, and DevOps started as a collaboration with Ops people, who already were not speaking the same language of robustness that we do. Which is a little weird, and probably part of the friction between the groups. They historically took on the reliability role, if nobody else did, but they were implementing reliability on top of a house of car…

Because you're not talking about that thing that ops people are talking about. We build reliable systems. You're talking about reliable software. Ops people come from the perspective that all software is inherently unreliable including your app, especially your app and have to work within those constraints.

Terraform and Ansible look like gyroscopes compared to the build process of any modern software stack. We offered our dev teams a whole ass pizza party every time they had 10 green builds (on main) in a row. In three years we've paid it out once.

Re: System Initiative: Second Wave DevOps

#24

Post by mailund » Wed, Jun 21, 2023, 5:07 PM UTC

> Things like using source control, shared observability, feature flags, dark launching, continuous integration, and continuous delivery are widely considered best practices.

I seriously want to know which places this is! I've been at 5 different companies, and I've never been a place where people don't look at me like I'm speaking French when I suggest dark launching a feature or introducing feature toggles. I've yet to experience a place that actually integrates continuously, as opposed to merely having a ci pipeline without actually doing continuous integration.

Re: System Initiative: Second Wave DevOps

#25

Post by arpyzo » Wed, Jun 21, 2023, 5:08 PM UTC

I don't understand why the prevailing opinion is that it's acceptable for software development to be complex. It's simply the nature of the beast! Not infrastructure though. Infrastructure should be simple because...?

Perhaps the reason your organization is only deploying once a month is the same reason it takes it a month to make simple code changes, which is because you haven't hired sufficiently capable engineers, and not because you're missing some magic.

edited: grammatical error

Re: System Initiative: Second Wave DevOps

#26

Post by jsiva » Wed, Jun 21, 2023, 5:12 PM UTC

chologrande wrote: post #17

jsiva wrote: post #15

Earlier quoted context omitted.

Text-based interfaces still have the same issue with being able to make a typo or tab completing without checking. Seems like the major advantage is that these text based tools are able to be versioned well through scripts. This might be fixed with a GUI version of autohotkey, turning these gui interactions into a script.

Mandated peer review, planned actions, and automated risk evaluation are part of our infra pipeline. This typically doesn't exist outside of software dev style pipeline.

I was about to disagree but the automated risk evaluation (ARE) part definitely qualifies your whole statement. Going to go off on a tangent: how do we introduce automated risk evaluation to environments outside of software development. Implementing ARE as software is probably the most efficient method in terms of time and resources. But ideally (some) users of ARE should be able to improve on it. In the case of "traditional" engineering (civil/electrical/chemical etc.) there are engineers who specialize in numerical methods and can improve on ARE. But what about professions where software development skills are not as widespread (or seen as a legitimate contribution to the field). There are still probably going to be members of these professions with software development abilities but is there a point where other methods could be considered (i.e. electrical/mechanical methods) for ARE implementations.

Re: System Initiative: Second Wave DevOps

#27

Post by hinkley » Wed, Jun 21, 2023, 5:17 PM UTC

zsoltkacsandi wrote: post #22

hinkley wrote: post #10

Earlier quoted context omitted.

Fundamentally, I think the “why” is still the fact that DevEx started with devs, and DevOps started as a collaboration with Ops people, who already were not speaking the same language of robustness that we do. Which is a little weird, and probably part of the friction between the groups. They historically took on the reliability role, if nobody else did, but they were implementing reliability on top of a house of car…

My experience is (as a former dev, current ops) that the problem isn’t that dev and ops people aren’t speaking the same language, nor the tooling or processes. The problem comes from the management/business side. They hire devs and tell them that ship features as fast as you can. Also they hire ops guys and tell them that I want this whole thing super reliable, we can’t afford a minute downtime. In my opinion this is…

That’s definitely part of the dynamic. My biggest regret with automated testing is that software used to be a triumvirate of Quality, Dev, and Management, and when dev was fucking around they had two teams hitting them, and when Management was out of control, they had everyone mad at them. Get rid of QA and it’s Us vs Them and that worked briefly at the dawn of Agile but they got wise.

OP’s should replace QA at that table to rebalance the equation. But again, and as you illustrated, we have an adversarial relationship that takes a lot of across the aisle work to introduce sanity.

Re: System Initiative: Second Wave DevOps

#28

Post by hinkley » Wed, Jun 21, 2023, 5:22 PM UTC

Spivak wrote: post #23

hinkley wrote: post #10

Earlier quoted context omitted.

Fundamentally, I think the “why” is still the fact that DevEx started with devs, and DevOps started as a collaboration with Ops people, who already were not speaking the same language of robustness that we do. Which is a little weird, and probably part of the friction between the groups. They historically took on the reliability role, if nobody else did, but they were implementing reliability on top of a house of car…

Because you're not talking about that thing that ops people are talking about. We build reliable systems . You're talking about reliable software. Ops people come from the perspective that all software is inherently unreliable including your app, especially your app and have to work within those constraints. Terraform and Ansible look like gyroscopes compared to the build process of any modern software stack. We offe…

The operational tools should be the most stable bits and instead they are janky as fuck and I’ve spent too much of my career smacking victim-blaming tennis balls back over the net. If you look at what Ansible replaces it’s a wonder production ever worked at all. If you have to baby your automation it’s not automation.

Ops people are not used to thinking in boundary conditions. Hell, devs forget half the time. That’s part of why people wanted to merge them in the first place. Get the right sorts of cynicism together in a room and make me something with a big green button an idiot can push while everyone is in a meeting.

Re: System Initiative: Second Wave DevOps

#29

Post by thezilch » Wed, Jun 21, 2023, 5:32 PM UTC

Spivak wrote: post #23

hinkley wrote: post #10

Earlier quoted context omitted.

Fundamentally, I think the “why” is still the fact that DevEx started with devs, and DevOps started as a collaboration with Ops people, who already were not speaking the same language of robustness that we do. Which is a little weird, and probably part of the friction between the groups. They historically took on the reliability role, if nobody else did, but they were implementing reliability on top of a house of car…

Because you're not talking about that thing that ops people are talking about. We build reliable systems . You're talking about reliable software. Ops people come from the perspective that all software is inherently unreliable including your app, especially your app and have to work within those constraints. Terraform and Ansible look like gyroscopes compared to the build process of any modern software stack. We offe…

Oh boy, a whole pizza!

Re: System Initiative: Second Wave DevOps

#30

Post by deadeye » Wed, Jun 21, 2023, 5:36 PM UTC

Perhaps we don't deploy six times a day because we're responsible for something a little more important and delicate than a free photo sharing website.