Live data from Hacker News

2023 DevOps Is Terrible

abidmoon.hashnode.dev

21–30 of 126 posts

Re: 2023 DevOps Is Terrible

#21
post #4

The creation of DevOps teams instead of SRE / Platform seems pretty common and I think originates from ignorant engineering leaders and poorly defined roles and responsibilities. You want to leverage the expertise of the folks who focus on cloud infrastructure, but you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput…

Can you expand on what you mean by this? Specifically: "you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput and reliability"

Your application architecture dictates how it should be deployed and the other way around, if you try to decouple the two you're bound to run into issues. Depending on what your application does, high throughput with some loss might be ideal, or you might need at least once delivery instead while sacrificing throughput, or any other dynamic. If your deployment is done by an entirely different team from the team that builds the product itself you have little hope of predictable positive outcomes without a lot of pain.

Re: 2023 DevOps Is Terrible

#22
post #4

The creation of DevOps teams instead of SRE / Platform seems pretty common and I think originates from ignorant engineering leaders and poorly defined roles and responsibilities. You want to leverage the expertise of the folks who focus on cloud infrastructure, but you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput…

Can you expand on what you mean by this? Specifically: "you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput and reliability"

SRE needs authority to hand back the pagers. That’s a key aspect.

Re: 2023 DevOps Is Terrible

#23
post #4

The creation of DevOps teams instead of SRE / Platform seems pretty common and I think originates from ignorant engineering leaders and poorly defined roles and responsibilities. You want to leverage the expertise of the folks who focus on cloud infrastructure, but you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput…

Can you expand on what you mean by this? Specifically: "you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput and reliability"

[dead]

Re: 2023 DevOps Is Terrible

#24

I'm technologically stuck in the era when everyone hated JavaScript. I provision instances with bash and services over REST APIs.

To be fair JS was a bit ridiculous pre ES2017. These days, you have async/await (even at the top level) and it just works when you use `node file.mjs`

For TS, install '@digitak/esrun' and run:

esrun file.ts

(no tsconfig, package.json, etc. required)

That basically turns JS into the async equivalent of python and is great for boring sysadmin tasks.

Re: 2023 DevOps Is Terrible

#25
post #3
post #2

The fact that we have a “devops team” at work infuriates me. It’s enough to distract me me from how “Scrum” has destroyed Agile.

What is so horrible about it? Our devs are happy they can focus on fixing unit tests in pycharm, while others care about where the code is going.

So in other words, you made silos. More throwing things over the wall and having it be someone else's problem. That's fundamentally the antithesis of DevOps.

People do not fix the pain they don't see, and managers doubly so. By shifting operations off to a "devops" team, you just end up with developer teams that don't know about, and don't care to fix, the things that make it painful to deploy or run their code in production. Those that have to deal with the pain rarely have any political leverage they need to even enact change. There's zero motivation for either the developers writing the software _or their management_ to fix it. Dev manager incentives are purely aligned with getting code and features built.

To give a nice real world example, building out new regions for AWS services used to be a pain in the arse (since I left I hear some major top down initiatives have fixed that). The service team I worked for, almost doubly so painful for region build. Standing up an entire region was something like ~50 days of engineering time. Not because it was especially hard, per se, but because the ops team always had to wade through piles of refactored shit that introduced new circular dependencies, or wasn't documented, or broke other aspects of any automation we could produce. (I remember one DynamoDB schema overhaul, where we did a complicated migration. Next region build, service components wouldn't stand up because we'd used the automation to create the new table schemas that had replaced the old table schema entries, but the service code still expected to see the old schemas in the right format. It took several months to get the dev manager to agree to engineers spending time cleaning up old schema references from their code base.)

The dev manager incentives were not aligned with making it easy. It wasn't their engineering resources being consumed, and they had features they needed to ship. The developers never changed the way they worked because even though they were told the issues (and even though they were, virtually to a person, great people to work with, considerate etc.), they never experienced the pain so they never thought about those consequences of what they were doing when they were doing it.

Ops manager never had any leverage, because every single time we managed to get a region launched in time as to not be an issue, and every time things were down to the wire it was clearly as a result of the ops team not building the right automation :eyeroll:, rather than that the dev teams were making some kind of brand new nightmare each time.

It wasn't until the ops team had lost enough members of staff to that BS, and region build shifted on to the developers themselves, that suddenly most of the problems went away, never to appear again. They were aware and conscious of the pain certain decisions would cause because they had felt it themselves, and they made sure not to introduce more pain.

DevOps is about making sure incentives and politics align to make things overall better. Shoving it in to a silo is literally the complete opposite.

Re: 2023 DevOps Is Terrible

#26

DevOps just makes the problem worse which it is intended to solve. DevOps started from "deploying is too hard, and the developers don't know how to run complicated shell scripts to deploy our software to release, anyone should be able to deploy it and have it scale!". However, most companies started to cargo-cult Google and just built everything in the "most scalable" way, incurring tons of overhead, both in deployme…

Devops absolutely fixed a lot of issues. Going from manually merged SVN branches on a quarterly release monolith to a "click and release application" is a huge saver in developer time and pain.

Re: 2023 DevOps Is Terrible

#27
post #10
post #3

Earlier quoted context omitted.

What is so horrible about it? Our devs are happy they can focus on fixing unit tests in pycharm, while others care about where the code is going.

Because that is the software developer/systems engineer division that already existed 20 years ago. The idea behind devops was to be a single team.

> The idea behind devops was to be a single team.

I assume you are pointing out that split-teams is the insane part, in which case I agree. The philosophy was supposed to define the blurred line between developers and operations, using tooling to help facilitate the shared responsibilities where sensible and make hand-off a breeze.

As you pointed out, the division existed 20 years ago, and it still exists now at many places. That lack of integration doesn't offer much improvement in relation to the past.

Re: 2023 DevOps Is Terrible

#28
All this IT stuff fails, needs to be replaced, repaired, or adapted to a new situation.

The best solutions, whatever the methodology and avenue of implementation, are made better when the builders "live below the dam."

The incentive to build the best dam is when the party responsible must live below the dam.

The best metric is customer retention. Directors of customer support need to be on the phone with escalations, not just watching metrics. There's a leak? Take the complaints. Embrace the hate. Fix the damn dam.

Re: 2023 DevOps Is Terrible

#29
post #6

Yep, bang on. As for the PE being the “solution” it is incredibly important and sorely needed but it won’t solve your enterprise culture.

Regarding enterprise culture I’m curious who will be in this “platform engineering” team and what expertise they’ll have.

In (enterprise) reality it will more than likely be a bunch of sysadmins and former “network/storage guys” who saw which way the wind was blowing and got a cloud cert. They really have no business forcing a particular infra pattern on application teams because they have no application level expertise and nothing about adding LUNs to ESXi, typing Cisco configs, or writing bash scripts to automate yum update gives you expertise in application design/architecture.

Then because they know nothing about application level concerns the PE team’s golden patterns will either be ignored or will be customized for every application to the point that they’re unmaintainable.

Note this is the enterprise view. It’s probably different at tech focused companies.

Re: 2023 DevOps Is Terrible

#30
post #3

Earlier quoted context omitted.

What is so horrible about it? Our devs are happy they can focus on fixing unit tests in pycharm, while others care about where the code is going.

Because it's just a hip way of saying you have a sysadmin team. How can something or someone be "devops" if all the developers are in one team doing dev tasks, and the ops in a separate team doing ops stuff?

If we keep reinventing the same paradigms; isn't it a perfectly fine paradigm?

It's unfortunately inherent that some people want predictable, scalable, observable, stable- and at the same time they want fast moving and many new features.

I think it's not so much that dev and admin work is terribly difficult, it's more that "a man cannot serve two masters" and someone will be focusing on stability and someone else will be focusing on quickly deploying.

Trying to do both yourself I think leads to burnout.

Post reply on HN