2023 DevOps Is Terrible • Hacker News Board

Re: 2023 DevOps Is Terrible

#81

Post by sisve » Sun, Oct 01, 2023, 9:10 PM UTC

notanormalnerd wrote: post #57

KaiserPro wrote: post #51

I think the biggest issue with devops is that it originally meant "socialising" your sysadmins, by getting them to sit with your devs, so that shit didn't get lost because nobody thought to talk to the right team. But then it morphed into "oh lets innovate with infrastructure" but the innovation turned into "lol lets just restart from scratch and ignore history" Anybody who used early k8s can attest to how un product…

So you mean, DevOps has made Sysadmins more service oriented and now instead of having their pets and being in overprotective silos, they actually provide best practices and proper platforms for the poeple developing the software?

Im not sure how former sysadmins feel about it, but from a dev standpoint, DevOps worked great for us. Making a clean cut on a plattform. Was my app down, I got message/called. If the platform was down it was the Infrastructure team / PaaS provider that needed to fix it.

In the old days we had so few deploys and a hotfix was something that was extraordinary. I actually had a board of something in another country approving that we where allowed to deploy. They knew nothing about the application..

Re: 2023 DevOps Is Terrible

#82

Post by mschuster91 » Sun, Oct 01, 2023, 9:16 PM UTC

shrimp_emoji wrote: post #74

mschuster91 wrote: post #54

Yuck, Portainer... when I read that I get nightmares. The problem is: when you give developers access to Kubernetes, to ECS, to Portainer, to whatever they. will. not. care. about literally anything. You'll find a hotpot of cobbled together Dockerfiles with zero provenance, with base images pinned to stuff years old, and completely inefficient layer ordering. Pipelines won't use caching or parallelism (okay, sometime…

And some devs do learn how Linux works but don't want to waste their time memorizing how shitty $DEVOPS_PRODUCT that'll be replaced in two years works. :D

Indeed. Which is why I waited for three years to take on Docker, and two more for Kubernetes after I got burned with DC/OS, and why I entirely left the NodeJS/frontend ecosystem. I'm too old to act as the ripener for bananaware any more, and I'm pretty sick of companies exploiting other companies or, even worse, individual developers to act as their free QA department.

Re: 2023 DevOps Is Terrible

#83

Post by fmajid » Sun, Oct 01, 2023, 9:36 PM UTC

Someone at a Linux company once said: if you combine "Terraform" and "Ansible", you get "Terrible".

Re: 2023 DevOps Is Terrible

#84

Post by tstrimple » Sun, Oct 01, 2023, 9:44 PM UTC

anotherhue wrote: post #75

medellin wrote: post #17

Earlier quoted context omitted.

The thing i have come to dislike is that in both companies i work for it lead to strange working relationships between ops/eng. anything that breaks in the deploy pipeline is now “the devops team problem” and they normally act like it’s also your fault. In the past this has meant people messaging me saying they are blocked because their dockerfile fails in CI. Then you walk through it with them and it’s also broken l…

I have seen infra/devops scapegoated too much. Half the time the developers in question won't even read the error output before raising a ticket. The only solution I see is to enforce commit checks before hitting shared environments, but that comes with its own set of complaints.

> Half the time the developers in question won't even read the error output before raising a ticket.

If they had no DevOps team to raise a ticket with, and it had to get done they would find some way to figure it out if they had the ability to modify and initiate their own pipelines. The reason I like "you build it you run it" is I feel like I have far more control over my own destiny. You're not having to negotiate and coordinate with a separate team through some ticketing system. Far less wasted back and forth and miscommunication. If that means you have to expand one of your teams with an additional DevOps person to foster understanding of the tooling on a team who hasn't done it before that's fine. But don't make them the DevOps person. Their goal is to enable the rest of the team to pick up these skills. Their goal should be to transition into a normal contributing member of the team working on product development because the team no longer has a need for a DevOps person.

Re: 2023 DevOps Is Terrible

#85

Post by dijit » Sun, Oct 01, 2023, 9:48 PM UTC

hinkley wrote: post #49

ecshafer wrote: post #42

Earlier quoted context omitted.

A devops team is in practice typically very different than a sysadmin team. Good sysadmins automate the hell out of everything, but it’s usually not something other people can use. Devops teams automate all of that stuff but then they also let teams build new infrastructure when they do deploys and automate that whole process, but it’s client facing. It’s development and operations, it’s a pretty good name, though I…

The quality of devops scripts, from the perspective of a person who likes to write code that has to work right the first time, is about two standard deviations better than sysadmin scripts. But still of a par that would keep someone from getting promoted past a Senior Dev title. What’s worse is they can be condescending while shoveling out absolute garbage. If I’m honest that’s the part that bugs me the most.

I don't deny your experience, but my experience is that when it comes to domain specific scripts sysadmins fared much better than developers.

I still see this, developers use archaic or absurd incantations in their build-scripts because they kludge something that works in bash on their machine then push it into yaml and move on, forgetting to comment, document or try to simplify the line so that it's easy to break apart later when it has to be debugged.

Some of the best interpreted code I have ever seen came from a former sysadmin (it was powershell though, that comes with it's own warts of course).

The issue is that we collectively decided that sysadmins should just stop existing instead of allowing it to grow properly. We forget how bad things were for developers back when we decided we didn't like sysadmins. There were companies still passing around code on a USB stick because developers didn't want to depend on SVN being down. (I am not kidding).

Git was introduced 2005 and wasn't really everywhere until 2012, but 2014 sysadmins had all but been retired. This is the timeframe we talk about.

Developers of that time were as bad as the sysadmins were, the problem was that the developers had time and space to grow and we judge the developers of 2012 as if they had all the tools, knowledge and incremental improvements in workflow that they have today.

Re: 2023 DevOps Is Terrible

#86

Post by whstl » Sun, Oct 01, 2023, 9:50 PM UTC

KaiserPro wrote: post #56

whstl wrote: post #41

Earlier quoted context omitted.

The only thing I like about the return of sysadmins and the end of the "you build, you run it" culture of Devops it is that I'm 100% off the hook if something dies in the middle of the night. The so-called DevOps guys are stressed out of their marbles, but when the CTO calls me 4 AM I can't say anything but "Sorry dude, I'm not a clairvoyant. I have no permission to do shit in prod, call Ops or give me AWS access now…

> The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes. Yeahnah, you're part of the problem. If you've not documented it properly, and more importantly, if the team are weary of prodfucking TF/CF scripts…

Sorry, but how did you manage to make so many assumptions from that?

Everything I need from Ops is already enumerated and specified more than well in tickets (which is required, as is par for course with the sysadmin from hell stereotype), and documented in Markdown files within my app's source code.

I generate scripts for them since that's what they WANT and ASKED for several times.

They literally want me and other managers to help with DevOps (Heck, I'm even AWS certified), but don't trust me and others to give access. They ask for scripts and I comply to save their time. How am I on the hook if they want me to create scripts, but some of the new hires don't know how to use Terraform/CloudFormation?

> "there is a culture problem. You both need to understand how that works, if they don't have enough time, then you're understaffed, that needs to be fixed."

The culture problem is Ops wanting to do everything and not trusting. If they don't want to "prodfuck", just review every script. Or perhaps test them in the staging AWS account that I pay for with my own money. Or just stop asking me to make infrastructure as code. Or suggest a fourth option instead of wasting my time and being disrespectful.

I can't change other people's behavior. I can only change mine.

If they are understaffed I can help, or they can hire more. Wasting my time and making me do pointless work is not the answer.

Re: 2023 DevOps Is Terrible

#87

Post by jiggawatts » Sun, Oct 01, 2023, 9:56 PM UTC

In my opinion, more than half the problem is the tooling, as indicated by the cartoon in the linked article.

DevOps pipeline languages are generally either weakly typed or stringly-typed. They generally expect users to do the control character escaping, in several different obscure formats. Did I say escaping? I meant three layers of nested escaping!

Variable/macro substitution as a rule uses syntax that overlaps with either PowerShell, Bash, or both because otherwise it would be too easy.

Another basic thing is the absence of a debugger — I have to run a complete pipeline with a change to dump out environment variables!

Instructions contain the display names of paths, scripts use the variable names, but the output (and errors!) use the values with no clear mapping backwards. What the eff is C:\s\1\x? I dunno, but it caused an error!

The worst sin is not having a local development experience where the entire pipeline can be run without having to commit code and push stuff almost all the way to production. Even if the pipeline agent is made available, it’s useless without the underlying VM image which is huge and not easily obtained. Not to mention KMS/KeyVault and other similar services on which pipelines can depend.

Developers complain about having to do DevOps, but in my opinion the problem isn’t who is doing it, but how. The tooling is just so bad, so very very bad, that anyone would hate it if they were the ones forced to do it. I’m an ex-dev-now-SRE and I hate it.

Re: 2023 DevOps Is Terrible

#88

Post by whstl » Sun, Oct 01, 2023, 10:07 PM UTC

hinkley wrote: post #47

whstl wrote: post #41

Earlier quoted context omitted.

The only thing I like about the return of sysadmins and the end of the "you build, you run it" culture of Devops it is that I'm 100% off the hook if something dies in the middle of the night. The so-called DevOps guys are stressed out of their marbles, but when the CTO calls me 4 AM I can't say anything but "Sorry dude, I'm not a clairvoyant. I have no permission to do shit in prod, call Ops or give me AWS access now…

I always do a better job of writing logic when I sympathize with the victims. Some of those late night calls are devs setting ops up for failure. It was meant to close an open loop.

Yep. I'm otherwise happy to help with things in the middle of the night, but I need the means to actually diagnose and fix it.

As it is, the Ops manager is probably going to have a stroke due to overwork, and it wasn't for lack of people wanting to help.

Re: 2023 DevOps Is Terrible

#89

Post by whstl » Sun, Oct 01, 2023, 10:15 PM UTC

mschuster91 wrote: post #58

jen20 wrote: post #31

Earlier quoted context omitted.

To be more blunt about it: if the person writing the code is not the one on the pager at 3am, they have no serious incentive to make it correct or resilient. This is why "you write it, you run it" is so important - the most reliable software is the software never written, but if something must be written, it's important that all incentives are aligned for it to work right. Of course, this is only at the operational e…

> To be more blunt about it: if the person writing the code is not the one on the pager at 3am, they have no serious incentive to make it correct or resilient. The problem is in 99% not that the developers don't want to do correct code. The problem is that product/project management doesn't allocate the time and budget for clean development practices, proper QA, and refactoring because their metrics are revenue, rele…

Can 100% confirm. Last place that had that pager duty tried to throw this shit on me. Product Owner was constantly denying time to build monitoring.

Everything in this loop was misaligned and rigged against devs. The only power I had as was to quit.

Now that this is over and traditional Ops is back and called SRE, it is Product Owners, Support, Managers, Developers and CTOs screaming all at once at their team. Let's see where this will lead us.

Re: 2023 DevOps Is Terrible

#90

Post by ecshafer » Sun, Oct 01, 2023, 10:18 PM UTC

dijit wrote: post #85

hinkley wrote: post #49

Earlier quoted context omitted.

The quality of devops scripts, from the perspective of a person who likes to write code that has to work right the first time, is about two standard deviations better than sysadmin scripts. But still of a par that would keep someone from getting promoted past a Senior Dev title. What’s worse is they can be condescending while shoveling out absolute garbage. If I’m honest that’s the part that bugs me the most.

I don't deny your experience, but my experience is that when it comes to domain specific scripts sysadmins fared much better than developers. I still see this, developers use archaic or absurd incantations in their build-scripts because they kludge something that works in bash on their machine then push it into yaml and move on, forgetting to comment, document or try to simplify the line so that it's easy to break ap…

I think part of this issue is: what is a sysadmin? That’s a pretty hard question to answer today. Maybe it was more straight forward in the 80s or 90s. But if you go to /r/sysadmin you have desktop support, help desk, office 365 admins who only use a gui, solo it guy in a small business, and also programmers, devops, cloud architects and network engineers.

The sysadmins who were already automating large amounts of tasks for n Unix servers and k switches with p services, had a vastly different skill set and migrated to the more modern software defined devops SRe world no problem. But plenty of sysadmins basically could (and some still) only use a gui, and do things from trial and error. Those people might have retired or become managers, but they had a lot more issues with the transition. Which to be fair the transition happened fast and hard in some companies.