Live data from Hacker News

2023 DevOps Is Terrible

abidmoon.hashnode.dev

91–100 of 126 posts

Re: 2023 DevOps Is Terrible

#91
post #38

To me, DevOps culture just feels like a way for businesses to save money by not staffing a dedicated infrastructure team and pushing all those responsibilities onto application developers. The amount of time I've spent fiddling with Terraform, Ansible, Kubernetes manifests, Helm charts, Jenkins configuration, GitHub Actions configuration, AWS IAM, and so on over the past few years is absurd, probably more than the ti…

One of the things that bothers me the most with the cooperate software development rat race is how many problems are being solved over and over again. Every company is staffing their own devops teams to build their own abstractions over these technologies so app developers don't have to worry about it. I personally know multiple devs who basically move from company to company reimplementing the same devops tools at each one.

It all just feels like a collosal waste of energy and collective resources.

Re: 2023 DevOps Is Terrible

#92

DevOps just makes the problem worse which it is intended to solve. DevOps started from "deploying is too hard, and the developers don't know how to run complicated shell scripts to deploy our software to release, anyone should be able to deploy it and have it scale!". However, most companies started to cargo-cult Google and just built everything in the "most scalable" way, incurring tons of overhead, both in deployme…

> DevOps just makes the problem worse which it is intended to solve. DevOps started from "deploying is too hard, and the developers don't know how to run complicated shell scripts to deploy our software to release, anyone should be able to deploy it and have it scale!". However, most companies started to cargo-cult Google and just built everything in the "most scalable" way, incurring tons of overhead, both in deployment friction and hardware costs, so we are back to DevOps being an almost completely separate field from software engineering, and the programmers still can't deploy software.

Absolutely this. Instead of collaborating to solve problems we are wasting time on pet projects.

Re: 2023 DevOps Is Terrible

#93

I think the biggest issue with devops is that it originally meant "socialising" your sysadmins, by getting them to sit with your devs, so that shit didn't get lost because nobody thought to talk to the right team. But then it morphed into "oh lets innovate with infrastructure" but the innovation turned into "lol lets just restart from scratch and ignore history" Anybody who used early k8s can attest to how un product…

Maybe in FAANG it's "back like we are in 2015" but the tools that have been brought into the mainstream like Kubernetes, automation tools and such have had a great impact across many small to large sized organizations.

It's done a great job to enforce consistency and immutable infrastructure in a way that is more approachable for most when you get past the scale of a few resources.

I've seen start ups write their on bash tooling, watch it devolve into some opaque thing that no one knows how it works or what it does. I've seen platform teams devolve into managing VM updates by hand, babysitting operating systems of every application deployed because it's treated as a long living pet, and it gets worse when team leads move on and then someone else comes in and half bakes their own flavor of change. Even better when networking gets involved, load balancing with HA where people roll their own solution. On top of that is the certificate story and how that gets managed. Don't even start with RBAC from a platform perspective, it's a nightmare. The toil filled platform days should be behind us. Plenty of companies are slow to evolve and you can find a lot of pain.

Perhaps I've had bad luck in my history of being a network engineer, infrastructure engineer, and an SRE. The one big benefit to these tools that are coming out is it can give some sort of standardization and it's giving you a ton of power in abstraction so that platform teams aren't micromanaging things that are just sorted for you.

Something like Kubernetes gives you load balancing, high availability, RBAC, networking and routing, resource management, node draining and rotations, auto scaling. Sure it's managed in yaml manifests, but it's doing a lot of lifting. I won't say that the abstraction can't bite you, you still need to understand the core concepts of things. However people should be understanding those concepts regardless.

The amount of work that takes for a small team at a startup that needs to scale services is immense without it. Right now I'm working with a 3 man platform team and we are running over 200 services in multiple regions, multiple database technologies and analytics stacks, we auto scale aggressively and we serve over a billion requests and terabytes of traffic a month with over 99.99% of uptime. We also deal with operations, security and every other hat that someone wants to throw our way. The metrics solutions to come out of this are also such a huge win, I remember the days of Nagios, Zabbix, SolarWinds etc. They were mostly static graphs showing you snapshots of what happens at a very high level, now we have metrics for everything and you can feed that into anything to self-remediate or escalate. I absolutely love it and can't wait to see it evolve more!

The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them. Some older sysadmins were legitimately afraid of rebooting servers because they were afraid they wouldn't come back. Some machines had over 900 days of uptime because they just couldn't keep up with the toil. Also this creates a wall between the platform and developers because it turns into a hand-off process where developers throw code over and the platform team has to implement some server and load balance logic, let alone meeting SLO/SLI criteria. Sadly, that's not uncommon to any industry. This team had 15+ people and had trouble keeping up let alone deploy and innovate. Outages in this mode are unavoidable, and pulling away from that old school mindset and starting over is VERY hard.

I'm sure a lot of that can be contributed to bad management and bad hires, however from my perspective using tooling and standardization will only help you build a platform that can actually grow. Making that predictable and easy to digest is important, and so is understanding the details underneath. I just know I will never want to go back to 2015 or anytime before. I'll babysit kubernetes node issues or upgrade woes all day if that's what it takes (not that I need to).

Granted this isn't the case at FAANG, they've got the brain power to build a consistent platform regardless. Everyone below that though benefits greatly.

Re: 2023 DevOps Is Terrible

#94

Earlier quoted context omitted.

Devops absolutely fixed a lot of issues. Going from manually merged SVN branches on a quarterly release monolith to a "click and release application" is a huge saver in developer time and pain.

that's great for you. at my company devops are still doing monthly releases and unable to deploy single microservices because all the services are too interconnected.

That doesn't sound like a devops problem, but a culture one.

Re: 2023 DevOps Is Terrible

#95
post #38

To me, DevOps culture just feels like a way for businesses to save money by not staffing a dedicated infrastructure team and pushing all those responsibilities onto application developers. The amount of time I've spent fiddling with Terraform, Ansible, Kubernetes manifests, Helm charts, Jenkins configuration, GitHub Actions configuration, AWS IAM, and so on over the past few years is absurd, probably more than the ti…

Who patches your VMs? Installs Antivirus.. installs vulnerability protection.. defines your resource groups.. virtual networks.. sets up your Azure subscriptions.. performs billing management.. etc?

I hope its not you. The whole idea was that there is a platform team responsible for providing a service or better yet the CSP does this for you and then you just have to architect as a platform team the things I listed above. It does not happen via magic. But the CSP platforms makes it certainly easier to provide a pre-audited version of the OS and tools.

You should be able to self-service yourself within say a developer like portal as part of an IDP as described here. Not figure out ALL of the infra yourself.

Re: 2023 DevOps Is Terrible

#96

I think the biggest issue with devops is that it originally meant "socialising" your sysadmins, by getting them to sit with your devs, so that shit didn't get lost because nobody thought to talk to the right team. But then it morphed into "oh lets innovate with infrastructure" but the innovation turned into "lol lets just restart from scratch and ignore history" Anybody who used early k8s can attest to how un product…

So you mean, DevOps has made Sysadmins more service oriented and now instead of having their pets and being in overprotective silos, they actually provide best practices and proper platforms for the poeple developing the software?

god do I hate the "pet" analogy

yes this server is special to me. No, I cannot move workloads somewhere else and shoot the server in the head. Because customers are running GPU workloads there, I cannot transparently migrate them

Re: 2023 DevOps Is Terrible

#98

I think the biggest issue with devops is that it originally meant "socialising" your sysadmins, by getting them to sit with your devs, so that shit didn't get lost because nobody thought to talk to the right team. But then it morphed into "oh lets innovate with infrastructure" but the innovation turned into "lol lets just restart from scratch and ignore history" Anybody who used early k8s can attest to how un product…

DevOps was originally about moving ops into dev teams. At Amazon they just gave devs pagers. Devs hated that. The SRE role seems to have taken over the app-facing side of what system engineers did being an interface between an app and the hardware. A lot of what I see SREs doing is things like app metrics and SLAs which were not what system engineers really did. And the SRE role has been pushed around from being embedded in dev teams, to being their own teams that managed apps to being a centralized DevOps team that looks an awful lot like the old silo'd split between Dev and Ops. I don't know where the actual system engineers went, they probably run Kubernetes clusters at FAANGs and have SRE titles these days.

Re: 2023 DevOps Is Terrible

#100
post #64

Since the in house server days, technology solutions have only become more and more time consuming to implement, ID & PII obsessed, and bulky (data footprint-wise). I am thankful I remember much more simple times, when apps were tiny, run on regular computers in my office, and you could symply restore from a local backup to bring a service back online. Now we have companies paying thousands of dollars a month to host…

> Simpler solutions, conveyed and implemented in human language, will be king in 2024. I would love this to be the case, but it seems that complex and expensive systems are too lucrative to everyone involved: CTOs get to manage big budgets and bit teams, devops people get paid large salaries to justify the CTOs salary and large service budgets, etc.

Agreed, many of those huge and overly complex projects also fail and overrun implementation schedules though, and they only ultimately operate with a ton of workarounds and shortcuts. I find a lot of projects that barely match original documentation limping along, or just even after a major failure or breach in my line of work, with the customer desperate to find a patch to just keep them running after a bunch of critical data is under siege within apps.

A key for me is convincing a customer to simplify goals and implement an MVP on a reliable framework, to plan versions, and to thoroughly test... Makes me a code cop, but there are reasons why these steps are in place for mission critical services, and if a customer wants to skip them and just deploy in careless "Twitter Dev Mode" I have them physically sign paperwork that states their accountability for the bad decision... When individual accountability is mentioned, they usually opt for the more stable & reliable methods of dev. Hah.

Post reply on HN