Live data from Hacker News

DevOps is broken

blog.massdriver.cloud

101–110 of 363 posts

Re: DevOps is broken

#101

> The problem is most engineers don’t want to do operations work. There's your problem. You have people who build stuff without caring where and how it runs. Recipe for disaster.

I don't understand. I want to develop code. I don't want to become an AWS/S3/Github/Jenkins/Action/terraform/etc expert. I know enough of this to be dangerous but not at a level that passes as professional. Yet I am regularly tasked with maintaining the full deployment of code. There's a reason to have a team of people doing this "DevOps" work. Just like we have a team of people who do SRE. It creates a standard and…

> Yet I am regularly tasked with maintaining the full deployment of code.

Same shoes, but I have different perspectives. I can figure out where it's not working and if it's my/our team's area of responsibility then we go fix it.

Since we handle infrastructure (as code) and deployments in team, along with all the development, most of the problems are handled by us. Unless it's clear that it isn't, e.g. some API that we consume that keeps throwing 500, can't fix that.

Our operations is helped by 100s of automated tests and 1000s of metrics. I always thought that this is DevOps, but it sounds different from what most of the people here are alluding to.

Re: DevOps is broken

#102
post #17

I’ve long held this opinion but I consistently get drowned out. DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways. All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be. There is also a lot of rewriting of history that gets…

I think everyone can agree that for most complicated things there are many sides to a story that makes up the present state of the system. I think taking a very opinionated stand in any complicated subject by itself negates the possibility of a mature and complete analysis of any subject. So the title itself of this article is unfortunate.

Re: DevOps is broken

#103
I was semi salty reading this....but they are spot on in a lot of ways. There are times where my code looks like a bad 4 year old wrote it, but 10 minutes later I'm in a conversation where I have to explain basic security concepts, like not leaving everything wide open to the internet, to some random Sr Software Dev.

   For every operations person without software development skills, there are FORTY engineers without cloud operations skills. If you are going to build an internal platform, you’ll need experts with overlapping experience in both fields working together.
   
I guess I just need to find a role with more inter-team collaboration; Being able to mostly self teach is great, until you have no one to learn from anymore.

Re: DevOps is broken

#104
post #26

> If the “DevOps” team ships a Postgres RDS instance it will run fine forever, that is until an application starts using it. All of a sudden a cascade of N+1s hit, the CPU spikes, and queries grind to a halt. Who is woken up? And why does this always happen at 2 AM? In this scenario, there is nothing for operations personnel to do, yet here they are. This is definitely a symptom of a broken model and not what I would…

> This is definitely a symptom of a broken model and not what I would call devops.

I'm I the crazy one here? This kind of work is my bread and butter as a "devops" person. PagerDuty fires, I find the bad query, match it up with the most recent PRs to find where the n+1 got introduced and either patch it right there at 2AM with the on-call manager's approval or roll it back. Then we have a postmorterm in the morning with the team.

I'm the person positioned the best to do this work because I'm god in my little ops domain, have the most visibility and the biggest toolbox of potential fixes.

Re: DevOps is broken

#106
DevOps has suffered the same fate as Agile. In particular there is one thing that both had in common that got lost somewhere in most implementations: cross-functional teams [0].

Agile teams were supposed to be composed of not just devs doing everything (quickly, because agile is about raw speed, right?) but people competent in the various aspects of the system development, potentially deployment, customer needs, etc. working together as multidisciplinary teams to meet a particular objective. The purpose of this was to break the silo that is common because it feels natural to many managers (the same sort, I presume, who don't like when their mashed potatoes touch their fried chicken on the plate).

Silos impede communication and promote the "throw it over the wall" approach to product/system development. A system engineer (or team of) made the design after sales (and possibly only sales) talked to the users. Throw that design over the wall and let the devs build it. Devs throw it over the wall to test, maybe there's a volley. Eventually it's tossed to ops. A goat is sacrificed and maybe it works.

Multidisciplinary teams are able to communicate across those boundaries because instead of the role-silo the roles are all in the same team, working (more clearly) towards one common objective. But then businesses managed to fuck it up. They got rid of test, devs do all testing now. Devs do all the database logic. There are no UI/UX experts anymore, it's all full-stack, and on and on.

DevOps was supposed to be the same. It was supposed to take that Agile cross-functional team and add in the sysadmin/operators (among other things). The critical problem being solved was the two (really more, but at the limit) silos of dev and ops failing to communicate. A Friday release with Ops spending the weekend rolling things back because Dev wasn't there and didn't even know the system wouldn't work as intended. Maybe their test environment was too different from the operational environment, the reasons matter a bit but there are too many to enumerate.

Instead, businesses did what they do best, they fucked it up. Again. They said, "Take the sysadmins and teach them to code. If they can't, dump them in the woods. The devs will takeover the world." Then they started making "devops" job listings and full-stack grew to encompass a new set of skills.

[0] https://en.wikipedia.org/wiki/Cross-functional_team

Re: DevOps is broken

#107
post #17

I’ve long held this opinion but I consistently get drowned out. DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways. All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be. There is also a lot of rewriting of history that gets…

I think everyone can agree that for most complicated things there are many sides to a story that makes up the present state of the system. I think taking a very opinionated stand in any complicated subject by itself negates the possibility of a mature and complete analysis of any subject. So the title itself of this article is unfortunate.

Shouldn't a mature and complete analysis result in a conclusion, i.e. an opinionated stand backed up by correct reasoning?

Re: DevOps is broken

#108
post #17

I’ve long held this opinion but I consistently get drowned out. DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways. All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be. There is also a lot of rewriting of history that gets…

>> I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.

This makes a lot of sense, but you need to realize that this is only a tiny part of a successful organization. There are many setups like that which you describe across many industries that have failed reasons beyond the role definitions. It's not enough to establish the organization. You have to keep the behaviors in check over time when you hire a build engineer with different aspirations.

Re: DevOps is broken

#109
post #93
post #70

> When was the last time you saw a product manager high-five the ops person and say, ‘fast fucking autoscaler!’? That's me. I did that.

And I know it's somewhat exaggerated but if the product people don't have visibility into your wins as an ops person then you need to make a more conscious effort to talk about them. Once you peel away the jargon, and tbf doing this is a skill that requires practice, non-techies do think ops work is cool.

Disclaimer: I am an Ops engineer turned product manager.

But I don’t like this rift. There are many ops people having a good understanding of product and vice versa. It’s just the stereotype which gets reproduced all the time, including in this blog post.

Sure, if management doesn’t get that there is value (and cost!) in proper ops work and having product management attached to it, then all is turned to “DevOps” anyways

Re: DevOps is broken

#110
post #43
post #17

I’ve long held this opinion but I consistently get drowned out. DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways. All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be. There is also a lot of rewriting of history that gets…

Agreed, anytime I talk with someone about DevOps ... we end up having to hash out the entire process to really know what either of us are actually talking about. Otherwise you have these situations "Yea the DevOps guy messed up the widget and nobody notic---" "Wait, what is the DevOps guy doing even touching that widget.... what is even DevOps to you?" "Bro that widget IS DevOps." -silence- Same applies to the topic…

SOA is just Spaghetti Architecture. I haven't seen an implementation that convinces me otherwise.
Post reply on HN