The Big DevOps Misunderstanding • Hacker News Board

Re: The Big DevOps Misunderstanding

#171

Post by ethbr0 » Mon, Dec 20, 2021, 1:29 AM UTC

2fast4you wrote: post #167

ethbr0 wrote: post #155

Earlier quoted context omitted.

I think the above is what happens when a devops tools team doesn't do discovery and user interviews. We've been at a point of software and development where we can build pretty much anything. But people are still making the same mistakes in figuring out what we should build. Fix the cause, not the symptom. Edit : And because I know it's coming... if IT security (or insert other org here) is the reason things can't be…

Do you have good resource in how to discovery/user interviews? I’m on a DevOps team working to mature the first few tools we made

I'd look at UX books. They've got more formal approaches.

Personally? Make a list of all the teams that use your product. Schedule interviews (30 minutes to an hour for initial) with a few that use it the most, and a few that use it less. Ask them what the best and worst parts are. Ask them details about the things they bring up, and prompt about related areas they might be struggling with too.

If you get any repeats from team to team, assume those are systemic issues and try and find a solution.

The hard part usually isn't figuring out the nuances, but rather realizing that someone wants to do something at all and/or can't because of some restriction. Assume you know nothing about how they actually do what they do and listen closely.

Re: The Big DevOps Misunderstanding

#172

Post by Pxtl » Mon, Dec 20, 2021, 1:30 AM UTC

INTPenis wrote: post #103

cortesoft wrote: post #17

As someone who has lead a devops team, and still has a devops team reporting to me, I have also struggled with the idea of having a team called devops, since it does go against the devops philosophy of devs owning their software in production. However, in our case (and probably many others), our devops team is responsible for creating and maintaining the tools and processes that allow the dev teams to manage their sy…

Being a devops guy these last 2 years I have noticed that besides doing what you just mentioned, I am basically end user support for developers. And through this I've noticed that developers can be just as bad at communicating issues as any end user I've ever dealt with. Just because they've written the system end users use, does not mean they're better at reporting problems to me, because their problems are about th…

> And through this I've noticed that developers can be just as bad at communicating issues

I swear to god the next guy who sends me the screenshot of an error log...

Re: The Big DevOps Misunderstanding

#173

Post by nightpool » Mon, Dec 20, 2021, 1:37 AM UTC

ethbr0 wrote: post #155

capn_duck wrote: post #152

Earlier quoted context omitted.

In my experience, its because the tools that are given to the dev team are opaque, poorly documented, made for a general case that rarely is sufficient, and usually locked down with some role based access control to the point where, even if I can figure out whats wrong with some pipeline, I rarely have the access to change something. I have an AWS Codepipeline that I can see running, but the logs from each step are s…

I think the above is what happens when a devops tools team doesn't do discovery and user interviews. We've been at a point of software and development where we can build pretty much anything. But people are still making the same mistakes in figuring out what we should build. Fix the cause, not the symptom. Edit : And because I know it's coming... if IT security (or insert other org here) is the reason things can't be…

I understand where you're coming from, but I don't think you really need "discovery and user interviews" to tell you that app developers need to be able to see their deployment logs, or that having your deployment code spread out across 3-4 repos with frequent changes makes it harder to troubleshoot. I think a lot of people might read this comment as blameshifting, where obvious flaws in the engineering and reliability of the system are blamed instead on a failure to define the specifications properly ahead of time.

Re: The Big DevOps Misunderstanding

#174

Post by smus » Mon, Dec 20, 2021, 1:40 AM UTC

quadrifoliate wrote: post #48

avip wrote: post #32

Personally I think it's pretty great that I can write Dockerfiles, run them locally, and call it the day because this is also what runs in production. So basically I have no clue what the dude is talking about - we have a great setup which is x10 better than what we had before. No?

> Personally I think it's pretty great that I can write Dockerfiles, run them locally, and call it the day because this is also what runs in production. It goes far, far beyond Dockerfiles. Are you on-call for production? Can you debug a problem in production? If not, then that's the root of the problem that the dude is talking about. If these are true for you (not specifically you, any reader of this): - You don't h…

I'm glad there's a dichotomy. I (a dev) never signed up to be on call 24/7 for production and it's ridiculous to just assume that I should be. I have a life outside of my work. If the system needs 24/7 uptime, they better be paying someone other than me to ensure that's the case, because my time is too valuable to waste my already precious freetime on fixing bugs in prod.

Re: The Big DevOps Misunderstanding

#175

Post by datavirtue » Mon, Dec 20, 2021, 1:42 AM UTC

capn_duck wrote: post #152

INTPenis wrote: post #103

Earlier quoted context omitted.

Being a devops guy these last 2 years I have noticed that besides doing what you just mentioned, I am basically end user support for developers. And through this I've noticed that developers can be just as bad at communicating issues as any end user I've ever dealt with. Just because they've written the system end users use, does not mean they're better at reporting problems to me, because their problems are about th…

In my experience, its because the tools that are given to the dev team are opaque, poorly documented, made for a general case that rarely is sufficient, and usually locked down with some role based access control to the point where, even if I can figure out whats wrong with some pipeline, I rarely have the access to change something. I have an AWS Codepipeline that I can see running, but the logs from each step are s…

Agreed. I approach shared services like a customer because that is all the power I have. It is supposed to work; that is all I know or have time to deal with. Off the the slack support channel...if you are real lucky.

Re: The Big DevOps Misunderstanding

#176

Post by datavirtue » Mon, Dec 20, 2021, 1:45 AM UTC

Pxtl wrote: post #172

INTPenis wrote: post #103

Earlier quoted context omitted.

Being a devops guy these last 2 years I have noticed that besides doing what you just mentioned, I am basically end user support for developers. And through this I've noticed that developers can be just as bad at communicating issues as any end user I've ever dealt with. Just because they've written the system end users use, does not mean they're better at reporting problems to me, because their problems are about th…

> And through this I've noticed that developers can be just as bad at communicating issues I swear to god the next guy who sends me the screenshot of an error log...

Blame management. They thought it would be a good idea to go into the DevOps PaaS business.

Re: The Big DevOps Misunderstanding

#177

Post by nickcox » Mon, Dec 20, 2021, 1:53 AM UTC

DyslexicAtheist wrote: post #150

shrimpx wrote: post #62

Earlier quoted context omitted.

Devops is having an identity crisis, but what you describe is what I call devops. It's the subset of engineering that supports engineering by defining/inferring workflows and building systems and tools to codify them. But I've seen it called ops, sysops, internal platform, and even infrastructure -- in some companies, devops and infra are the same people.

> it's the subset of engineering that supports engineering only it's not what the idea behind it was. It was exactly the opposite of another compartmentalization. it was supposed to bridge silos not carve out new job titles for quality control

> only it's not what the idea behind it was

It's like the article says:

> At some point in time, this idea has been so widely misunderstood that the wrong definition of DevOps became the right one.

Re: The Big DevOps Misunderstanding

#178

Post by datavirtue » Mon, Dec 20, 2021, 1:56 AM UTC

cortesoft wrote: post #143

Fiahil wrote: post #136

Earlier quoted context omitted.

I worked with a client and a devops team before, I still thinks having a separate devops team is not the right approach. In my case, the team « responsible for creating and maintaining the tools and processes that allow the dev teams to manage their systems in production », was just a regular OPS team. So they built tools that were not working properly and they struggled to focus on the most valuable parts because th…

This makes sense for a smaller org, but what do you do when there are dozens of dev teams? You can try to make every team a full stack team and have a devops engineer embedded with every team, but you will end up with a different process and set of tools for every team. I feel like having a dedicated devops (or tools or platform or whatever you want to call them) is needed in a large org.

"but you will end up with a different process and set of tools for every team"

This keeps being trotted out, without question, as the reason for the shared devops platform team. So what if each team has their own opinionated build, scan, deploy tooling?

It's used for compliance security scans now, so we are never going back.

Re: The Big DevOps Misunderstanding

#179

Post by aduitsis » Mon, Dec 20, 2021, 2:09 AM UTC

kazen44 wrote: post #122

geodel wrote: post #50

Earlier quoted context omitted.

Now I feel this is how it is exactly happening at work even though I am not part of "devops" team. Any working solution at my place is now legacy ready to be dumped and replaced with next generation stack. Other day "devops" engineer couldn't figure out wrong directory permissions and whole deployment failed.

we once had this issue at an old job. People wanted to replace a monitoring solution for something new, because snmp was "bad and outdated". SNMP is definitely not the holy grail of solutions (to be honest, its kind of a mess). But atleast SNMP is widly supported, works reasonably well, and gets you the data in a dashboard you need. Even for those old Cisco 7200's with an uptime of over a decade. Being new and shiny…

Thank you for the post! At some point years ago I started realising that most problems in SNMP are stemming from badly implemented SNMP agents. Those problems can be:

1. Returning values different than what the MIB says.

2. Missing values that are not optional in the MIB.

3. Broken getnext code that returns an incorrect oid, which could even lead to loops. Net-snmp will detect it, but proper snmpwalk gets broken.

4. Broken getbulk repeater code that can cause crashes if the repeater is too large.

5. OIDs that will crash/stall/slow the agent.

6. Implementations that cannot handle simultaneous requests. Admittedly this is notoriously difficult to achieve: most MIB implementation code is not reentrant for various reasons.

7. SNMP SET is very difficult to implement, cannot be meaningfully used for almost anything beyond extremely simple stuff.

8. UDP encapsulation can cause MTU/fragmentation issues that are a nightmare to debug. Net-snmp actually implemented transports such as TCP and it was very nice, but it never caught on in general.

Bonus, there is also the not related to implementation problem of:

9. The query semantics of SNMP are non-existent, and the indirection in the SMI is cumbersome, which means that as a client/manager you'll have to do all sorts of data wrangling to find what you need.

Re: The Big DevOps Misunderstanding

#180

Post by VMtest » Mon, Dec 20, 2021, 2:15 AM UTC

The comments are really a mess

Some people are suggesting to change how DevOps works by going against the definiton of DevOps

In that case, please don't put DevOps in the engineering title anymore