Live data from Hacker News

The Big DevOps Misunderstanding

wolfoliver.medium.com

181–190 of 321 posts

Re: The Big DevOps Misunderstanding

#181
working in a corp (10k++ employee) where everyone is specialized and outdated with the current tech scene. it's difficult to move sysops team to devops because

1. old mindset

2. outdated technologies

3. learning capabilities. devops approach is more friendly for those with dev background but not ops background.

everything about devops screams developers. it's like it was built for developers only if you think about it

- immutable?

- reproducible?

- faster loop feedback?

- testability (with cdk)? writing test units?

all these sounds foreign to sysops people from where i work.

Re: The Big DevOps Misunderstanding

#182

As someone who has lead a devops team, and still has a devops team reporting to me, I have also struggled with the idea of having a team called devops, since it does go against the devops philosophy of devs owning their software in production. However, in our case (and probably many others), our devops team is responsible for creating and maintaining the tools and processes that allow the dev teams to manage their sy…

> You might say this is just sysops, but I think the key difference is that the dev teams are the ones who are using the production management tools, not the sysops team.

FYI, that is sysops, in case that you missed it

Re: The Big DevOps Misunderstanding

#183

Earlier quoted context omitted.

Being a devops guy these last 2 years I have noticed that besides doing what you just mentioned, I am basically end user support for developers. And through this I've noticed that developers can be just as bad at communicating issues as any end user I've ever dealt with. Just because they've written the system end users use, does not mean they're better at reporting problems to me, because their problems are about th…

In my experience, its because the tools that are given to the dev team are opaque, poorly documented, made for a general case that rarely is sufficient, and usually locked down with some role based access control to the point where, even if I can figure out whats wrong with some pipeline, I rarely have the access to change something. I have an AWS Codepipeline that I can see running, but the logs from each step are s…

Yeah, one of the reasons I left a previous team was the company's attempt at embracing this. The standards were set down from a central platform team, the access control was held by department specific operations team, but developers were now "responsible" for the production deployments but granted neither the agency or access to do shit about it. So being paged about an issue with a load balancer setup that is understand by 2 guys in the platform team and configured by a different guy and having him have to send you screenshots of the config while digging in the source of the platform team's tool because of course there's no docs.

Re: The Big DevOps Misunderstanding

#184

Earlier quoted context omitted.

Being a devops guy these last 2 years I have noticed that besides doing what you just mentioned, I am basically end user support for developers. And through this I've noticed that developers can be just as bad at communicating issues as any end user I've ever dealt with. Just because they've written the system end users use, does not mean they're better at reporting problems to me, because their problems are about th…

> I am basically end user support for developers. Pull up a seat at the bar with us.

And drink whenever the dev works out that it was their fault the entire time.

Re: The Big DevOps Misunderstanding

#185
post #62

As someone who has lead a devops team, and still has a devops team reporting to me, I have also struggled with the idea of having a team called devops, since it does go against the devops philosophy of devs owning their software in production. However, in our case (and probably many others), our devops team is responsible for creating and maintaining the tools and processes that allow the dev teams to manage their sy…

Devops is having an identity crisis, but what you describe is what I call devops. It's the subset of engineering that supports engineering by defining/inferring workflows and building systems and tools to codify them. But I've seen it called ops, sysops, internal platform, and even infrastructure -- in some companies, devops and infra are the same people.

You're the librarians of the dev world by this definition. That's interesting to think about; that's what good academic librarians do. They also have a very similar identity crisis.

Re: The Big DevOps Misunderstanding

#186
post #132

Earlier quoted context omitted.

I've been in two production environments where IaC was done with Turing-complete languages (Google and Facebook). It's really not better (it just lets the truly impossible/unmanageable become possible) because they were both dynamically typed. If anything, Terraform comes the closest to being a good IaC language because providers form an API to production resources as opposed to unstructured code/config like Helm or…

Cuelang is a nice compromise: https://cuelang.org Composition over encapsulation, expressive enough for complex configs, not expressive enough to be turing complete.

+1. Cue is actually very nice. I wish there's more adoption of Cue instead of most other config languages. Config languages is one of those fields where the problem looks easy and the barrier for entry is low enough that we have many different designs where they don't provide meaningful enough differences, yet the actual problem is quite difficult and subtle. Cue is one of few that has a clean theoretical foundation and looks very promising. Too bad we have YAMLs and friends everywhere.

Re: The Big DevOps Misunderstanding

#187
post #155

Earlier quoted context omitted.

I think the above is what happens when a devops tools team doesn't do discovery and user interviews. We've been at a point of software and development where we can build pretty much anything. But people are still making the same mistakes in figuring out what we should build. Fix the cause, not the symptom. Edit : And because I know it's coming... if IT security (or insert other org here) is the reason things can't be…

I understand where you're coming from, but I don't think you really need "discovery and user interviews" to tell you that app developers need to be able to see their deployment logs, or that having your deployment code spread out across 3-4 repos with frequent changes makes it harder to troubleshoot. I think a lot of people might read this comment as blameshifting, where obvious flaws in the engineering and reliabili…

Here's the reason, 9 out of 10 times, this situation persists -- no one realizes it's happening.

No one realizes some dev teams can't see their own deployment logs. No one realizes teams don't have the correct Jenkins access. No one realizes there's a mysterious gatekeeper to use the corporate time series database as a service. No one realizes all new containers are transferred to a few people who have the rights to publish them.

And when I say "no one realizes", I mean two things. One, the people who do know, because they're the ones trying to do it, have given up bitching about it, because they did for years and nothing happened. Two, the people who can actually change things don't know, either because they actively ignored / forgot or because the requests never reached them (usually because a middle transport manager didn't understand what was actually being asked).

And so... the status quo prevails, and everyone has a vague sense that something isn't working, but doesn't know what to do about it.

And to that problem, the best solution I've found is to get the fixer and the problem user as close as possible, wherein the problem is usually revealed as a relatively simple change or feature oversight. And for that, discovery and user interviews are the most reliable method.

Re: The Big DevOps Misunderstanding

#188

We built the https://controlplane.com platform to let developers focus on the Dev and get best practices Ops so they can deliver many times faster. Our customers get 99.999% availability and ultra low latency no matter where users are. Check it out and I'd love to hear your feedback on the Ops part in DevOps with Control Plane.

Control Plane is awesome and as a former SRE I recommend it whole-heartedly.

Re: The Big DevOps Misunderstanding

#189

As someone who has lead a devops team, and still has a devops team reporting to me, I have also struggled with the idea of having a team called devops, since it does go against the devops philosophy of devs owning their software in production. However, in our case (and probably many others), our devops team is responsible for creating and maintaining the tools and processes that allow the dev teams to manage their sy…

The problem is what happens when there are niggling small problems. Do you do the work to handle the problem? If so dev gets a free pass to write unreliable code again. Next what about monitoring? Is it the same as dev environment or all stuff you wrote? How do you test it? So much duplication and overlap.

Re: The Big DevOps Misunderstanding

#190
Are you running your application on a UNIX system? (You probably are). Do you know your octal file permissions? Your various FS choices? Your reserved ports? Your inode limits? Your uid mapping? Your SSL root trust location? Your free-space limits? Your TCP handshake? Your TLS 1.1, 1.2, 1.3 handshake? Your HTTP verbs? Firewall rules? NAT configurations? Load Balancer health checks? JVM -~XmX values? Swap size? Database connection pools? LibC implementations? GNU vs BSD command arguments? cgroups? jails? eBPF? retpolines?

You're probably a developer, and these are concerns of a lower level. They are different skill sets, and pretending otherwise is marketing bullshit.

Post reply on HN