As someone who has lead a devops team, and still has a devops team reporting to me, I have also struggled with the idea of having a team called devops, since it does go against the devops philosophy of devs owning their software in production. However, in our case (and probably many others), our devops team is responsible for creating and maintaining the tools and processes that allow the dev teams to manage their sy…
Devops is having an identity crisis, but what you describe is what I call devops. It's the subset of engineering that supports engineering by defining/inferring workflows and building systems and tools to codify them. But I've seen it called ops, sysops, internal platform, and even infrastructure -- in some companies, devops and infra are the same people.
The Big DevOps Misunderstanding
201–210 of 321 posts
Re: The Big DevOps Misunderstanding
#202Earlier quoted context omitted.
So this, but instead of starting with Sysadmins with tens of years of unix experience, start with junior developers with 1 year of experience using Docker as a developer and have them read a blog post on how-to set up Kubernetes.
I really wish this wasn't the case but after sitting on the other side of the interview table I'll take the junior developer almost every time. All the best ops people start as devs with an interest in infra. The worst interviews I've ever been in have been sysadmins trying to pivot to infra work. You can't teach the tinkerer mentality or the culture of "I'll just look at the source and fix the problem."
Re: The Big DevOps Misunderstanding
#203I think that the op is missing a side. The article is about dev -> ops. The other side is ops->dev. I.e. since infrastructure is software, ops become dev (and not just dev become ops).
has ops ever not been dev? Atleast in my experience (ISP/DC world) most operational people where expected to do troubleshooting which at some points, results in one having to program to get to the nitty gritty efficiently. At a certain scale, issues start to creep up that are hard if not impossible to troubleshoot without atleast some programming and CS experience. Are operational people developers? Not in the tradit…
It's one of the biggest differences I see day to day in enterprise operations and developer mindsets. Developers generally do not worry about who will support code before they write it, because their entire job is to write code. Operations teams on the other hand want to find an off the shelf tool, with enterprise support, even for what could be a fairly simple task.
Re: The Big DevOps Misunderstanding
#204I'm frankly astounded at the complexity around modern development due to dev ops and how much worse developer experience is because of it. I used to develop in a great IDE with debugging, right click re-run failed tests, I could follow the console right there in a nice, clean window integrated with my IDE, click on stack traces, etc. Now I'm running my app via multiple docker images, trying to get a buggy remote debu…
devops was able to convince the industry that "software engineers own their own infra" is a viable thing to do so they could do engineering work, expect, they're out of their element. That's not their job. devops should be pulling in open-source technology that's developed by real engineers and write tiny packets of glue code to make it work to add value.
Never mind devops teams, I remember when my team had a _BUILD_ team. Serious companies still do this, engineers don't touch production. devops and sysadmins make sure that the business software stays operational, they could care less what it is. their mandate is to make sure it's operational and they hold engineers accountable.
k8s is a terrible experience and a terrible abstraction. I'm not smart enough to do better but I'm smart enough to know it terrible. It's snake-oil. My team is now migrating to the cloud, along with of course k8s, and guess what? the k8s env. in the cloud are to be managed and owned by the software engineering teams, because devops doesn't want to do their job. What a disaster.
at my company, a bunch of devops folks that deployed k8s 80% of the way left soon later to make 2x the pay as contractors at other companies to help them push through their last 80%. This is another reason that there is a lack of parity on the new infrastructure. Unfortunately, everyone is still stuck at 80%. Strange.
Re: The Big DevOps Misunderstanding
#205Earlier quoted context omitted.
In my experience, its because the tools that are given to the dev team are opaque, poorly documented, made for a general case that rarely is sufficient, and usually locked down with some role based access control to the point where, even if I can figure out whats wrong with some pipeline, I rarely have the access to change something. I have an AWS Codepipeline that I can see running, but the logs from each step are s…
I think the above is what happens when a devops tools team doesn't do discovery and user interviews. We've been at a point of software and development where we can build pretty much anything. But people are still making the same mistakes in figuring out what we should build. Fix the cause, not the symptom. Edit : And because I know it's coming... if IT security (or insert other org here) is the reason things can't be…
The same thing with a single DBA or whatever admin team. My dream team structure is a task-force like one. Each team has its own DevOps/DBA. They don't have to be dedicated roles, but someone in the team needs to be well-trained on those topics.
But I guess this model never scales in larger corporations so eventually they all adopt the One X-admin team model.
Re: The Big DevOps Misunderstanding
#206You can stop right there. There is no easy way. That is the DevOps misunderstanding. To understand why, look at your car.
Your car is a complex system that appears simple. You just purchase it, put gas in it, turn a key, shift a gear, press a pedal, and it goes. And goes, and goes, and goes. But over time the tires will go bald, the brakes will wear out, you will run out of coolant and brake fluid, the engine oil will turn to sludge, and the car will break down. Past those inevitable faults, some parts will randomly fail and require troubleshooting and replacement. You have two choices: either become a mechanic, or pay for one. (Or I guess. run it into the ground and buy a new one...) Computer systems are the same.
Say you just want to "buy and drive". Ok, so you buy a PaaS. You still have to learn how to drive it. And they are only easy to drive up til the point they break down and need more maintenance. And that's just driving it...
Now say you're in auto racing (building your own high performance, high reliability system). Someone has to build the car, someone has to maintain it, and someone has to drive it. You can't build a race car without knowing how to change a tire; otherwise you could build a car that runs like shit on the track. But mechanical engineers don't generally race cars, so they hand it over to mechanics and drivers to deal with running it on the track. And in doing so, they teach the mechanics and drivers all the insides and outs of the car. If they didn't, the car would never leave the garage, much less win a race. Imagine a random Ford mechanic or your cousin Bobby at the helm of an F1 car.
People who build systems should know how to run them. However, if people building systems don't know how to run them, they must teach whoever is running them how it was built and how it works inside and out. If anything significant changes, the mechanics need to know. And if something fails on the track, or even some unexpected brake fade, the engineers need to know so they can redesign their parts.
NASCAR, Formula 1, Le Mans, etc would be completely impossible otherwise. When the cars fail, the mechanics would be constantly shouting at the engineers pulling teeth trying to figure out how to fix the car in the middle of the race. There is no time for that madness. Toyota also builds their cars by having a continuous improvement cycle between engineers, the people assembling the cars, and service technicians. Toyota has basically been doing DevOps for 70 years (if you don't know the TPS system, you don't know dick about DevOps).
When I was fixing my brakes the other day, I ran into eight different operations I need to do properly. I needed to use a breaker bar and torch to unseat stuck bolts, because banging a wrench with a hammer strips bolts (guess how I know). I needed to use a custom tool to push the caliper piston back for the rear piston, because it requires twisting to reseat rear pistons. I needed anti-seize put on the rotor and wheel hub surfaces to prevent them sticking the next time I remove them. I needed to remove the caliper slide pins, clean them of dirt, and re-grease them, to prevent uneven/premature brake failure. I needed to grease the brake pad slides as well as the caliper contact points (without getting any on the pad material). I needed to torque the bolts to the right specification (the front and rear bolts take different torque). I needed to bleed the brakes with a custom appliance to remove air bubbles from the lines. And I needed to properly bed the brakes. All of those operations are necessary to prevent failure. But if I'd never learned about all the failure modes, I would have just slapped it all together and the brakes probably would have failed soon after. (The brakes will still fail eventually, I just control how soon it happens)
There is no getting around the knowledge gap. There is no easy way. If you're not all on the same page, you can ship a new car, but it's likely to be a lemon.
Re: The Big DevOps Misunderstanding
#207Earlier quoted context omitted.
> Personally I think it's pretty great that I can write Dockerfiles, run them locally, and call it the day because this is also what runs in production. It goes far, far beyond Dockerfiles. Are you on-call for production? Can you debug a problem in production? If not, then that's the root of the problem that the dude is talking about. If these are true for you (not specifically you, any reader of this): - You don't h…
I'm glad there's a dichotomy. I (a dev) never signed up to be on call 24/7 for production and it's ridiculous to just assume that I should be. I have a life outside of my work. If the system needs 24/7 uptime, they better be paying someone other than me to ensure that's the case, because my time is too valuable to waste my already precious freetime on fixing bugs in prod.
Re: The Big DevOps Misunderstanding
#208Kubernetes is a cancer for 99% of companies. Most companies need little more than docker compose/swarm. Kubernetes is the antithesis to KISS. If you’re Google, then you need something like K8s. Most orgs using it are pretenders or trend followers. Sadly it has become the “standard” way to deploy apps.
100% this, same for microservices. One of the most damaging belief a developer can have is "if my stack looks like * insert unicorn *'s stack, we will be a unicorn"
Microservices & Kubernetes are poster children for:
Re: The Big DevOps Misunderstanding
#209mashing together devs and ops inevitably results in schisms of devsecops and netops and ^ops because developers have enough shit to worry about on the daily without needing to learn the absolute intricacies of the OS as it applies to dev and prod. employers tried to get them around this by insisting "move fast and break things" with the push-to-prod mentality only to realize a 7 hour outage as a result kindof dims the outlook for everyone, shareholders included.
so employers wound up embracing the buzz and holding the line where it made sense, with some devops as developers, other devops as mostly ops, and still more devops as network-centric roles that handle things like firewalls and traffic too. now, people are going to point to automation and say "infra as code means devops is still a departure from traditional ^ops" but heres the secret: shit like cfengine has been around since 1993 and grandstand conferences on devops didnt show up until nearly 16 years later, and hyperconvergence showed up in 2009 which basically expands on software defined network and infra, so a cogent argument can be made that automation/convergence alone isnt devops, it was always happening anyways...so theres the wizard behind the curtain:
devops was just a trick to get you to do more so your boss didnt have to hire network and ops.
Re: The Big DevOps Misunderstanding
#210Earlier quoted context omitted.
To adapt Greenspun's tenth rule: You have a dedicated team building an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Google App Engine (or Heroku, or Digital Ocean App Platform, etc).
I know my company is likely not the norm, but we can’t use Google App Engine or anything like that because we are a CDN. Our infrastructure is core to our product. I know this might be unreasonable, but I always get frustrated when everyone suggests that anyone running their own infrastructure is clearly doing it wrong. I get that most dev groups are creating web apps and should use a PaaS, but there are also a lot o…