Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Big DevOps Misunderstanding

wolfoliver.medium.com

Post reply on HN
Print view View original

281–290 of 321 posts

Re: The Big DevOps Misunderstanding

#281

Post by hnthrowaway0315 »

liotier wrote: post #222
hnthrowaway0315 wrote: post #205

Earlier quoted context omitted.

I actually think that it's an issue of company structure. Once you concentrate ALL deployment/operational power to a single DevOps team, you really lose the power to touch individual teams and make human budgets clearly. This is more of a "cost-center" move that implicitly signals that company X needs a single platform and individual requirements are not going to be prioritized. The same thing with a single DBA or wh…

> My dream team structure is a task-force like one. Each team has its own DevOps/DBA. They don't have to be dedicated roles, but someone in the team needs to be well-trained on those topics. Looking back at the last twenty years, oscillating between central sysops and everyone doing their own thing (is that "devops" ?), I feel that an optimum requires both: local skills to iterate fast, central skills to consolidate.…

This is good point and definitely more doable when the company is already centralized.

Re: The Big DevOps Misunderstanding

#282

Post by throwdbaaway »

solatic wrote: post #248
pas wrote: post #234

Earlier quoted context omitted.

Letting/making devs own the services they deploy/develop is not the full picture, nor is it a real goal, but still it sums up the problem and solution space concisely. Yes, specialization is a must for productivity, but that's a given already in the old model. What the DevOps effort tries to cut through is the trust barrier between the two. Sure, it's communications engineering too, but it's easy to weasel out of any…

> Letting/making devs own the services they deploy/develop is not the full picture, nor is it a real goal, but still it sums up the problem and solution space concisely. What I'm trying to explain is, that I don't believe that this is a realistic expectation, because owning a service in production includes aspects of the stack that are not related to development, like networking, storage, and various non-dependency s…

I agree with the concern regarding leaky abstractions, but ultimately the developers should still own their services and be on-call for it, e.g.

- On-call dev deploys a buggy change that messes up some transactional data. Product owner passes the complaints received by customer service. On-call dev rolls back the deployment, writes a script to fix the bad data, and then prepares a post-mortem.

- On-call dev and on-call dba are paged because the database for the service is slow. On-call dev checks the service metrics and logs, and confirms that this is not caused by a deployment nor abnormal traffic. On-call dba checks the database metrics and logs, and finds some runaway queries due to silly optimizer. On-call dba kills the runaway queries, and proposes to add an index hint. On-call dev agrees, works on it, and asks on-call dba to review the change.

- On-call sec is notified because of a new rce vulnerability. On-call sec comes up with the steps to identify and remedy the issue. All on-call devs are notified to follow the steps, to ensure that their services are not impacted.

Re: The Big DevOps Misunderstanding

#283

Post by t_sawyer »

emptysongglass wrote: post #56

DevOps was always about de-siloing. In that, it is a culture but you need people to "be the change" and that's how I've played a DevOps Engineer for the last five years (well now SRE) and how I believe DevOps Engineers (and SREs by extension) ought to be playing their role today. Corporate management is always surprised when my first move is to go sit with the developers and ask how I can unblock their workflow, redu…

DevOps should be embedded on the team. Doing development and taking care of pipelines/iac when needed. If these people are not embedded on the team, it’s just more of the same with a new title.

I get paid a lot because I can write api endpoints, front end code, iac, pipelines, and understand networking enough to make our apps SOC2 or HIPAA compliant.

Re: The Big DevOps Misunderstanding

#284

Post by t_sawyer »

jchmbrln wrote: post #229
emptysongglass wrote: post #56

DevOps was always about de-siloing. In that, it is a culture but you need people to "be the change" and that's how I've played a DevOps Engineer for the last five years (well now SRE) and how I believe DevOps Engineers (and SREs by extension) ought to be playing their role today. Corporate management is always surprised when my first move is to go sit with the developers and ask how I can unblock their workflow, redu…

I think you've hit the nail on the head. In a previous job I tried to outsource "DevOps" to Rackspace, only to learn the hard way that if you can outsource it it's not DevOps—_increasing_ siloization is the antithesis of DevOps. But this is also the point where I think SRE is most compelling. Rather than trying to build an organization around "you build it, you run it", SRE recognizes the need for specialization whil…

I see the function of SRE as a place to pass off an app when an app team (DevOps included) need to develop another app.

SRE sets standards so a centralized team can manage the ongoing maintenance and on call break/fix of many apps in an organization while development teams move on to something else.

Re: The Big DevOps Misunderstanding

#285

Post by therealbilly »

I'm personally waiting for the whole DevOps thing to revert back to what we had before. I come from the mainframe world. In those days, there was real trust in the developers and we migrated our own code. We had testers and we couldn't move stuff until they verified our work. The intentions were good for implementing DevOps but I think it's gotten out of hand. It's just too complex. Just my opinion.

Re: The Big DevOps Misunderstanding

#286

Post by jeffisabelle »

recursivedoubts wrote: post #18

I'm frankly astounded at the complexity around modern development due to dev ops and how much worse developer experience is because of it. I used to develop in a great IDE with debugging, right click re-run failed tests, I could follow the console right there in a nice, clean window integrated with my IDE, click on stack traces, etc. Now I'm running my app via multiple docker images, trying to get a buggy remote debu…

I started my career as a backend developer and did a fair bit of frontend development as well. However, I'm solely doing infrastructure engineering for the last 6-7 years.

I think you are kind of mistaken here, as a _devops_ I never enforced our development teams to use docker, or another specific tooling. The thing is, software engineering become a whole lot difficult and complicated over the years. When I was first learning web-development 15 years ago, notepad was the only thing you needed to do web-development, now look what you need to have to have a simple CRUD app running locally. Setting up local environments become so much harder, so people found the answer within the containers as you set it up once and you can share it within the team easily. Please do not blame _devops_ for that.

You don't have to run your app via multiple docker images locally, you can still configure everything to work natively. It's just way more difficult because of the dependencies of modern technologies.

Re: The Big DevOps Misunderstanding

#287

Post by ethbr0 »

croon wrote: post #264
ethbr0 wrote: post #187

Earlier quoted context omitted.

Here's the reason, 9 out of 10 times, this situation persists -- no one realizes it's happening. No one realizes some dev teams can't see their own deployment logs. No one realizes teams don't have the correct Jenkins access. No one realizes there's a mysterious gatekeeper to use the corporate time series database as a service. No one realizes all new containers are transferred to a few people who have the rights to…

> And to that problem, the best solution I've found is to get the fixer and the problem user as close as possible You mean put development and operations together, into some kind of dev..ops.. :) I'm sorry for the snark, it was not meant as disagreement with what you said. The whole idea of the role was to try to remedy that exact issue, but old habits die hard and sometimes things just change names but stay the same…

devops is the new agile

Re: The Big DevOps Misunderstanding

#288

Post by ethbr0 »

hnthrowaway0315 wrote: post #281
liotier wrote: post #222

Earlier quoted context omitted.

> My dream team structure is a task-force like one. Each team has its own DevOps/DBA. They don't have to be dedicated roles, but someone in the team needs to be well-trained on those topics. Looking back at the last twenty years, oscillating between central sysops and everyone doing their own thing (is that "devops" ?), I feel that an optimum requires both: local skills to iterate fast, central skills to consolidate.…

This is good point and definitely more doable when the company is already centralized.

IMHO, the "centralized team" + "decentralized liasons" model isn't explicitly used nearly as often as it should be.

It usually happens in practice ("Sarah knows someone in security"), but there's no reason "devops liason" can't be an explicit 2nd+ hat for someone on a team to wear.

Centralized teams are necessary for budgetary reasons: one person on every team asking for a devops product will never make a coherent case to management for why it's needed, but a team will.

But having been on both sides of the equation, the central-decentralized relationship usually breaks down because there's always a new face on the right side of that equation, and that requires reteaching / recoordinating all the basics. More consistency on the user side (a hat) helps significantly.

And ultimately, it's a trust problem: do I (central person) trust this person who's asking me for something? If they're new to me... probably not very much. If I've been working with them for awhile... probably a lot more.

Re: The Big DevOps Misunderstanding

#289

Post by cortesoft »

xxpor wrote: post #200
cortesoft wrote: post #88

Earlier quoted context omitted.

Yeah, I am not sure how typical this is, but my company has DC (data center) ops, NetOps, SysOps, and DevOps… DC ops is in charge of the physical system, from racking and stacking to hardware replacements. NetOps handles the networking stack, managing the routers and switches, vlans, announcements, IP block management, network access requests, etc. SysOps handles provisioning systems, working with dc ops to get syste…

Who's clicking the button that says deploy this service to production?

The devs do. We have a system that slowly releases software to production in stages, and devs watch their deployment and a-b metrics to make sure the deployment is working.

Re: The Big DevOps Misunderstanding

#290

Post by [deleted] »

jordanbeiber wrote: post #94
thrower123 wrote: post #78

Earlier quoted context omitted.

Infrastructure in enterprises is usually completely uninteresting, to be fair. You need like eight different people on a call in order to stand up a VM, install a service, and configure a database connection.

Broken processes and missing self-service, as you point out, is why creating a platform team is important for these types of places. They might not know it yet, although some are trying. It doesn’t make the infra technology more or less interesting?

[deleted]
Post reply on HN