Live data from Hacker News

Software Developers Should Have Sysadmin Experience

blog.professorbeekums.com

271–280 of 424 posts

Re: Software Developers Should Have Sysadmin Experience

#271
post #34

This is a short little article that just barely touches on a much deeper, often hidden issue; The state of system administration in business is abysmal. It's not the developers fault though, at least not as much as devops types would like you to believe. I think the author has a good point, in that its good to get Devs thinking about real world environments on deploy, but the real world is much more complex than conc…

I agree that the dialectic shouldn't be played...but not that it's "cross-training." Rather, it's the same skillsets being applied in slightly different ways. There was a day in which "sysadmin" very often meant "shit-hot Perl slinger." It was before my day, but I know some of the graybeards who can still lay claim to it. The sysadmin who can't write good, maintainable code is going to rapidly see their positions red…

> The sysadmin who can't write good, maintainable code is going to rapidly see their positions reduced to sinecures in large and slow companies.

I completely disagree, but respectfully. What I think we are lacking in this discussion is a differentiation between what we mean by sysadmin as a product and as a job description. Systems Administration is a aggregate management of the technical infrastructure. (for $reasons).

In super small technical infrastructure, such as webdev startups, there is very little active sysadmining to do if done properly, but as complexities grow, you need multiple people to perform all the various duties needed to maintain a system. In the performance of these duties, there are various descriptions with varying levels of requirements.

What I postulate is that traditional businesses in seeing the agility of the dotcom and startup culture have been attempting to cut costs in the technical infrastructure, but for the most part because the managers of the technical infrastructure (the sysadmins) haven't had anyone to advocate on their behalf, this has resulted in poor business performance bottlenecks.

I would be curious to hear what other have to say, but in the "shit-hot Perl slinger." days, usually the senior sysadmin was the slinger, and he reported directly to the CFO and CEO, if not ownership.

What I argue is that systems have gained in complexity from a computing infrastructure standpoint such that this old structure no longer worked well, hence the creation of the CTO/CIO class(very similar). The problem is, they aren't doing a good job in my opinion. Therefore, in this current business climate, my argument is that the main problem is we already have good code slinging sysadmins on hand (or can train them), but what businesses need/the industry is lacking, is business/political game aware senior sysadmins to make up for that failing of the C's. Hence I disagree the days of non-code slinging sysadmins are going anywhere. Indeed, some of the best senior sysadmins I know live in meetings, but if they are performing well in fulfilling that role, and not slinging code, but instead making big picture, high-level overview decisions and then monitoring progress, I don't think there is anything wrong with that. Now, if we got CIO/CTOs back on the right track, the political/business game demand could fall and those sysadmins could get back into hard work, (which just happens to sometimes involve code-slinging to solve problems.) Devs are an entirely seperate entity, but still part of the process, in anything other than a pure web startup type businesses with little to no real infrastructure (eg, workfrom home contractors).

Of course I want to qualify this in that this is ancedotal and I admit I haven't seen every environment, but I have seen many environments (as a contractor, seeing more insides than the guy going for retirement), from fortune 500s to 2 man lawyer shops.

Any business that can see this issue coming and address it head on will be far ahead of the game. Those that don't, will one day have very rude wakeup calls as the complexity exponentially increases and they don't have the structures in place the handle the demand, mostly due to lack of foresight/vision.

Re: Software Developers Should Have Sysadmin Experience

#272
post #74
post #50

This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. And I've never thrown code over a wall without having tested it…

I've done the dual sysadmin/developer thing for a small company, and the problem I experienced there was completely incompatible working modes. Sysadmins must deal with interrupts (requests, crises, things driven by external schedules etc) and then in the rest of their time build systems to manage or reduce the interrupts. Developers are expected to produce work on a predictable schedule. This is disrupted by interru…

that's a great point about the types of schedules for each discipline.

Re: Software Developers Should Have Sysadmin Experience

#273

Earlier quoted context omitted.

Counter-wit: Why can't people get their software to work with the existing libs? Hint: It's very rarely that the existing libs disqualify.

this is the core of the problem between Devs and sysadmins. Sysadmins come from a mindset of a polished working system which never needs to change. They deliver stability and reliability to the business. Devs come from a mindset to actively create change. This is to add new features and deliver new value and product to the business. As a Dev I do have to say that many Devs don't have enough experience in operations t…

> To respond directly to your question: because there are new things available in new libraries that allow us to develop new features!

If it were only that, we would have an easy time. The new things you need to develop new features are far and far between.

Re: Software Developers Should Have Sysadmin Experience

#275
sysadmin really means "technical debt manager" most of the time these days.

I really spend the majority of my time trying to slot something into place in a way that no one will notice that doesn't disturb existing stuff...often that means duplicating bad behavior because its become a dependency.

Re: Software Developers Should Have Sysadmin Experience

#276

Earlier quoted context omitted.

> They aren't kids playing with toys that we need to nanny over In my experience (university), yes they are, and they should do that at home. Why do you need the latest bleeding versions in the first place? In my sysadmin experience, people believe software gets bad and deprecated as soon as the glory next breaking version appears. I don't think I need to argue why this is an illogical stance. With my developers hat…

I think the pain of keeping any project fully up-to-date is (sometimes far) less than the pain of updating an out-of-date project.

If it is, your project has a problem.

Re: Software Developers Should Have Sysadmin Experience

#277

Earlier quoted context omitted.

Another perspective: Why don't you want your software to be compatible with the system that your sysadmin provides? (Assuming that system is not completely obsolete). Minimize your dependencies. It's incidentally also what leads to clean code bases.

> Minimize your dependencies. It's incidentally also what leads to clean code bases. You also crush velocity. Smart use of libraries lets you ship code 10x faster. Two identical businesses.. one writes all their own code, one is smart about using libraries. Which one makes it to IPO first, and which goes bankrupt?

That only holds true (as much as it does) if you assume the goal is always an IPO.

Re: Software Developers Should Have Sysadmin Experience

#278
post #50

This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. And I've never thrown code over a wall without having tested it…

>Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them. This is not personal criticism but you know how I know you're not working in a highly regulated environment? Check out the Carnegie Mellon Ca…

It's funny that you bring up CMMI in the discussion. CMM(I) is nearly the antithesis of Deming's approaches toward quality management. What's really interesting is that Deming's approaches were adopted by Toyota decades ago to historically great effect (sadly with few other large examples of named successes in the business world) while Taylorist approaches (including CMMI) a handful of other Japanese daibatsu and especially US companies back to the 19th century. These companies have had vastly different growth trajectories over time, but when it comes to quality most consumers in surveys will associate Toyota with it over Hitachi, Mitsubishi, and Fujitsu (I believe all of these companies are full-blown CMMI 2.0+ adherents and champions). Similarly in the US, what has Six Sigma really done for companies that have adopted it? GE is hardly known for anything in the public eye resembling technical chops, for example, and most studies show that companies that adopt Six Sigma more than 70%+ of the time lag the S&P 500 upon adoption with no long-term recovery afterward either (perhaps Six Sigma adoption is not a cause but a simple correlation with poor performance similar to private equity oftentimes getting a bad rap in the public eye).

When even the US military - one of the world's foremost investors in management and leadership research - has largely abandoned command and control (the military equivalent of Taylorism) we really need to ask whether structures that enforce a management/worker caste vs. one that empowers those closest to a problem are effective beyond any meaningful scale.

Re: Software Developers Should Have Sysadmin Experience

#279

Earlier quoted context omitted.

My comment to you two is -- why not both? Stability, sanity, all that is amazing, and a must have. But also bug-fixes, security improvements, and performance improvements are wonderful too, which tends to come with using up-to-date dependencies. The problem with the latter, as you mentioned, is when it introduces breaking API changes and is wholly not backwards compatible. This is not a "kids playing with toys wantin…

Backwards compatibility is seriously under appreciated. When I tell developers to ensure that their changes are backwards compatible, they tend to look at me like I'm green. I do not understand the disconnect that developers have with understanding all of the benefits that it brings. Yes, you have some extra code in your code base so it's less clean. You also have a stable environment as a result. The first affects o…

I do not understand the disconnect that developers have with understanding all of the benefits that it brings.

Because they've never worked on an old codebase, because front-end technologies change so often and everything just gets re-written anyway. It's a waste of time worrying about this when the code won't make it to is first birthday.

If you were speaking to seasoned C and DB developers about stability in the tools and the platform, you'd be preaching to the choir.

Post reply on HN