Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software Developers Should Have Sysadmin Experience

blog.professorbeekums.com

Post reply on HN
Print view View original

311–320 of 424 posts

Re: Software Developers Should Have Sysadmin Experience

#311

Post by crdoconnor »

xorcist wrote: post #203
brianwawok wrote: post #184

Earlier quoted context omitted.

That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading? I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in sett…

> why should a sysadmin who doesn't even know Python Because a Python sysadmin has been through all the transitions of packaging systems, all the nasty corners of "backwards compatible" changes, and knows how underlying changes to the operating system will affect your code, what the storage behaves like under load, and why one tech is not "better" than another. If you really hired an admin (cough, "reliability engine…

>You don't know what you don't know.

That's the point of regression tests. The sysadmin also doesn't know. Unless he's the one writing the tests (and IME he's not) or he's painstakingly regression testing everything by hand (trust me, he's not doing that either), making him a gatekeeper for all library upgrades achieves very little except adding bureaucratic friction.

Re: Software Developers Should Have Sysadmin Experience

#312

Post by crdoconnor »

warbiscuit wrote: post #300
xorcist wrote: post #203

Earlier quoted context omitted.

> why should a sysadmin who doesn't even know Python Because a Python sysadmin has been through all the transitions of packaging systems, all the nasty corners of "backwards compatible" changes, and knows how underlying changes to the operating system will affect your code, what the storage behaves like under load, and why one tech is not "better" than another. If you really hired an admin (cough, "reliability engine…

I think a lot of this debate argues for the sysadmin role being part of the dev team. The only real way to get both constraints (production stability and update to date fixes/features) is to have fast feedback between the interest holders of two sides. In the python-specific case -- the requirements.in / .txt files for the virtualenv should be part of the software VCS, but the sysadmin should be able to edit & pin th…

IMO if a sysadmin wants to have visibility into the requirements.txt that's fine.

If they want to enforce a policy of pinning versions, that's very welcome (though I would do that anyway).

If they have specific, relevant comments about upgrades of specific packages - again, fine (though in practice they never do).

If they want to be a gatekeeper for changes to that file they can fuck off.

Re: Software Developers Should Have Sysadmin Experience

#313

Post by ihattendorf »

protomyth wrote: post #283

Actually, I would rather language and tool designers have a go at a true System Admin job. There is a reason PHP gets installed by default and Ruby on Rails does not. Frankly, at the end of the day, most businesses don't want something that was "working" breaking. All the conflict comes from that one desire. There is a reason System Admins have to do the "Patch Tuesday" dance and despise any software that just update…

True for the most part, however PHP is a language while Ruby on Rails is a framework.

Re: Software Developers Should Have Sysadmin Experience

#314

Post by eropple »

eeZah7Ux wrote: post #102
bthornbury wrote: post #3

With the rise of devops (docker, puppet, ansible, etc...) , and cloud VMs, how often are sysadmins different from software devs?

With the rise of devops system engineering knowledge is being lost. I often see developers trying to replace it with bloated, complex piles of hipster software. While a devops team is attempting to build a toy Google-like infrastructure on 50 nodes for weeks, another company is deploying code just fine using some 15 years old setup with netboot + disk imaging system + OS packages.

I don't think this is true at all. Which is not to defend or even endorse the Kubernetes wank, "devops" is also "hey, we have codified what our systems look like in code (with Chef/CloudFormation/etc.)." Which is a hell of a step up from what existed prior. That I occasionally have to then go debug why AWS's Xen network drivers are spitting the bit or why this application's disk access patterns are making EBS sad certainly keeps me very firmly in the details of the systems in question, too.

(As an aside: I mentally replace the words "hipster software" whenever I see it with "things I don't understand" and the sentences never seem to change their meaning.)

Re: Software Developers Should Have Sysadmin Experience

#315

Post by prodigal_erik »

bandrami wrote: post #303
gaius wrote: post #279

Earlier quoted context omitted.

I do not understand the disconnect that developers have with understanding all of the benefits that it brings. Because they've never worked on an old codebase, because front-end technologies change so often and everything just gets re-written anyway. It's a waste of time worrying about this when the code won't make it to is first birthday. If you were speaking to seasoned C and DB developers about stability in the to…

This gets to the complaint that so much of the open source ecosystem gets to version 0.8.6 (whether it's named that or not) and then completely rewritten "this time the right way". That's not actually a good thing.

As jwz put it,

> It hardly seems worth even having a bug system if the frequency of from-scratch rewrites always outstrips the pace of bug fixing. Why not be honest and resign yourself to the fact that version 0.8 is followed by version 0.8, which is then followed by version 0.8?

Re: Software Developers Should Have Sysadmin Experience

#316

Post by JamesBarney »

nokya wrote: post #37

I'd say there is too much effort in reasoning on the wrong problem. What worries me the most is the 'why': why do (too) many software developers don't know about sysadmin? I have been involved as a consultant in large software projects in the last two years and a vast majority of money lost in delays and bugs was caused by devs not understanding: 1) the difference between virtual memory and physical memory 2) the dif…

I've been a consultant on large projects and the vast majority of money that I've seen flushed down the toilet was on

1) Building the wrong piece of software/end-users not having enough influence on what get's built.

2) Lack of delegation, having people make technological/feature decisions about a product they only spend 5% of their time thinking about.

3) Organizational incentives not aligned correctly.

4) Not following software engineering principles that were discovered in the 70's(they also don't follow any software engineering principles discovered since then, but I'll give them a pass)

Re: Software Developers Should Have Sysadmin Experience

#317

Post by technofiend »

devonkim wrote: post #278
technofiend wrote: post #170

Earlier quoted context omitted.

>Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them. This is not personal criticism but you know how I know you're not working in a highly regulated environment? Check out the Carnegie Mellon Ca…

It's funny that you bring up CMMI in the discussion. CMM(I) is nearly the antithesis of Deming's approaches toward quality management. What's really interesting is that Deming's approaches were adopted by Toyota decades ago to historically great effect (sadly with few other large examples of named successes in the business world) while Taylorist approaches (including CMMI) a handful of other Japanese daibatsu and esp…

No doubt valid points about CMMI; I was exposed to it as part of a program to improve quality and in that specific instance it was constructive. However the program office oversight was shut down as all portions of the business were certified as "level 2" and without that level of structure and control most of the process died within a year.

Even so my original point remains - in some kinds of highly regulated shops there's enough external pressure for controls and separation of duties that the developer simply cannot have access to production. I'm not defending either practice (CMMI or seperation of duties), I'm just saying in some places it's reality, regardless of perceived drawbacks or overhead.

Re: Software Developers Should Have Sysadmin Experience

#318

Post by peterwwillis »

eropple wrote: post #67
peterwwillis wrote: post #63

Earlier quoted context omitted.

There's reasons we have these classes of people with certain labels. Until the 18th century, buildings were designed and built by "artistan craftsmen", people who had no formal training or education in building things (or in any capacity) but said, sure, I can make you a house. Then it would fall down and kill a family of five. All around the world today, architects and engineers are required to be licensed and prove…

I've read your post two or three times now, and I straight-up don't get what you're driving at. Yes, there are levels of subject matter expertise that you need to know--or at least be conversant enough with to deep-dive when necessary--for system administration. That's also the case for database programming or bioinformatics or graphics programming or whatever. But it is still expressed through the writing of code ,…

I would be horrified if you had to be a software developer to be a sysadmin. It would be like requiring an auto mechanic to be an engineer, or a cook to be a chemist or something.

In actual fact, the complexity of software design is antithetical to the job of system administration. If at all possible you should use the least code possible to do a given job, and rely on the reuse of tools to serve various functions. This is not coding, just like auto repair is not engineering. If you're writing code you are not adminning, you are developing.

There is a time and a place for software development in Ops, but it should be based around projects led by teams to serve specific functions that existing tools won't solve, again similar to auto repair because some tools are needed simply to repair things more efficiently, but then you use those tools and don't keep engineering new ones. The distinction is small but highly important, because code is often a source of headaches in Ops. This is coming from a guy who is usually hired to write code for Ops departments.

A software developer can of course perform some of the functions of a sysadmin, but again the complexity of the whole becomes too much to understand just by doing single tasks. Either job is too complex to learn without a lot of study.

I just find comparing the two in the context of one becoming the other invites oversimplification of either role, and devops doesn't help that comparison.

Re: Software Developers Should Have Sysadmin Experience

#319

Post by jessety »

fauigerzigerk wrote: post #134

What he's talking about is systems architecture not sysadmin experiece. Of course developers have to write code for a logical model of a machine that isn't necessarily the same as the laptop they're working on. But I don't think anyone would ever dispute that. Storing pictures on a local disk when the software is supposed to run on a cluster has absolutely nothing to do with sysadmin experience.

I came here to say this.

If a developer is building an app for 100K users the same way they'd build an app for 100 people, the systems architect dropped the ball- not the developer.

Re: Software Developers Should Have Sysadmin Experience

#320

Post by protomyth »

ihattendorf wrote: post #313
protomyth wrote: post #283

Actually, I would rather language and tool designers have a go at a true System Admin job. There is a reason PHP gets installed by default and Ruby on Rails does not. Frankly, at the end of the day, most businesses don't want something that was "working" breaking. All the conflict comes from that one desire. There is a reason System Admins have to do the "Patch Tuesday" dance and despise any software that just update…

True for the most part, however PHP is a language while Ruby on Rails is a framework.

PHP comes with all the stuff to make a website. Ruby needs a framework, and Ruby on Rails is the most popular. I think you proved my point for me.
Post reply on HN