Software Developers Should Have Sysadmin Experience • Hacker News Board

Re: Software Developers Should Have Sysadmin Experience

#251

Post by eightysixfour » Mon, Jan 09, 2017, 4:06 PM UTC

btreecat wrote: post #215

bandrami wrote: post #188

Earlier quoted context omitted.

why should a sysadmin who doesn't even know Python tell me what version of a library I will use Because he maintains that installation and you don't? But, yeah, that's why virtualenv, Docker, etc. were invented, because devs kept getting sick of installations having consequences. What does a sysadmin have to approve or deny an upgrade? Check for conflicts of this version of this library with other software currently…

>Because he maintains that installation and you don't? So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK? >Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). That's not the case w…

I have some perspective as someone who has done a bit of all of these jobs over the last 10 years as well as working in a hosting company that handled release management for large Fortune 500s.

> So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK?

Proprietary software generally has a support agreement and SLA for fixing things instead of getting the response "it works in dev!"

> That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it.

That's why containers are great, but you have to remember most of the world isn't as fast as this community to adopt things, a lot of things are still being managed the hard way on shared servers with literally thousands of dependencies. Migrating to containers in these instances can't happen fast enough.

> There are solutions to the issues you raise (often trivial ones at that), they just require an adjustment to workflow and an increase in communication between developers and their sysadmins.

Implementing even trivial changes to processes that impact hundreds of people across multiple continents is often not trivial. Devs in India, devs in the US, hosting teams, release management, etc. A lot of those people are doing just enough to get by and not up-to-date tech wise, so not only are you implementing new tools and processes, but you're building out training programs around using them, etc.

These processes are old and will be modernized in time but that's the reality for a lot of "sysadmins."

Re: Software Developers Should Have Sysadmin Experience

#252

Post by huherto » Mon, Jan 09, 2017, 4:07 PM UTC

pjmorris wrote: post #12

Should sysadmins have software development experience (e.g. DevOps)? For what values should 'X have Y experience?' Should we go as far as... "A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new proble…

> Should sysadmins have software development experience (e.g. DevOps)?

May be. Not only you understand better the needs of software developers. But also, you are able to automate a large chunk of your work. Paraphrasing Larry Wall: Sysadmins should be Lazy. "Laziness: The quality that makes you go to great effort to reduce overall energy expenditure. ..."

Re: Software Developers Should Have Sysadmin Experience

#253

Post by rch » Mon, Jan 09, 2017, 4:08 PM UTC

jstimpfle wrote: post #207

AlexB138 wrote: post #200

Earlier quoted context omitted.

As a Sys Admin turned Automation/Tools Engineer, I think you're missing part of the point. You've got the beginning of it right in saying that Sys Admins used to be involved in pinning down versions, and even in why that was necessary, but I believe you're incorrect in saying that the containerization technologies are bad for removing that. Those technologies don't exist so Developers can get around Sys Admins and ig…

> They aren't kids playing with toys that we need to nanny over In my experience (university), yes they are, and they should do that at home. Why do you need the latest bleeding versions in the first place? In my sysadmin experience, people believe software gets bad and deprecated as soon as the glory next breaking version appears. I don't think I need to argue why this is an illogical stance. With my developers hat…

No offense intended, but your university probably isn't competing for top engineers. Grad students and postdocs aren't professionals yet either.

The sysadmin role has traditionally been a focus in that environment (e.g. controlling access to cluster resources).

Re: Software Developers Should Have Sysadmin Experience

#254

Post by bluejekyll » Mon, Jan 09, 2017, 4:09 PM UTC

jstimpfle wrote: post #237

adrianN wrote: post #227

Earlier quoted context omitted.

People could do great things just with punch cards, yet somehow technology kept marching on. If developers want to use newer stuff usually they have a good reason. The ability to hack around the deficiencies of old dependencies does not mean that one couldn't get a better, cheaper solution with newer technology.

> People could do great things just with punch cards, yet somehow technology kept marching on. That's not the situation I've described - punch cards disqualify. The situation I mean is where developers insist on writing software on version X, which doesn't compile on X-1 and is buggy on version X (and might not compile again on X+1). For a concrete example, new C++ features that aren't correctly implemented and lead…

If it is as you say, then why upgrade ever? How would we even discover bugs in software until it is used?

To have progress we need to change things. When we change things, we may break things, regardless of tests.

To quote deijkstra: "testing can be a very effective way of showing the presence of bugs, but it is hopelessly inadequate to show their absence". From 'the humble programmer'.

Production is the only way to eventually discover the stability of any software, even with 100% test coverage. It's a necessary evil in the support of progress.

Re: Software Developers Should Have Sysadmin Experience

#255

Post by astine » Mon, Jan 09, 2017, 4:09 PM UTC

amorphid wrote: post #233

iask wrote: post #226

Earlier quoted context omitted.

At what point do you determine who is responsible for securing the environment? The "Gatekeeper" mentality is stemming from this. There is no clear line in any organization and I see the blame-game all the time.

That's kind of like asking which employees at a bank are responsible for keeping cash in the vault. Hopefully it’s a group effort.

There are actually generally rules for which employees have responsibility over the vaults in banks. The employees aren't equally responsible. Roles and responsibility are defined strictly. Generally, the lowly tellers aren't allowed the same access to the money as the general manager, and they aren't held responsible to the same degree if money in the vault goes missing.

Re: Software Developers Should Have Sysadmin Experience

#256

Post by onion2k » Mon, Jan 09, 2017, 4:10 PM UTC

FLUX-YOU wrote: post #195

onion2k wrote: post #83

Earlier quoted context omitted.

How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. A good sysadmin (or the sysadmin's boss) should be able to pull up some numbers and say "Build tasks are being queued for an hour before they run. What impact is th…

>If you're blindly following "orders" to reduce costs and doing things that push up costs elsewhere then you're not doing a good job. This is the responsibility of someone above to know whether or not the orders they give should be given. If they need to ask for information from people below them, fantastic, please help them along. Please don't fall on the sword for incompetent managers.

This is the responsibility of someone above to know whether or not the orders they give should be given.

Yes, and part of that is the people in their team(s) helping them and understanding that they're fallible and may fail to ask a pertinent question. Equally, the manager needs to be open to updates volunteered by their team without a prompt. Ultimately everyone does better if the entire group works together.

Re: Software Developers Should Have Sysadmin Experience

#257

Post by StillBored » Mon, Jan 09, 2017, 4:11 PM UTC

gregmac wrote: post #243

jstimpfle wrote: post #220

Earlier quoted context omitted.

Counter-wit: Why can't people get their software to work with the existing libs? Hint: It's very rarely that the existing libs disqualify.

Pick your poison. If you run into a bug or problem with a 3rd party component (open source library, commercial tool, whatever), one of the first things they are going to ask you to do is upgrade. The fact you're on an old version of some library is an easy (and sometimes correct) scapegoat for problems. Put yourself in the 3rd party's shoes: if you spend a bunch of time trying to fix a problem that turns out to be a…

Upgrading the version of the library wouldn't be a problem if the concept of stable ABI's were as prevalent as it was 15 years ago. Back then, the major.minor version number system was used as a signal that it was safe to upgrade to a newer version of a library without worrying that the entire application stack was going to come falling down around itself because the developer of said library decided to rework some part of the library without providing any backwards compatibility.

Put another way, a sysadmin could feel confident that moving from 1.52->1.53 would be a painless and transparent operation and that the provider of said library would continue to release 1.x branches with little ABI changes for some length of time. The expectation was that at some point the library provider would release a 2.0, which would require a more careful testing/deployment schedule likely with other upgrades to the system.

Today, that is all out the window, very few open source projects (and its infecting the commercial software too) provide "stable" branches. The agile, throw out the latest untested version mentality is less work than the careful plan/code/test/release, followed by fix/test/release, cycles.

This is a major rant of mine, as upgrading the vast majority of open source libraries usually just replaces one set of problems with another. Having been on the hook for providing a rock solid stable environment for critical infrastructure (think emergency services, banks, power plants, etc) I came to the conclusion that for many libraries/tools you had better be prepared to fix and backport bug fixes yourself unless you were solely relying on only libraries shipped in something like RHEL/SLES (and even then if you wanted it fixed fast, you had better be prepared to duplicate/debug the problem yourself).

Re: Software Developers Should Have Sysadmin Experience

#258

Post by [deleted] » Mon, Jan 09, 2017, 4:11 PM UTC

Aloha wrote: post #39

nokya wrote: post #37

I'd say there is too much effort in reasoning on the wrong problem. What worries me the most is the 'why': why do (too) many software developers don't know about sysadmin? I have been involved as a consultant in large software projects in the last two years and a vast majority of money lost in delays and bugs was caused by devs not understanding: 1) the difference between virtual memory and physical memory 2) the dif…

I've worked with all kinds - from windows devs who can't figure out how to install visual studio - to people who understand Windows, Linux, and macOS - as well as basic system administration for each platform. The people who are most successful at rapidly developing good high quality software are more in the later group. Would you trust a RF engineer who couldn't troubleshoot his own radio designs? why would you trou…

[deleted]

Re: Software Developers Should Have Sysadmin Experience

#259

Post by jstimpfle » Mon, Jan 09, 2017, 4:13 PM UTC

treehau5 wrote: post #246

jstimpfle wrote: post #207

Earlier quoted context omitted.

> They aren't kids playing with toys that we need to nanny over In my experience (university), yes they are, and they should do that at home. Why do you need the latest bleeding versions in the first place? In my sysadmin experience, people believe software gets bad and deprecated as soon as the glory next breaking version appears. I don't think I need to argue why this is an illogical stance. With my developers hat…

My comment to you two is -- why not both? Stability, sanity, all that is amazing, and a must have. But also bug-fixes, security improvements, and performance improvements are wonderful too, which tends to come with using up-to-date dependencies. The problem with the latter, as you mentioned, is when it introduces breaking API changes and is wholly not backwards compatible. This is not a "kids playing with toys wantin…

I like your answer. It's civilized, balanced, and I agree with every word of it :-)

I'll add as an anecdote that I do follow your practices (limiting dependencies). It works wonderfully on Debian stable (most of the software there is now >2 years old, the next version has just been soft frozen). I have the occasional package pulled from testing: I recently toyed with Perl6. And currently I use a newer version of python3-sphinx for a nicer doc syntax but I could do without. It causes no headaches at all.

Re: Software Developers Should Have Sysadmin Experience

#260

Post by Diederich » Mon, Jan 09, 2017, 4:13 PM UTC

I worked in the home office of WalMart Stores, the retailer, from the mid 90s until 2009 as a hybrid network engineer/developer. That is, my team worked in Network Engineering, but we wrote tons of code, because when there are millions of different IP addressable nodes on a centrally managed network, there will be code to manage it. (:

In that era, I think WalMart handled this apparent developer/sysadmin confliact quite well. Believe it or not, even though we (the whole company) had exceptional uptime, we were also extremely agile. LOTS of code was written and rolled out across thousands of sites, multiple times a day.

There were a number of keys to this, and I'll enumerate a few.

1. Most new hires (those that had minimal previous experience) to development positions had to spend their first six months working at one of the four or so major help desks. Note: they were paid their target salary, but in an hourly fashion. 2. The various operations teams had complete and final say so over what went out, and how incidents were handled. A couple of the big operational areas were Unix Operations, Network Operations, Windows Operations and Mainframe Operations. I called them 'teams' but each had a number of teams, and their own help desk. 3. Any time there was an impacting problem associated with a program developed by team X, a war room was called by the relevant operational area. A member of team X (a developer) had to stay in that war room (switching off between team members if it ran very long) until the production problem was not only fixed, but also deemed unlikely to recur, except in cases where that dept of a fix would take a long time. 4. Every development team had a pager rotation, with rigorous expectations about responding to such pages. This was primarily to support the previous point. 5. Because of the enormous operational scale, all of the major operational areas had dedicated teams focused on automation. My team was that team for the networking area. Furthermore, most of the rest of the operational folk read/wrote code to some extent.

In short, incentives were aligned. Teams that wrote externally facing code felt pain if the stuff they wrote and released caused problems. Operational folks wrote/managed/interacted with tons of their own code in order to manage the enormous infrastructure. Also, ops folk were far more willing to let things move with velocity knowing that the people who actually wrote the code would be required to support it, globally, any time of the day or night.

Another, perhaps even more important reason we were so successful during those years (and the years before) was a strong and vibrant esprit de corps. The entirety of Information Systems was, at the time, around 2000 people, and we were facilitating double digit year over year growth over a 150 billion dollar company. We had over 5000 remote sites in 15+ countries, with a diversity of software and infrastructure that was honestly pretty astounding. Each of those sites had quite a surprising amount of infrastructure.

We worked hard, and we produced huge velocity with fantastic uptime. For example, the network achieved six 9s of availability a couple of quarters.

In end, while things were sometimes contentious, we trusted each other, and only minority of teams were forced to work bad hours for any length of time.