Live data from Hacker News

Software Developers Should Have Sysadmin Experience

blog.professorbeekums.com

11–20 of 424 posts

Re: Software Developers Should Have Sysadmin Experience

#12
Should sysadmins have software development experience (e.g. DevOps)? For what values should 'X have Y experience?' Should we go as far as...

"A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects." — Robert Heinlein, Time Enough for Love

Re: Software Developers Should Have Sysadmin Experience

#13
post #2

Somewhat related, I feel that mechanical engineers who design cars should have some experience servicing vehicles. Sometimes engineers will not leave enough space, use weird fasteners, etc. that make a simple job much more complex.

That is a function of design requirements, not lack of foresight. If your manager tells you the cab needs to be 20% roomier without changing the outside dimensions, or they change the engine going into the thing without changing the structural integrity of the car, there goes your maintenance space.

I feel that managers of mechanical engineers who design cars should have some experience servicing vehicles.

Re: Software Developers Should Have Sysadmin Experience

#14
I don't think "should have" is the right statement; more like having at least one person on the team with sysadmin experience is extremely helpful. I know it has been for me.

Of course in HS/college I ran a website that was a frequent target of hate in the late 90s/early 2000s and it taught me about XSS and CSRF before people invented fancy terms to describe them. It also taught me about HTML/JS escaping, DoS/DDoS, SQL injection, and how all your defenses are useless if someone social-engineers their way into root and nukes everything. I have the assumption that everything is compromised and user input is toxic waste burned into my subconscious.

Re: Software Developers Should Have Sysadmin Experience

#15
Sysadmin or devops is responsible for uptime directly, being ultimately responsible for the services to remain performant. It's a mindset and temperament that may help developers see solutions or foresee pitfalls while designing and implementing code. Zero-downtime attitude. Making and testing deployments, upgrades and migrations to be least impactful as a core function is probably the biggest effect of putting a developer in a sysadmins shoes.

Re: Software Developers Should Have Sysadmin Experience

#16

With the rise of devops (docker, puppet, ansible, etc...) , and cloud VMs, how often are sysadmins different from software devs?

They're not really and they shouldn't be (although systems knowledge here becomes increasingly important). This is the point of DevOps, that operations can be treated as a software problem and managed in an automated fashion. The role of "SysAdmin" as presented in this article is fading quickly.

Re: Software Developers Should Have Sysadmin Experience

#17
That's what I did at my tiny engineering team (3 folks) and results have been great.

If you have a small organization (say Automation is the key though. If everyone connects to the boxes and does random things manually over ssh, nothing good will come out of it.

Still, you need to have a person or two who are responsible for the vision of the architecture/systems and who make sure that things don't go off the rails.

Re: Software Developers Should Have Sysadmin Experience

#18
This is a short little article that just barely touches on a much deeper, often hidden issue; The state of system administration in business is abysmal.

It's not the developers fault though, at least not as much as devops types would like you to believe. I think the author has a good point, in that its good to get Devs thinking about real world environments on deploy, but the real world is much more complex than concurrency of servers.

All that being said though, very rarely have I as a sysadmin of 10+ years seen problems so easily attributable to devs. Of course I haven't lived in hn/sv startup land either, so take that into account, but failures in systems I have seen have almost always been a failure of management, up to and beyond C level.

I could go into detail, but I'll save it for another time. Suffice it to say, what businesses need to be doing is getting better CTOs and CIOs who can bridge the gaping chasm between sysadmins and managment.

Devs, you keep being Devs, and let the sysadmins be sysadmins. Cross train and communicate when you can, but don't fool yourselves, it is management that bears the responsibility and burden of you both. Management just doesn't like to admit that to themselves or anyone else, so don't play into this Devs vs sysadmins dialectic too much, lest ye find yourself the scapegoat next go round.

Re: Software Developers Should Have Sysadmin Experience

#19
post #8

Sysadmin knowledge definitely helps, but so does an MBA, knowledge of writing, public speaking, design, user experience, networking. Oh, and the domain of the problem itself. The skills I require of my developers depend on the rest of the team and the project.

The difference with this is that server-side software has direct operational consequences.

A server-side developer who never deals with ops is like a chef who never tastes the food. It's in theory possible to get right, but in practice the results tend to be poor.

Re: Software Developers Should Have Sysadmin Experience

#20
We took it one step further on my team. The best way to ensure that the pager(duty) doesn't go off at 2am in the morning is to put developers in the first on-call group. Not only did it work, but the developers got a much more nuanced view of how systems operate. When we first started, I'd see developers using ping to determine whether a nameservice entry was correct. After a few months of handling almost all of our own ops, developers knew how each piece of the puzzle worked and how it all fit together rather than the hazy, largely abstracted view they had before.

But we learned that we needed it to go the opposite direction too. We had ops people who, when given a corporate-wide mandate to apply a security patch or some such task, would log into every machine and apply the patch, despite the fact that we'd been practicing immutable infrastructure with zero-downtime deployments. We hadn't given them the necessary exposure to the dev side to understand that you had to apply those fixes to a base image and trigger a redeploy. There was a lot of finger pointing a couple of weeks later after it was discovered that the fixes were overwritten by an application deploy.

Post reply on HN