Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Software Developers Should Have Sysadmin Experience

blog.professorbeekums.com

Post reply on HN
Print view View original

351–360 of 424 posts

Re: Software Developers Should Have Sysadmin Experience

#351

Post by cableshaft »

ryandrake wrote: post #347
cableshaft wrote: post #249

Earlier quoted context omitted.

> I half believe the reason developers are embracing cloud architecture so much is to remove so many sysadmins out of the equation. It was literally true at one of my previous jobs. We couldn't install anything on our own dev machines without approval from Net Ops, not even Notepad++ (I don't think I ever got that installed, never got approval). We once asked for a new server which mirrored the software of an existin…

I've seen this as well. "Timeline from internal IT for provisioning a box and deploying our app is 6 months, and subsequent changes go through a ticketing system with a 2 week average turnaround. Or, we can have it running ourselves on AWS in 30 minutes."

Yeah, that sounds spookily similar to the process we had, including the ticketing system changes timeline.

Re: Software Developers Should Have Sysadmin Experience

#352

Post by WorldMaker »

user5994461 wrote: post #348
serge2k wrote: post #338

Earlier quoted context omitted.

and we get yelled at when we can't deliver a feature because some gatekeeper is sitting atop his little throne in the kingdom of servers saying no. ;) This is institutional failure.

Honestly. Developers should stop thinking that their jobs is to release features all the time at all costs. That's simply not true and that's counter productive for the business.

If only the business might learn that it's not their jobs to request new features all the time at all costs...

If only customers weren't fickle and might learn not to demand new features all the time whatever the costs...

It's turtles all the way down.

Re: Software Developers Should Have Sysadmin Experience

#353

Post by serge2k »

user5994461 wrote: post #348
serge2k wrote: post #338

Earlier quoted context omitted.

and we get yelled at when we can't deliver a feature because some gatekeeper is sitting atop his little throne in the kingdom of servers saying no. ;) This is institutional failure.

Honestly. Developers should stop thinking that their jobs is to release features all the time at all costs. That's simply not true and that's counter productive for the business.

Developers do what the business wants them to.

Re: Software Developers Should Have Sysadmin Experience

#354

Post by just2n »

Anecdotal as it is, I started my career working as a sysadmin while studying CS. When I was younger I was quite interested in security (e.g. I followed defcon and CVE lists and read a lot of manuals and source code). I built server software, broke applications, and did a lot of reverse engineering. In that time I was forced effectively to learn about how the OS worked and how to utilize it, both Linux and Windows, from C APIs to scripting, package management, and everything else needed to effectively work in those environments, primarily around reverse engineering and vulnerability research.

That experience naturally lead to me becoming a sysadmin during my study at university. It was a fairly straightforward application of what I already had learned with a much larger scale of management. The primary thing I gained out of it was a drive to automate everything. When I started that job most of the sysadmin work was manual, but a few of us spent a huge amount of time focusing heavily on automation and when I left most of the work was automated and we were just doing meaningful firefighting and supporting development.

As an engineer, the main benefits have been understanding how my software is going to run in an actual software/hardware stack, easily jumping into a production environment and debugging complicated issues, being able to quickly have my OS do what I want, and that drive to automate everything. A lot of that informs how I build software and in general it feels like it makes me a lot more productive.

Re: Software Developers Should Have Sysadmin Experience

#355

Post by fl0wenol »

Sacho wrote: post #206
briffle wrote: post #181

Earlier quoted context omitted.

what your probably not seeing, is the CIO/CSO screaming at the SA to get AV deployed on every machine in the company, to meet some audit requirement checkbox, or PCI compliance, by the end of the month.

So the problem is effective communication? Why can't the sysadmin in this imagined scenario explain their actions this way?

When I have seen this problem, it's because the sysadmins are instructed (or have learned via experience) not to explain their reasoning to developers or end-users. Because if they did, then it becomes a discussion or argument that becomes a time sink since there was very little chance they could change the mandate even if they agreed.

So they become intentionally opaque to move that discussion out of their laps and make it come via the development team managers confronting the operations managers and having the fight on that turf.

Such situations occurring is a sign that the organization is not set up effectively. This sort of confrontation shouldn't need to be happening.

Ideally the development team's lead and/or project managers are involved with, are informed ahead of time, or are even contributing to the policy decisions on the operational side.

Re: Software Developers Should Have Sysadmin Experience

#356

Post by dozzie »

user5994461 wrote: post #350
brianwawok wrote: post #184

Earlier quoted context omitted.

That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading? I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in sett…

> That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? What if the sysadmin can code in the same language as you, faster and with less bugs?

Better yet: what if the sysadmin can code better and faster in the same language and also three others?

Re: Software Developers Should Have Sysadmin Experience

#357

Post by UK-AL »

SomeStupidPoint wrote: post #208
AlexB138 wrote: post #200

Earlier quoted context omitted.

As a Sys Admin turned Automation/Tools Engineer, I think you're missing part of the point. You've got the beginning of it right in saying that Sys Admins used to be involved in pinning down versions, and even in why that was necessary, but I believe you're incorrect in saying that the containerization technologies are bad for removing that. Those technologies don't exist so Developers can get around Sys Admins and ig…

Doesn't that constant updating and lack of version oversight create security risks? Developers may not be as aware of those topics as sysadmins.

The lack of updating is also a massive security risk.

Re: Software Developers Should Have Sysadmin Experience

#358

Post by LoSboccacc »

hueving wrote: post #309
LoSboccacc wrote: post #234

Earlier quoted context omitted.

funny how most of the things you list are either stuff that can be audited in code alone (query optimization, cache thrashing) or totally out of control of the developer (load balancer issues, tcp tuning) sure if you bounce them all up like that it might look like you have a point, except it falls apart when you attribute concerns properly. or please explain, how would dealing with kernel tcp tuning part-time help Jo…

Query optimization can't be audited in code alone. The indexes you need depend heavily on the database system that you are using in production. Do you know at what point your DBMS stops loading the whole table in memory? Do you know what datastructure and algorithm it's using when you do a LIKE query? Cache thrashing also can't be audited in code alone without understanding the architecture that the the app is going…

> Joe might learn that connections aren't as cheap as he thinks and maybe it isn't a great idea for each client to require 50 connections

so we're back to point one, you need devs that go trough basic education and stop cheapening out hiring Joe / or Joe should ask a refund from his tuition fees.

> snip of stuff that one does not know off the bat

sure but it is knowable, it's not exactly hard. database are predictable, building indexes on the right places is not an esoteric practice that can only be done by trial error and rituals etc etc. literature is quite adbuntant, easy to process and complete with tradeoffs about different approaches and how they impact performance, maintainability etc.

99% programmers aren't breaking new ground.

Re: Software Developers Should Have Sysadmin Experience

#359

Post by joshwcomeau »

I think that there's good advice in this article, but a couple counter-points:

- Plenty of software developers work for small- to mid-size companies where the Node or Rails server running on your machine is nearly identical to the one running in production.

- Plenty of software developers work exclusively on the front-end, which means that the computer your code runs on in production is very very similar to the one you develop on (although then you have the added concern of testing on lower-end mobile devices, other browsers, etc)

Re: Software Developers Should Have Sysadmin Experience

#360

Post by dozzie »

btreecat wrote: post #215
bandrami wrote: post #188

Earlier quoted context omitted.

why should a sysadmin who doesn't even know Python tell me what version of a library I will use Because he maintains that installation and you don't? But, yeah, that's why virtualenv, Docker, etc. were invented, because devs kept getting sick of installations having consequences. What does a sysadmin have to approve or deny an upgrade? Check for conflicts of this version of this library with other software currently…

>Because he maintains that installation and you don't? So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK? >Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). That's not the case w…

>> Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer).

> That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it.

This illustrates why developers should have some experience with administering systems: do not deploy unrelated services on the same machine.

And you know what happens as a byproduct of this rule of hygiene? Suddenly the version conflicts disappear, at least for things that aren't broken anyway.

Post reply on HN