Software Developers Should Have Sysadmin Experience • Hacker News Board

Re: Software Developers Should Have Sysadmin Experience

#301

Post by eropple » Mon, Jan 09, 2017, 5:19 PM UTC

arca_vorago wrote: post #271

eropple wrote: post #34

Earlier quoted context omitted.

I agree that the dialectic shouldn't be played...but not that it's "cross-training." Rather, it's the same skillsets being applied in slightly different ways. There was a day in which "sysadmin" very often meant "shit-hot Perl slinger." It was before my day, but I know some of the graybeards who can still lay claim to it. The sysadmin who can't write good, maintainable code is going to rapidly see their positions red…

> The sysadmin who can't write good, maintainable code is going to rapidly see their positions reduced to sinecures in large and slow companies. I completely disagree, but respectfully. What I think we are lacking in this discussion is a differentiation between what we mean by sysadmin as a product and as a job description. Systems Administration is a aggregate management of the technical infrastructure. (for $reason…

To me, you're just describing another layer of management. Which is fine--management is important!--but it's not the same thing as the actual implementor class that "sysadmin" usually refers to. And those people are inexorably going to be doing their administration via code. There isn't enough time in the day to waste with "pet" servers. They're cattle. Infrastructure as code is here, and it's just going to grow more extensive. If you can't write code effectively--both in terms of stuff like "hey, this code can be read by other people" and "this code can be automatically tested like any other code"--you're in trouble. One of my clients is a top-10 commercial bank in the United States that has not only moved entirely into the cloud (save for some legacy mainframe stuff that they're working on) and every team is not only required to provision and operate strictly with automated tools (both instance and cloud-level provisioners), but their non-database servers are killed automatically every few months to ensure that they're rotating successfully and without human involvement.

I'm pretty confident that I've seen where we're going with this. The management piece of the puzzle is totally important...but the practice is going to continue converging with every other bit of software development.

Re: Software Developers Should Have Sysadmin Experience

#302

Post by eropple » Mon, Jan 09, 2017, 5:24 PM UTC

KallDrexx wrote: post #289

AlexB138 wrote: post #200

Earlier quoted context omitted.

As a Sys Admin turned Automation/Tools Engineer, I think you're missing part of the point. You've got the beginning of it right in saying that Sys Admins used to be involved in pinning down versions, and even in why that was necessary, but I believe you're incorrect in saying that the containerization technologies are bad for removing that. Those technologies don't exist so Developers can get around Sys Admins and ig…

This has actually been a very insightful response to me. After reading your comment it now occurs to me that Docker and other container systems are actually a huge organizational tool. One issue I have encountered at companies is keeping the IT and development departments on the same high level organizational incentives to keep political barriers from coming up between them (and conflicts arising). Containers can hel…

At almost every shop I've ever seen, "sysadmins" are also the ones whose responsibility it is to at least attempt some sort of security practice and business continuation. Which opaque, "just run it" containers actively fight against. Did the developers actually audit what they've pulled in as dependencies? Did they make sure that they can be rebuilt if whatever package source goes away? Where is everything documented? Containers, as usually implemented, rather than "keep everyone's incentives aligned", instead damages the ability of the adults in the room to keep everything from falling apart.

(I have been both the sysadmin saying "no" and the developer mad at sysadmins saying "no". But going slower and doing our homework has never, ever hurt me or my employers.)

Re: Software Developers Should Have Sysadmin Experience

#303

Post by bandrami » Mon, Jan 09, 2017, 5:25 PM UTC

gaius wrote: post #279

brightball wrote: post #270

Earlier quoted context omitted.

Backwards compatibility is seriously under appreciated. When I tell developers to ensure that their changes are backwards compatible, they tend to look at me like I'm green. I do not understand the disconnect that developers have with understanding all of the benefits that it brings. Yes, you have some extra code in your code base so it's less clean. You also have a stable environment as a result. The first affects o…

I do not understand the disconnect that developers have with understanding all of the benefits that it brings. Because they've never worked on an old codebase, because front-end technologies change so often and everything just gets re-written anyway. It's a waste of time worrying about this when the code won't make it to is first birthday. If you were speaking to seasoned C and DB developers about stability in the to…

This gets to the complaint that so much of the open source ecosystem gets to version 0.8.6 (whether it's named that or not) and then completely rewritten "this time the right way". That's not actually a good thing.

Re: Software Developers Should Have Sysadmin Experience

#304

Post by crdoconnor » Mon, Jan 09, 2017, 5:26 PM UTC

jstimpfle wrote: post #211

brianwawok wrote: post #184

Earlier quoted context omitted.

That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading? I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in sett…

Another perspective: Why don't you want your software to be compatible with the system that your sysadmin provides? (Assuming that system is not completely obsolete). Minimize your dependencies. It's incidentally also what leads to clean code bases.

>Minimize your dependencies. It's incidentally also what leads to clean code bases.

Oh hell no. I have wasted far too much of my life maintaining buggy, technical debt ridden reinvented wheels where there was a well maintained package that could just have been used instead.

Re: Software Developers Should Have Sysadmin Experience

#305

Post by [deleted] » Mon, Jan 09, 2017, 5:28 PM UTC

brianwawok wrote: post #240

jstimpfle wrote: post #211

Earlier quoted context omitted.

Another perspective: Why don't you want your software to be compatible with the system that your sysadmin provides? (Assuming that system is not completely obsolete). Minimize your dependencies. It's incidentally also what leads to clean code bases.

> Minimize your dependencies. It's incidentally also what leads to clean code bases. You also crush velocity. Smart use of libraries lets you ship code 10x faster. Two identical businesses.. one writes all their own code, one is smart about using libraries. Which one makes it to IPO first, and which goes bankrupt?

[deleted]

Re: Software Developers Should Have Sysadmin Experience

#306

Post by bandrami » Mon, Jan 09, 2017, 5:31 PM UTC

paulddraper wrote: post #293

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

The problem is that sysadmins don't know everything. > in the year 2015, pip went from version 1.5.6 to 8.1.1 The only releases in 2015 were 6.x and 7.x. There were 8 documented backwards incompatibilities, 4 deprecated the previous year, and 3 documenting a couple bugs that were fixed several days after the 7.0.0 release. These are the sorts of thing an aware Python developer will know.

You're right; it was the period from December 22nd, 2014 to March 17th, 2016, so about 15 months centered around 2015.

We may be counting regressions differently; I'm including both adding and removing the spinner as a regression, for instance (since both the addition and removal added unexpected behavior).

Note that the undeniable regressions that occurred in releases during those 15 months included:

1. Exceptions raised in any command on Windows

2. Switching from not installing standard libraries to installing them back to not installing them

3. Blocking if the particular server pypi.python.org was down

4. An infinite loop on filesystems that do not allow hard links

Note that in that time they also added yet another internal package management system (incompatible with the existing two), changed the versioning semantics twice, and dropped support for versions of python that were 3 years old at that point.

And, again, there's nothing particularly wrong with or bad about pip; this is just what a younger generation of developers are used to.

Re: Software Developers Should Have Sysadmin Experience

#307

Post by eropple » Mon, Jan 09, 2017, 5:32 PM UTC

astrostl wrote: post #239

awinder wrote: post #216

Earlier quoted context omitted.

> 15 years ago, when a project would kick-off, as a sysadmin > I'd be invited in and the developers and I would hash out > what versions of each language and library involved the > project would use To wit: the new ideologies in infrastructure management are actually designed to solve the underlying problem that necessitated that kind of working setup. Why should the version of a lib in one part of the software someh…

> Why should the version of a lib in one part of the software somehow pose existential threat to the infrastructure? Because that's how software developers wrote every dominant packaging system :P There are tradeoffs to self-contained units. Disk space isn't so much of a practical concern these days, but security is very real: with a dozen apps, you could be at the mercy of a dozen different entities to update their…

Or the statically compiled application that "just works" and is "so easy to build and maintain". Lookin' at you, Golang.

Re: Software Developers Should Have Sysadmin Experience

#308

Post by zeveb » Mon, Jan 09, 2017, 5:34 PM UTC

AlexB138 wrote: post #200

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

As a Sys Admin turned Automation/Tools Engineer, I think you're missing part of the point. You've got the beginning of it right in saying that Sys Admins used to be involved in pinning down versions, and even in why that was necessary, but I believe you're incorrect in saying that the containerization technologies are bad for removing that. Those technologies don't exist so Developers can get around Sys Admins and ig…

> It removes the underlying need for a Sys Admin to worry about the versions.

No, they really don't: they remove the ability of system administrators to administer versions of software across the total system.

This is bad, e.g. when a new OpenSSL vulnerability comes out (it being a day ending in -y) and every piece of software has to be updated.

> We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better.

I am a developer, and I disagree. We are, by and large, kids playing rather than adults making carefully considered decisions. We'd rather use v3.0.rc-1-awesome rather than 2.17.12, because the former is the version that adds an API that saves us from writing twenty lines of code, never mind that it also is untested, unstable and very likely insecure.

We need adult supervision. We need oversight. That's why I argue for using stable, LTS-style distributions, and running against the distro packages unless there is a very good business reason not to (and yes, 'we can't implement necessary functionality in a cost-effective timeframe' is a valid business reason). I'm not opposed to using the bleeding edge when it makes business sense; I'm opposed to developers using the bleeding edge because they like it, and keeping the business in the dark.

Re: Software Developers Should Have Sysadmin Experience

#309

Post by hueving » Mon, Jan 09, 2017, 5:37 PM UTC

LoSboccacc wrote: post #234

hueving wrote: post #229

Earlier quoted context omitted.

>the complexity and memory requirement of code Those are only tiny slices of real production bugs. No amount of complexity analysis of your code ahead of time is going to protect you from all of the issues that arise with integrating any large system dealing with lots of requests. You run into all kinds of things like query optimization, kernel TCP tuning, load balancer problems, cache thrashing, high latency clients…

funny how most of the things you list are either stuff that can be audited in code alone (query optimization, cache thrashing) or totally out of control of the developer (load balancer issues, tcp tuning) sure if you bounce them all up like that it might look like you have a point, except it falls apart when you attribute concerns properly. or please explain, how would dealing with kernel tcp tuning part-time help Jo…

Query optimization can't be audited in code alone. The indexes you need depend heavily on the database system that you are using in production. Do you know at what point your DBMS stops loading the whole table in memory? Do you know what datastructure and algorithm it's using when you do a LIKE query?

Cache thrashing also can't be audited in code alone without understanding the architecture that the the app is going to be deployed on. It's highly unlikely that the servers will have the same processor cache sizes, memory sizes, and numa architecture of the dev's laptop.

Load balancer is something a developer should know about as well. A developer has to consider the behavior required by the application (e.g. backend session persistence, headers injected, etc).

>please explain, how would dealing with kernel tcp tuning part-time help Joe Random developer write better code?

Joe might learn that connections aren't as cheap as he thinks and maybe it isn't a great idea for each client to require 50 connections for the app to function. He might also learn that TCP isn't very efficient on high bandwidth, high latency, lossy networks and decide to switch to UDP with error correction.

Long story short, a good developer should know everything about the environment in which the app is intended to run. "It performed ideally on my laptop" is throwing code over the wall.

Civil engineers don't design bridges without understanding where the bridge will go. The same applies to software.

Re: Software Developers Should Have Sysadmin Experience

#310

Post by bandrami » Mon, Jan 09, 2017, 5:37 PM UTC

koffiezet wrote: post #285

bandrami wrote: post #196

Earlier quoted context omitted.

It's pretty much impossible to be a sysadmin without writing some software, though I acknowledge we sysadmins have a tendency to use duct tape (there's a reason Perl is the sysadmin language) rather than a more solid adhesive. But whereas I've met tons of developers who have never maintained an installed system, I've never met a sysadmin who has not written a good deal of code.

> there's a reason Perl is the sysadmin language That would be mostly Python these days. If someone here would touch Perl on my systems, they're gonna have a very bad day. Sure I wrote stuff in Perl back in the days, but those days are over. I see this as Perl's problem: to be good enough at Perl, you'd have to frequently use it, but Perl is in my eyes only suitable for quick hacky run-once scripts - which should not…

IDK. My blog is in catalyst; I really like it for that.