Software Developers Should Have Sysadmin Experience • Hacker News Board

Re: Software Developers Should Have Sysadmin Experience

#171

Post by berntb » Mon, Jan 09, 2017, 2:00 PM UTC

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

Interesting. Is this because the Perl ecosystem is more mature or because of the philosophy of backwards compatibility?

The "backwards compatibility" philosophy isn't so explicit for the ecosystem, mostly the language? Is the test-on-install-by-default making a big difference there?

Re: Software Developers Should Have Sysadmin Experience

#172

Post by tannhaeuser » Mon, Jan 09, 2017, 2:02 PM UTC

kowdermeister wrote: post #49

peterwwillis wrote: post #5

Earlier quoted context omitted.

That question doesn't make any sense at all. That's exactly the same as saying: with the rise of do it yourself home repair, how often are plumbers different from carpenters?

The question is great. DevOps is a field that I really don't want to touch as a developer. I build frontend and backend as well, design databases and focus on UX and configure a LAMP stack, but that's where sysadmin ends for me. What I don't want to mess with is the "DevOps" tools, like Docker and rest of cloud orchestration stuff. It's a very complex field and I couldn't care less. If I work on a project like that t…

DevOps, at its core, is just treating your configuration files and artifacts (IP/DNS, firewall rules, mail config, service and web app artifacts, libs and other stock software, SQL/NoSQL deployment descriptors, queue configs, users/roles/certificates) as development artifacts, and have an automated, reproducible deployment procedure. Going to a box and manually configure something in an ad-hoc fashion using fancy GUIs is considered a no-go. Rather, you're supposed to craft/test your configuration artifacts and then check it in to eg. git. If anything, this should be a workflow familiar to developers, hence "DevOps".

Docker and other Docker-like containers like Rocket don't have anything to do with DevOps per se, but they facilitate easier automated deployment of said configuration artifacts. Technically, Docker & Co. are just remedies to classical DLL hell situations, and have been used mostly for development and software evaluation purposes. Though they technically don't provide much more isolation than simple chroot-jails, increasingly they're also used in place of VMs because of density reasons, eg. because you can run a whole bunch of services on a single machine with less footprint than VMs (on Linux, that is).

The frenzy with Docker and the other cloud orchestration stuff, as you say, is IMHO mostly because of Sillycon startups with insane amounts of venture capital buying their way into people's minds.

Edit: ok, the last remark was a bit snarky; sure there's a need of mass-digestible tool for collaboratively editing DevOps artifacts and deployment plans, but I don't know any I like; feel free to point me to one (preferably without JSON and/or yaml configuration files)

Re: Software Developers Should Have Sysadmin Experience

#173

Post by gsnedders » Mon, Jan 09, 2017, 2:04 PM UTC

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

> You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they were release candidates rather than releases.

Releasing an RC often results in nobody using it and hence not finding the bug even in several weeks, but it gets caught almost instantly in a release… At least, that's my experience in shipping various RCs that have led to next-day regression-fixes once it does ship.

While yes, better testing would solve such issues, but at some point the line has to be drawn as "good enough", because there's ultimately a limit to what is reasonable.

Re: Software Developers Should Have Sysadmin Experience

#174

Post by bandrami » Mon, Jan 09, 2017, 2:06 PM UTC

berntb wrote: post #171

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

Interesting. Is this because the Perl ecosystem is more mature or because of the philosophy of backwards compatibility? The "backwards compatibility" philosophy isn't so explicit for the ecosystem, mostly the language? Is the test-on-install-by-default making a big difference there?

That's a good question. I think it's not a coincidence that CPAN predated the widespread use of distributed source control systems whereas pypi and gems blew up just as mercurial and git were unseating svn and cvs. It's a different release philosophy (remember, in the 1990s you often didn't even get to see pre-release CVS commits of open source projects; that was an innovation of OpenBSD).

I also think the widespread use of VPSs rather than accounts on shared servers (again, containerization) was a factor. In the 90s and early 2000s, you usually (even in a corporate setting) had an unprivileged account on a server with a given version of apache and perl, your own cgi-bin directory, and possibly some latitude on a personal CPAN install directory. The lack of containerization meant you had to compromise between using newer software and breaking existing use cases.

So I guess I think it's not so much about Python vs. Perl per se but about the technologies available when those languages became popular among developers.

Re: Software Developers Should Have Sysadmin Experience

#175

Post by base698 » Mon, Jan 09, 2017, 2:07 PM UTC

tajen wrote: post #66

nokya wrote: post #37

I'd say there is too much effort in reasoning on the wrong problem. What worries me the most is the 'why': why do (too) many software developers don't know about sysadmin? I have been involved as a consultant in large software projects in the last two years and a vast majority of money lost in delays and bugs was caused by devs not understanding: 1) the difference between virtual memory and physical memory 2) the dif…

> the difference between virtual memory and physical memory For my defense (I'm a dev), OSes don't make it clear. Mac OS becomes extremely slow when I load a big virtual machine and yet displays "Swap 450Kb, 500Mb RAM free". Or with a sole text editor open after a long session it may say "Swap 750Mb". In both cases my logic tells me the swap and free memory should display the opposite, so I can't match my knowledge w…

It's more than just swap. Second answer has more detail: http://stackoverflow.com/questions/4970421/whats-the-differe...

Udacity has a pretty good OS summary course. I had personally forgotten what a TLB was until I watched watched it.

Re: Software Developers Should Have Sysadmin Experience

#176

Post by arca_vorago » Mon, Jan 09, 2017, 2:07 PM UTC

TillE wrote: post #107

golergka wrote: post #61

Just a small nitpick with the article: it probably meant "backend software developers". Because a graphics developer who mostly writes shader code would surely benefit more from artist experience.

It's incredible how apparently 99% of HN are web developers who don't even bother considering that other things exist. It's not at all representative of the larger industry.

Very much this. I don't pretend to be a dev, but I listen and work with them to accomplish business goals. Sysadmin work is very much more business management oriented in the real world, but to see so many Devs say "I know how to admin my software in prod therefore I am sysadmin too" just makes me realize just how little the Hn/sv web startup dev knows about real world business systems administration for established business.

Once again, like I said in my other comment, even in web startup land, this is a failure of management, in particular CIOs and CTOs, and the management that selects them.

The problem is that companies can usually manage to survive for quite a while ignoring these problems. There is little short-term feedback mechanism for duct taped systems... Until that day when shit starts dying and no one can fix it. Or a cryptovariant hits the file server. Etc.

Re: Software Developers Should Have Sysadmin Experience

#177

Post by ChemicalWarfare » Mon, Jan 09, 2017, 2:08 PM UTC

cs02rm0 wrote: post #50

This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. And I've never thrown code over a wall without having tested it…

DevOps approach definitely shifts some sysadmin-type roles towards the development teams. That said though there are things that belong in the realm of sysadmin responsibilities - both "old school" like let's say setting up DB replication, VPNs or a puppet master server(not that "old school" but still) and "new" - things like Kubernetes and Fleet/CoreOS for example have plenty of configuration/maintenance complexity that is better suited to be handled by a dedicated sysadmin.

Re: Software Developers Should Have Sysadmin Experience

#178

Post by FLUX-YOU » Mon, Jan 09, 2017, 2:09 PM UTC

pjmorris wrote: post #12

Should sysadmins have software development experience (e.g. DevOps)? For what values should 'X have Y experience?' Should we go as far as... "A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new proble…

Development is sadly the job that is becoming the kitchen sink of "You ought to have skill X" where X is anything from sysadmin, computer science, business, security, product, bare-metal, networking, infrastructure, management, statistics, operating systems, math, social, and industry knowledge.

If you aggregate all of the "Every developer should know X" posts and blogs, the list would probably be very long. It only promotes shallow signaling instead of actual competence (I only need to know enough about X to make people think I know about X).

Meanwhile, your salary will still only compensate you for one skill set: software development.

Re: Software Developers Should Have Sysadmin Experience

#179

Post by vinceguidry » Mon, Jan 09, 2017, 2:09 PM UTC

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

> Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project".

As a Ruby developer, I can only laugh at this particular example. No Ruby project I've ever worked on ever upgraded their gems midway through a project, much less the version of Ruby. Developing procedures for this kind of ongoing maintenance is just way too much to ask.

This stuff tends to get done years after the original devs have all moved on. Maybe they tried that kind of thing back in the early days, before I started working with Ruby, definitely not today.

Re: Software Developers Should Have Sysadmin Experience

#180

Post by bandrami » Mon, Jan 09, 2017, 2:11 PM UTC

gsnedders wrote: post #173

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

> You'll also notice that none of these releases are tagged "-rc1", etc., though the fact that regressions were fixed in a new bump the next day means they were release candidates rather than releases. Releasing an RC often results in nobody using it and hence not finding the bug even in several weeks, but it gets caught almost instantly in a release… At least, that's my experience in shipping various RCs that have l…

Oh, definitely, and I want to be clear I'm not wagging my finger at pip here; it's a good project. It's just that for somebody like me on the far side of 40 (and sysadmins in general are a grayer cohort than devs), that's absolutely not a release tempo that I grew up with. I still shudder when I see a Dockerfile that begins "FROM whatever/latest..." because I have no idea if whatever/latest is going to be the next leftpad.