Software Developers Should Have Sysadmin Experience • Hacker News Board

Re: Software Developers Should Have Sysadmin Experience

#401

Post by jstimpfle » Tue, Jan 10, 2017, 3:21 AM UTC

flukus wrote: post #398

jstimpfle wrote: post #395

Earlier quoted context omitted.

You are still not providing evidence. You're also making bold claims ("any app from just about any...") that my personal experience can just not validate. It's very easy to write applications without fancy dependencies. Recently I did algorithms, systems and applications in C, C++, python, sdl, alsa, X11, Unix shell scripting, lp_solve, and some web programming in python, javascript, sqlite3. All rock solid and stabl…

.net MVC, nUnit 3, jquery, knockoutjs, and nHibernate to name a few. I could name a dozen similar tools on the java stack. Pretty soon I'll be experimenting with rust with libui. >There are 2050 python3-* packages on my system. Not that I think it's a good idea to use most of them. What's "compatible"? 2000 of them are random versions someone made a package for that are unknown to the core team and probably not recei…

If you consider making websites, and tools and web applications for internal processes, academic... I also did some contract work where I created a server and a client for displaying advertisement media to commodity screens, in a tiled fashion. The tools were all there, C, python, bash, X11, some media libraries, the versions there were all fine (I worked around a bug in mplayer though).

jquery, knockout. I don't know a first thing about bundling dependencies for client-side javascript code (not going super fancy there, don't need jquery, knockout - tried writing single page apps by hand but they are hugely complex) but anyway, isn't that independent from server installations? Don't you bundle these libraries in-tree? If so, it doesn't relate to the discussion.

.NET... It's MS, do you run on Mono? How does the question of requiring the latest version apply?

Re: Software Developers Should Have Sysadmin Experience

#402

Post by bthornbury » Tue, Jan 10, 2017, 3:40 AM UTC

deadbunny wrote: post #140

bthornbury wrote: post #77

Earlier quoted context omitted.

Not the same at all. Let me rephrase the question. Reading the article, the sysadmin is described as the person who constructs and maintains the environment the code runs in, including the physical machines, but also things like dependency management (I infer). Given that just about everything except the physical machine can be handled through devops automation (from experience), and those things don't require "speci…

Thinking like this is what leads to tens of thousands of Mongo databases being exposed online. Anyone can follow a "how to install X" guide and install a few packages and get a working config to get going and then automate it with their tool of choice but that doesn't come anywhere close to a production ready system. When your job focus is shipping product that's all you care about, get it out the door as fast as pos…

Getting it out the door will make you learn faster. The choice to stop learning is a personal one IMO.

Re: Software Developers Should Have Sysadmin Experience

#403

Post by bthornbury » Tue, Jan 10, 2017, 3:41 AM UTC

Intermernet wrote: post #87

bthornbury wrote: post #77

Earlier quoted context omitted.

Not the same at all. Let me rephrase the question. Reading the article, the sysadmin is described as the person who constructs and maintains the environment the code runs in, including the physical machines, but also things like dependency management (I infer). Given that just about everything except the physical machine can be handled through devops automation (from experience), and those things don't require "speci…

I started as a dev many years ago, then moved into sysadmin / devops stuff for over a decade, and then moved back to more hands on dev stuff again a few years ago. I think your statement that "those things don't require "special" skills other than docker/ansible knowledge" is over-simplifying things a bit. Writing a production quality Dockerfile or Ansible Playbook requires more than just a bunch of "apt-get install"…

I'll agree that I definitely oversimplified it. I think what I was getting at was that it was a similar set of skills to other types of software development.

Re: Software Developers Should Have Sysadmin Experience

#404

Post by bthornbury » Tue, Jan 10, 2017, 3:51 AM UTC

peterwwillis wrote: post #327

bthornbury wrote: post #77

Earlier quoted context omitted.

Not the same at all. Let me rephrase the question. Reading the article, the sysadmin is described as the person who constructs and maintains the environment the code runs in, including the physical machines, but also things like dependency management (I infer). Given that just about everything except the physical machine can be handled through devops automation (from experience), and those things don't require "speci…

Software is complicated and error prone and it takes a good amount of skill and time to write it well. Personally, I'm not impressed at all with most software made in an Ops environment. I think it usually sucks and that sucky quality, along with the complexity of debugging and maintaining it, makes it a net deficit to productivity. I have been writing software in Ops for a long time and it is just annoying how littl…

> Software is complicated and error prone and it takes a good amount of skill and time to write it well.

This we can agree on.

> Personally, I'm not impressed at all with most software made in an Ops environment. I think it usually sucks and that sucky quality,

Not sure what is an ops environment although I'll agree with the general statement that most software I see sucks and is of sucky quality, but I think that relates back to your first point.

> In fact, this is how Ops used to work. You'd use humans to oil the machinery of the basic automation you put in place, and they were damn good at keeping uptime high and catching problems nobody thought of in their automation. Now people just automate and pray, it seems, and then play catch-up indefinitely, or waste time redesigning.

I'm not proposing that the rise of devops precludes the need for people specialized in it's use and implications. Instead that sysadmin is essentially becoming a branch of software engineering with the rise of devops. When setting up a system IS writing code, why is the "software engineer" different from the "sysadmin"?

Re: Software Developers Should Have Sysadmin Experience

#405

Post by eternalban » Tue, Jan 10, 2017, 4:57 AM UTC

contingencies wrote: post #399

eternalban wrote: post #238

Earlier quoted context omitted.

Of course. (In my opinion, /high software/ has more in common with mathematics, music, theatre-film-dance, and architecture than it has with engineering, and /low software/ is begining to resemble boiler room operations.) But here, as example is my BSEE a.m.: http://eng.rpi.edu/academics And to this day, we hear about "software engineers" and "software engineering". Per my OP: "It seems either your conclusion is held…

Process engineering (~manufacturing) and logistics (~supply-chain) are not dissimilar to modern software workflow. The basic tools (modular management of complexity, discrete processing, statistics, monitoring, redundancy in processes/providers, feedback) are equivalent. In fact, I feel like a huge part of a successful software career is learning to see the similarities in disparate fields and draw from them positive…

Well that is certainly correct but it should be pointed out that one can say that about most organized (psuedo-)industrial production endeavors. But it seems incorrect to posit that that is the 'defining' characteristic of software development.

> In fact, I feel like a huge part of a successful software career is learning to see the similarities in disparate fields and draw from them positive architectural benefits, while keeping other-profession-spire-dwellers properly onside/placated.

Fully agreed. In fact that has been my guiding light in my own approach to software development. To clarify my view, I think software, very much like architecture, is a polyglot yet distinct discipline. It is not engineering. It not mathematical logic. It is not process engineering. It is not logistics (provisioning). Etc. (Just like architecture is not civil engineering. It is not philosophy. It is not art. It is not environmental systems engineering. Etc. It is architecture.)

-- p.s. edit --

I would like to bolster my earlier statement that software development has more in common with architecture, theatre, film, etc., than with engineering:

I would like to propose and roughly define a notion of 'semantic gap'. A sort of soft measure of the degree to which the formally expressible definition of a 'production' falls short of permitting the realization of the 'product' without the intervention of the 'designer'.

With that definition in hand, I propose that "engineering" discipines are those creative productions that have minimized the semantic gap to a degree that permits strict divisions of labor in the production.

Where as the "arts" are those creative endeavors that are faced with an intrinsic constraint on the degree to which the semantic gap can be minimized, and, that this maximally reducible semantic gap requires subjective and/or contextual 'interpretation' of the formally expressed design.

Re: Software Developers Should Have Sysadmin Experience

#406

Post by gaius » Tue, Jan 10, 2017, 6:41 AM UTC

jdbernard wrote: post #337

gaius wrote: post #292

Earlier quoted context omitted.

The new things you need to develop new features are far and far between 99% of web software written these days could fulfil identical use cases on an IBM 3270 from 40 years ago. You enter something into a form and it gets stored in a database. You enter something into a field and it generates a report. That's all Amazon, Facebook, Google, any e-commerce site are. Sure it might be nice to use a new version of that new…

And yet none of those things did arise 40 years ago. All of the nuances of all the code written since then make a difference, despite duplicating "identical use cases".

Amazon was founded in 1994, so over 20 years ago and somehow they managed to succeed without AngularJS 3.7 or whatever the fashion of the month is.

Re: Software Developers Should Have Sysadmin Experience

#407

Post by hueving » Tue, Jan 10, 2017, 7:44 AM UTC

LoSboccacc wrote: post #358

hueving wrote: post #309

Earlier quoted context omitted.

Query optimization can't be audited in code alone. The indexes you need depend heavily on the database system that you are using in production. Do you know at what point your DBMS stops loading the whole table in memory? Do you know what datastructure and algorithm it's using when you do a LIKE query? Cache thrashing also can't be audited in code alone without understanding the architecture that the the app is going…

> Joe might learn that connections aren't as cheap as he thinks and maybe it isn't a great idea for each client to require 50 connections so we're back to point one, you need devs that go trough basic education and stop cheapening out hiring Joe / or Joe should ask a refund from his tuition fees. > snip of stuff that one does not know off the bat sure but it is knowable, it's not exactly hard. database are predictabl…

>so we're back to point one, you need devs that go trough basic education

There is no basic education that covers the associated costs of a TCP connection in the kernel of a modern operating system or in the load balancers it passes through on the edge of the network.

>sure but it is knowable

So you're saying it is important for a developer to understand the infrastructure the code will run on. Thank you

The reason I brought up all of those points is because they are things not covered in CS educations and they hammer "hands off" devs all of the time.

I've worked with tons of junior devs from all kinds of good schools (Stanford, MIT, UC Berkeley, etc) and they almost always get bitten by this stuff because they throw their code over the wall and don't make an effort to understand the operational environment. It has nothing to do with a good education, it has to do with a mindset of not operating in a vacuum.

Re: Software Developers Should Have Sysadmin Experience

#408

Post by gedrap » Tue, Jan 10, 2017, 8:05 AM UTC

JeremyNT wrote: post #142

gedrap wrote: post #65

Earlier quoted context omitted.

>>> The worst sysadmins get in the way of developers. Ones that scale down your CI server to the cheapest, throttled, one the hosting company has, leaving $800/day contract developers waiting for builds that run in 20 seconds on their laptops take nearly an hour. How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain?

> How likely is that the sysadmins were told to 'just make it run cheaper, I don't care' by someone higher in the foodchain? Having worked in ops for > 10 years, this is how it usually goes. The SA's job tends to involve a lot of scepticism and caution. You look for problems and try to solve them proactively. One (often easy) way to solve many classes of problems is to throw hardware at them. Management always pushes…

Thanks for such a detailed response :)

Re: Software Developers Should Have Sysadmin Experience

#409

Post by xorcist » Tue, Jan 10, 2017, 11:53 AM UTC

crdoconnor wrote: post #311

xorcist wrote: post #203

Earlier quoted context omitted.

> why should a sysadmin who doesn't even know Python Because a Python sysadmin has been through all the transitions of packaging systems, all the nasty corners of "backwards compatible" changes, and knows how underlying changes to the operating system will affect your code, what the storage behaves like under load, and why one tech is not "better" than another. If you really hired an admin (cough, "reliability engine…

>You don't know what you don't know. That's the point of regression tests. The sysadmin also doesn't know . Unless he's the one writing the tests (and IME he's not) or he's painstakingly regression testing everything by hand (trust me, he's not doing that either), making him a gatekeeper for all library upgrades achieves very little except adding bureaucratic friction.

Look, do you want a gatekeeper or not? For your small little web project you don't need it, and a little downtime probably isn't catastrophic. But as soon as you are under audit rules you need it, and we call this specialized role the admin. When you grow bigger this will likely branch out to a dedicated change manager, at which point you hopefully have other specialized roles for security as other things as well.

I understand this does not make sense when you are not more people than can fit around a table, but as you grow you will feel the need for more and more specialized roles to fit the changing requirements. The first specialized role is probably the sysadmin (devops, reliability engineer, whatever you call it) and he or she should preferrably be the one on the team with the most knowledge of how things work "under the hood" because that person is the one that can save you when things go haywire. Unless you trust this person to be more knowledgable than you are in those areas, as they rightfully should be, you're going to have a problem.

Re: Software Developers Should Have Sysadmin Experience

#410

Post by crdoconnor » Tue, Jan 10, 2017, 12:48 PM UTC

xorcist wrote: post #409

crdoconnor wrote: post #311

Earlier quoted context omitted.

>You don't know what you don't know. That's the point of regression tests. The sysadmin also doesn't know . Unless he's the one writing the tests (and IME he's not) or he's painstakingly regression testing everything by hand (trust me, he's not doing that either), making him a gatekeeper for all library upgrades achieves very little except adding bureaucratic friction.

Look, do you want a gatekeeper or not? For your small little web project you don't need it, and a little downtime probably isn't catastrophic. But as soon as you are under audit rules you need it, and we call this specialized role the admin. When you grow bigger this will likely branch out to a dedicated change manager, at which point you hopefully have other specialized roles for security as other things as well. I…

>Look, do you want a gatekeeper or not?

No, ideally not - that's the idea behind https://en.wikipedia.org/wiki/Continuous_delivery

Where gatekeepers are required (because regression testing is not yet fully trusted enough for continuous delivery), QA should be the gatekeeper, not sysadmins.

>For your small little web project

My comments are based upon working on projects with a turnover of > ~1-1.5 million USD / day.

>But as soon as you are under audit rules you need it, and we call this specialized role the admin. When you grow bigger this will likely branch out to a dedicated change manager

Every time I've worked with somebody whose role was "change manager" this role was introduced:

* As a response to repeated downtime in the past caused by some kind of idiocy.

* They were required to "sign off" on releases purely as an added bureaucratic step to cover some manager's ass.

* They never once prevented or caught a production issue.

* They always slowed down releases.

>The first specialized role is probably the sysadmin (devops, reliability engineer, whatever you call it) and he or she should preferrably be the one on the team with the most knowledge of how things work "under the hood" because that person is the one that can save you when things go haywire. Unless you trust this person to be more knowledgable than you are in those areas

Ironically the whole idea behind devops (which I fully agree with) is that it should not be a specialized role - developers and ops teams should be blended.

This is precisely because if the two teams are separate and one throws code over the wall to the other then things will go wrong. Then a manager will insist on a gatekeeper.