Software Developers Should Have Sysadmin Experience • Hacker News Board

Re: Software Developers Should Have Sysadmin Experience

#181

Post by briffle » Mon, Jan 09, 2017, 2:12 PM UTC

Clubber wrote: post #154

cs02rm0 wrote: post #50

This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. And I've never thrown code over a wall without having tested it…

I agree. I also think it should work both ways. The worst jobs I've ever had is when the sysadmins have the mindset where they own the servers and are unwilling to deviate from what they've read at the behest of the developers. "I will force AV on reads on the developer boxes." "I will install AV on the production DB servers without telling anyone in the development group, then make the developers prove AV was the ca…

what your probably not seeing, is the CIO/CSO screaming at the SA to get AV deployed on every machine in the company, to meet some audit requirement checkbox, or PCI compliance, by the end of the month.

Re: Software Developers Should Have Sysadmin Experience

#182

Post by kapilkale » Mon, Jan 09, 2017, 2:14 PM UTC

I understand the point here; it can be useful to understand the abstractions on top of which you are working.

But given most engineering projects are crud apps without scaling problems, and PaaS companies like Heroku totally abstract away the system adminstration piece for those projects... I certainly wouldn't recommend a new engineer spend any time learning system administration.

Re: Software Developers Should Have Sysadmin Experience

#183

Post by rootusrootus » Mon, Jan 09, 2017, 2:15 PM UTC

berntb wrote: post #171

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

Interesting. Is this because the Perl ecosystem is more mature or because of the philosophy of backwards compatibility? The "backwards compatibility" philosophy isn't so explicit for the ecosystem, mostly the language? Is the test-on-install-by-default making a big difference there?

That would seem to be ironic. As a longtime Perl developer who switched (for pragmatic reasons -- a job) two and a half years ago, my impression is that Python [as a language] is much better suited to a business environment. What makes Perl such a wonderful language and why I enjoy it so much is exactly why it blows as a business language. Python's rigidity is very useful if you ever want someone else to read and understand code written by someone else. So the idea that Perl ends up with the more mature package management is exactly the opposite of what I would have predicted.

That said, I haven't had any more problems with PyPi packages than I did in the past with CPAN. Yes, pip always wants to upgrade itself, but that sort of every-damn-day software upgrade cycle seems to have become quite prevalent, not just in the Python world.

Re: Software Developers Should Have Sysadmin Experience

#184

Post by brianwawok » Mon, Jan 09, 2017, 2:15 PM UTC

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use?

I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading?

I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in setting up the infastructure and keeping it running. They subtract value when they want to tell developers what version of a library to use.

Re: Software Developers Should Have Sysadmin Experience

#185

Post by brianwawok » Mon, Jan 09, 2017, 2:17 PM UTC

vinceguidry wrote: post #179

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

> Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". As a Ruby developer, I can only laugh at this particular example. No Ruby project I've ever worked on ever upgraded their gems midway through a project, much less the version of Ruby. Developing procedures for this kind of ongoing maintenance is just way too much to ask. This stuff tends to get done years…

I upgrade Python deps all the time. Java too. How often depends on the scope of the project.

Re: Software Developers Should Have Sysadmin Experience

#186

Post by SixSigma » Mon, Jan 09, 2017, 2:17 PM UTC

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

Ironically all-aboard-the-update-train was the actual reason I jumped off IIS back in the 90s when I got badly burned by updating Windows NT. Automatic database connection pooling for IIS was dropped and I started getting annoyed phone calls from clients who's websites were dying after updates.

One had to read MSDN every day to keep up with what might break on sites you had no control over.

Re: Software Developers Should Have Sysadmin Experience

#187

Post by bandrami » Mon, Jan 09, 2017, 2:17 PM UTC

vinceguidry wrote: post #179

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

> Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". As a Ruby developer, I can only laugh at this particular example. No Ruby project I've ever worked on ever upgraded their gems midway through a project, much less the version of Ruby. Developing procedures for this kind of ongoing maintenance is just way too much to ask. This stuff tends to get done years…

The time I'm thinking of was a team that wanted to switch from Rails 1.2 to 2.0 along with whichever interpreter bump was required to make that happen (IIRC 1.8.5 to 1.8.6, but this was a decade ago; I'm pretty sure 1.9 hadn't come out yet). Halfway through a project.

Re: Software Developers Should Have Sysadmin Experience

#188

Post by bandrami » Mon, Jan 09, 2017, 2:21 PM UTC

brianwawok wrote: post #184

bandrami wrote: post #164

As a grumpy evil sysadmin, I think the good Professor misses where the real disconnect is, at least nowadays: stack management. Why do things like Docker exist? Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". Why does virtualenv exist? A similar reason. Containerized ecosystems (which is to say basically all of them now) are really a sign of those of us…

That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use? I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading? I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in sett…

why should a sysadmin who doesn't even know Python tell me what version of a library I will use

Because he maintains that installation and you don't? But, yeah, that's why virtualenv, Docker, etc. were invented, because devs kept getting sick of installations having consequences.

What does a sysadmin have to approve or deny an upgrade?

Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Read the changelog and look for implementation problems. Read fora and look for performance problems people are reporting. Yes, beards get stroked during this process, but time and again we see that developers refuse to do this, and wind up coming to us when they break something because of that...

Re: Software Developers Should Have Sysadmin Experience

#189

Post by vinceguidry » Mon, Jan 09, 2017, 2:24 PM UTC

pjc50 wrote: post #120

sagichmal wrote: post #80

Earlier quoted context omitted.

> I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going I'm sorry you've had such an awful experience with sysadmin colleagues that you've developed such a corrosive attitude towards them. I've worked in lots of good environments, where dev/ops was being effectively practiced, and sysadmins there were the most effective force multipliers imaginable.

> awful experience with sysadmin colleagues that you've developed such a corrosive attitude This is going to be a sensitive topic, but can we talk about ""BOFH"" culture somewhere on this thread? (Maybe I'm old and it's now dead, but I think some of it persists) I kind of understand it as a product of working in an environment where everything is urgent and nothing is appreciated, but when sysadmins come to resent th…

When I was reading BOfH a few years ago I got a different gist.

The BOfH is the archetype of someone who is excellent both with technology and politics. When you are in a service role, you have two competing priorities. You must deliver people the things they want, but also keep things nice and stable for yourself so you don't go crazy.

An important dynamic in organizations is laziness vs. intimidation. Political savviness allows you to apply intimidation to get the lazy to do what you want. You can threaten to fire, or raise an issue that could possibly get them fired and even if it doesn't, won't make you look good. The BOfH is someone who can respond to political intimidation with adroit technical interventions to ensure that his second priority, ensuring a smoothly-running system, isn't threatened.

If you read the BOfH stories carefully, you see that the operator knows where his bread is buttered and is careful to remain on good terms with the people who really have the power in the company. The whole thing is a phenomenal read on organizational dynamics.

Re: Software Developers Should Have Sysadmin Experience

#190

Post by vinceguidry » Mon, Jan 09, 2017, 2:26 PM UTC

bandrami wrote: post #187

vinceguidry wrote: post #179

Earlier quoted context omitted.

> Because developers got tired of sysadmins saying "sorry, you can't upgrade Ruby in the middle of this project". As a Ruby developer, I can only laugh at this particular example. No Ruby project I've ever worked on ever upgraded their gems midway through a project, much less the version of Ruby. Developing procedures for this kind of ongoing maintenance is just way too much to ask. This stuff tends to get done years…

The time I'm thinking of was a team that wanted to switch from Rails 1.2 to 2.0 along with whichever interpreter bump was required to make that happen (IIRC 1.8.5 to 1.8.6, but this was a decade ago; I'm pretty sure 1.9 hadn't come out yet). Halfway through a project.

Unreal.

Yep, that sounds like that 'long time ago' I was talking about. Nowadays you can do that, no sysadmin to tell you not to, but nobody bothers.