Live data from Hacker News

Software Developers Should Have Sysadmin Experience

blog.professorbeekums.com

381–390 of 424 posts

Re: Software Developers Should Have Sysadmin Experience

#381
Wherever I went, I have seen the war between developers and sysadmins being fought. In enterprise the sysadmins have the upper hand, in frontend, mobile, SMB the developers.

Developers don't have the burden of maintaining and upgrading multiple applications on the same server, with conflicting dependencies. 24/7. Sysadmins don't feel the pressure by end-users to deliver new features yesterday.

At the moment developers are winning because Docker. But when they grab that responsibility they will be the ones called when their 50000 containers are being hacked because vulnerabilities. And they will be under fire for the system being down and that costing the company a lot of money, so everybody breathing down their necks.

Re: Software Developers Should Have Sysadmin Experience

#382

Earlier quoted context omitted.

But they don't both suck. Newer tools reflect the needs of newer problems. Like, I can point to exactly what I get from my Chef/Cfer stack--I can wrangle hundreds of concurrent machines that need to update to a policy while also getting, in a standardized and predictable format, a full itemization of all changes applied to all systems. I can leverage the community, too, to help me bootstrap new features and functiona…

It's funny you'd quote Chef (or puppet), because they've been replaced by Ansible/Salt. I spend a lot of time evaluating new tools, possibly more so than anyone else in this tread. There are very few novelties which solve a problem notably better than what existed before AND are reliable enough for serious usage AND don't come with so many drawbacks that it nullify their interests.

Chef hasn't been replaced by Ansible or Salt. Each tool has a different value prop that makes sense in different contexts. I might use Ansible in some environments where I wouldn't use Chef--there aren't a ton of these environments, IME, but I've chosen to use Ansible in the past despite it not being my favorite for reasons like this. I would use Chef in other contexts where Ansible's ecosystem or tooling is lacking--I find that chef-zero and berkshelf are a better way to self-bootstrap on AWS when using CloudFormation, for example--or where I desire the greater flexibility and expressiveness of a Ruby-based DSL.

The black-and-white thing you're painting is, if I'm gonna be frank...a little messed up. I mean this in the best of ways: please enhance your chill. It's not that bad out here.

Re: Software Developers Should Have Sysadmin Experience

#383

Earlier quoted context omitted.

This has actually been a very insightful response to me. After reading your comment it now occurs to me that Docker and other container systems are actually a huge organizational tool. One issue I have encountered at companies is keeping the IT and development departments on the same high level organizational incentives to keep political barriers from coming up between them (and conflicts arising). Containers can hel…

At almost every shop I've ever seen, "sysadmins" are also the ones whose responsibility it is to at least attempt some sort of security practice and business continuation. Which opaque, "just run it" containers actively fight against. Did the developers actually audit what they've pulled in as dependencies? Did they make sure that they can be rebuilt if whatever package source goes away? Where is everything documente…

I've never seen sys-admins verify security of dependencies outside of the major dependencies (like the language VM version). Security wise they have been much more concerned with the security of data storage systems (databases, elastic search, etc...), the operating system, and the network in general.

Security of the application is very much so responsibility of developers, not system admins, as the developers have the best point of view to understand the implications of the software they are developing/integrating with.

If there are routine violations of security at the application level that aren't being caught by developers working with those systems then the company as a whole needs to sit down and make sure the development teams have the proper security procedures in place, because putting a department in charge of security that has all accountability but no power to remedy the situation is a recipe for political fights between departments and a disaster. Proper code reviews and team leads with experience should be able to catch more security issues than sysadmins will.

If your sysadmins are in charge of security review of the application then they have to be in charge of security review of every low level dependency at the individual package level. Otherwise your developers won't think about it because it's not their problem (IT will review and let me know if anythings bad) and it encourages them to lack accountability of the security of their own software.

Re: Software Developers Should Have Sysadmin Experience

#384

Earlier quoted context omitted.

> Then you were lucky. I worked in 3 Microsoft dev shops and 1 that had a mixture. As far as I know I hadn't heard of anyone having issues getting it installed except for the rare, occasional error that could be Googled and fixed. I'm not sure I'd call that luck, sounds like you just had a bad experience. But yeah back in the day it was a stack of discs (I think the last disc version I used had 2 discs for visual stu…

There were 40 DVDs in a MSDN subscription in 2004, so not sure what you're talking about, you might have had the cheapo one?

MSDN subscription has (almost) all Microsoft software - its tangential to Visual Studio. MSDN Documentation is seperate and was only a few CDs.

Re: Software Developers Should Have Sysadmin Experience

#385

Earlier quoted context omitted.

> They aren't kids playing with toys that we need to nanny over In my experience (university), yes they are, and they should do that at home. Why do you need the latest bleeding versions in the first place? In my sysadmin experience, people believe software gets bad and deprecated as soon as the glory next breaking version appears. I don't think I need to argue why this is an illogical stance. With my developers hat…

For one thing, if there is any place to treat software like a toy and to play around with the latest version, it would be a college or university. I don't particularly care about having my software on the latest version. I personally prefer using the old version for six months while the newest version gets the bugs worked out of it. I know sysadmins value reliability and security, but it's really frustrating when eve…

I said it elsewhere, but personally I'm a Debian stable evangelist. There is one major upgrade every 2 years or so. It often goes without friction. The rest is mostly security updates. Breakage between major upgrades is very rare.

I don't think this thread is about "maintaining a modern code base" at all. Whatever that should mean -- My impression is you've fallen victim to the hype train.

In my perception the thread is about always catching up with the latest and greatest. Would you say in all earnest that my code is not modern because I make a point of developping against solid standards and not constantly longing for things that are not in my distribution (the software there is usually 0.5-2.5 years old)?

You can check some of my code at https://github.com/jstimpfle. Is it "not modern"? I'm a reasonable but not outstanding developer, and not saying that everything will work on your computer (since I'm usually the only tester) -- but I'm pretty sure I can get everything there to run on your computer with minor effort.

> When you upgrade often, it is easier to read the documentation and find where changes might break something, and when things do break it is easier to fix them.

No. Breakages are less frequent because the software is not brand new, and they are better known because all people using the stable release are on the same version. Documentation comes with the distribution, but I don't have any problems googling it by giving the version string either.

> Upgrades always have the potential to break things, but when you keep up with the upgrades it is easier to achieve the stability and security goals the sysadmin wants.

This thread was never about security and I don't approve. I don't think you are familiar with the concept of a stable release.

> Upgrades that jump over several versions at a time are a nightmare to debug, and it creates a lot of technological debt that you have to work out later.

No. If you develop against solid standards you have less breakage. It's not about incompatibility with the most recent versions. That would be a stupid idea. It's about compatibility with releases other than the latest and greatest. This means not depending on the hot new features that are only in these versions, simple as that.

Re: Software Developers Should Have Sysadmin Experience

#387
post #50

This might be controversial, but I don't think you get to be a half decent developer without being a reasonable sysadmin. Maybe my experience is unusual, but I've never worked anywhere that the sysadmins knew more than the developers about how best to run their code in production. And when things go wrong with it how best to find the cause of the issue. And I've never thrown code over a wall without having tested it…

>Often this stuff goes more smoothly where the developers (I mean, it's not as though if you're going to drop one of the two groups of people it's going to be them going) manage production and there aren't people with separate job titles and the resulting friction between them. This is not personal criticism but you know how I know you're not working in a highly regulated environment? Check out the Carnegie Mellon Ca…

> Have you ever had to explain to someone that a java out of memory error had nothing do with the fact that linux is using otherwise idle memory to buffer i/o and that they're misreading top output? That the actual issue is their object management and just increasing the JVM's heap size is at best a bandaid?

Is there somewhere to read about this, I this might have come up with one of our projects. (At the time, from googling, I suggested they try mark and sweep - I didn't really have any idea but was of the opinion they had lots of small objects.) I don't have much experience in Java but was trying to be helpful!

Re: Software Developers Should Have Sysadmin Experience

#388

Earlier quoted context omitted.

For one thing, if there is any place to treat software like a toy and to play around with the latest version, it would be a college or university. I don't particularly care about having my software on the latest version. I personally prefer using the old version for six months while the newest version gets the bugs worked out of it. I know sysadmins value reliability and security, but it's really frustrating when eve…

I said it elsewhere, but personally I'm a Debian stable evangelist. There is one major upgrade every 2 years or so. It often goes without friction. The rest is mostly security updates. Breakage between major upgrades is very rare. I don't think this thread is about "maintaining a modern code base" at all. Whatever that should mean -- My impression is you've fallen victim to the hype train. In my perception the thread…

> I said it elsewhere, but personally I'm a Debian stable evangelist. There is one major upgrade every 2 years or so. It often goes without friction. The rest is mostly security updates. Breakage between major upgrades is very rare.

That's fine for an OS, but what do you think business customers would say if you said "sorry, that feature won't be added until the next release in 2 years time"?. That's were tools like pip come in, they let the software move faster, which it often needs too.

Re: Software Developers Should Have Sysadmin Experience

#389
post #388

Earlier quoted context omitted.

I said it elsewhere, but personally I'm a Debian stable evangelist. There is one major upgrade every 2 years or so. It often goes without friction. The rest is mostly security updates. Breakage between major upgrades is very rare. I don't think this thread is about "maintaining a modern code base" at all. Whatever that should mean -- My impression is you've fallen victim to the hype train. In my perception the thread…

> I said it elsewhere, but personally I'm a Debian stable evangelist. There is one major upgrade every 2 years or so. It often goes without friction. The rest is mostly security updates. Breakage between major upgrades is very rare. That's fine for an OS, but what do you think business customers would say if you said "sorry, that feature won't be added until the next release in 2 years time"?. That's were tools like…

So what exactly are the latest-and-greatest libraries that you absolutely need to implement your own business critical features?

Please list more than only one. It's simple to make exceptions for exceptional requirements.

Re: Software Developers Should Have Sysadmin Experience

#390

Earlier quoted context omitted.

Pick your poison. If you run into a bug or problem with a 3rd party component (open source library, commercial tool, whatever), one of the first things they are going to ask you to do is upgrade. The fact you're on an old version of some library is an easy (and sometimes correct) scapegoat for problems. Put yourself in the 3rd party's shoes: if you spend a bunch of time trying to fix a problem that turns out to be a…

Upgrading the version of the library wouldn't be a problem if the concept of stable ABI's were as prevalent as it was 15 years ago. Back then, the major.minor version number system was used as a signal that it was safe to upgrade to a newer version of a library without worrying that the entire application stack was going to come falling down around itself because the developer of said library decided to rework some p…

Was there no automated testing that allowed you to go from 1.52 -> 1.53 with some degree of confidence?
Post reply on HN