The sad state of sysadmin in the age of containers (2015) • Hacker News Board

Re: The sad state of sysadmin in the age of containers (2015)

#201

Post by Sir_Cmpwn » Wed, May 16, 2018, 6:09 PM UTC

eropple wrote: post #19

megaman22 wrote: post #14

Earlier quoted context omitted.

But I just don't understand why we have to have 47 half-built over-complicated build systems or job runners or whatever the new fad term is for every language, when there's something that does what they all do, is battle-tested, and has been around for decades. Everyone repeat after me. Makefiles are not scary. I can write a shell script. Do I really need to learn grunt/gulp/webpack/npm/rake/fake/maven/gradle/ant and…

Makefiles aren't scary. But they're also not particularly good. I use Rake (or Gulp, or whatever) because then I can use Ruby (or JavaScript, or whatever). Shell plumbing is fine for informal and small-scale stuff, and I make my code conform if somebody down the line (who may be me) wants to get out their duct tape, but the world is more complex than what /bin/sh can see. Shell is the lowest common denominator. Expec…

>I use Rake (or Gulp, or whatever) because then I can use Ruby (or JavaScript, or whatever).

This is an anti-feature. When you need this, your build is too complicated.

Re: The sad state of sysadmin in the age of containers (2015)

#202

Post by kazen44 » Wed, May 16, 2018, 6:09 PM UTC

Animats wrote: post #179

If you thought having to deal with old COBOL programs was a problem, it gets much worse. When there are 10 year old containers in production, and the parts to rebuild them are long gone, then you have a real problem. There's a reason that Google has internal systems which can and do rebuild everything from source.

This is actually a very important point that gets ignored a lot.

What will happen if public container images are no longer available for building your application?

Your in a for a world of hurt if a major dependency of your application doesn't work anymore because someone pulled the image.

Re: The sad state of sysadmin in the age of containers (2015)

#203

Post by commandlinefan » Wed, May 16, 2018, 6:10 PM UTC

loteck wrote: post #23

The clearest explanation of why this happens is at the end: Before, admins would try hard to prevent security holes, now they call themselves “devops” and happily introduce them to the network themselves! 1) The merging of devs into the sysadmin role was a product of: the work of sysadmins (particularly systems change control and security compliance) not being valued in our culture. 2) Devs delighted to be free of th…

> shackles placed upon them by sysadmins

Well, potentially unpopular opinion here, but an awful lot of sysadmins brought their looming obsolesence on themselves. I'm an app (as in "a program that runs on a computer", not an iOS add-on) developer, always have been. I get requirements from the business types, code it up in vi or Eclipse or whatever, get it working, and then they (the business) want to deploy the working app out to production so people can use it and the business can make money off of it. And, for decades, sysadmins have been a brick wall of pure hostility. They're not all like this, but a lot more are than aren't. Like, I get it - you're overworked and the demands on you are unreasonable. Yeah, me too. But I just work here, man. You're right, I don't know how to do your job, that's why I sent you an e-mail asking you what steps are needed to deploy an app into production since it's not documented anywhere. But rather than just tell me what you need so I can go gather that up, you're going to unload on me because you feel overworked and unappreciated, but you're sure as hell not going to unload on a manager or somebody with actual power, you're going to take it out on the developers who have no pull or voice.

Re: The sad state of sysadmin in the age of containers (2015)

#204

Post by mcguire » Wed, May 16, 2018, 6:10 PM UTC

tw1010 wrote: post #52

Isn't this just an example of automation? Other than sunk cost, by what argument is it reasonable that the sysadmin job should in any way deserve to be protected or aught to continue to be a thing? To my ears containers sounds like they've kind of solved the problem.

https://xkcd.com/908/

Re: The sad state of sysadmin in the age of containers (2015)

#205

Post by seorphates » Wed, May 16, 2018, 6:12 PM UTC

ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

Grumpykins here. I think the term "Modern devops" sort of nails it but not quite how you used it. Most departmental/enterprise sys admins/engineers of lore that had even the slightest necessity for life outside the box scaled anything resembling automation to its breaking points. Combined with knowing and serving the reasons for their existence - developers, users etc., and devops is nothing new - it is now simply the necessary manifestation of progress at scale (albeit positively devoured by managers speaking business, not entirely unlike "agile").

"Use what works" definitely presents a lot more choices these days and likely will forever more.

"Use what works well" is something different where "well" implies helpful, dependable, predictable, manageable and so on that will continue to scale with your needs. Only breaking things down the "old-school" way will lead towards success, stability, security and life outside the box.

Good devops is still, primarily, good engineers engineering good things, for themselves and others.

Granted the article is from 2015 but my impression is author is not just cranky, but scared.

Re: The sad state of sysadmin in the age of containers (2015)

#206

Post by commandlinefan » Wed, May 16, 2018, 6:12 PM UTC

borplk wrote: post #198

mr_tristan wrote: post #91

This is definitely a rant that obscures the underlying point: the introduction of _untrusted_ or _unreliable_ network resources, frequently hidden in a string of dependencies. I'm baffled how often I see an someone throw this sort of craziness - "go fetch this thing from some random third party" - into very important places, such as the startup procedures of a container. It's something I see in a culture of the two p…

At the end of the day it comes down to the fact that businesses just simply don't care (Equifax etc). They like the idea of security and that's where it ends. In many places if you try to "do things right" you will get fired in two months for being too slow/strict and they will happily replace you with a clueless easily trusting person who "goes and fetches things from random third parties". Many times they get lucky…

> just simply don't care (Equifax etc).

And, honestly, why should they? Security breaches have yet to hurt an actual company (they hurt users plenty, but not the organization that's actually responsible).

Re: The sad state of sysadmin in the age of containers (2015)

#207

Post by Karrot_Kream » Wed, May 16, 2018, 6:12 PM UTC

eropple wrote: post #19

megaman22 wrote: post #14

Earlier quoted context omitted.

But I just don't understand why we have to have 47 half-built over-complicated build systems or job runners or whatever the new fad term is for every language, when there's something that does what they all do, is battle-tested, and has been around for decades. Everyone repeat after me. Makefiles are not scary. I can write a shell script. Do I really need to learn grunt/gulp/webpack/npm/rake/fake/maven/gradle/ant and…

Makefiles aren't scary. But they're also not particularly good. I use Rake (or Gulp, or whatever) because then I can use Ruby (or JavaScript, or whatever). Shell plumbing is fine for informal and small-scale stuff, and I make my code conform if somebody down the line (who may be me) wants to get out their duct tape, but the world is more complex than what /bin/sh can see. Shell is the lowest common denominator. Expec…

> Shell is the lowest common denominator. Expecting everything to at all times be written in and for that lowest common denominator is not reasonable

Is it that hard to learn shell? Why is it so painful? What makes it the "lowest common denominator"? I use it all the time, but I admit at work I am one of the few.

> Expecting everything to at all times be written in and for that lowest common denominator is not reasonable. We're a tool-using species and we refine tools over time to make them better.

This is too vague. What makes a Ruby based build or JS based build a more "refined" tool? It sounds like familiarity is the real issue here.

> If old tools were sufficient, people would use them because learning new ones is hard.

How many people even know Makefiles these days anyway? The "modern" approach seems to be, learn a programming language and then try to do everything inside of it. Some languages are more interested in this cloistered philosophy than others (like JS).

If anything, I think the reason these build tools keep being proliferated is because nobody wants to learn anything more than the bare minimum to "be productive" (which, depending on what you're working on, can be anything from pushing out customer demos for a company that will never sell, to microservices operating at scale). Learning a language and never leaving its paradigms/comforts is easy.

Re: The sad state of sysadmin in the age of containers (2015)

#208

Post by kazen44 » Wed, May 16, 2018, 6:12 PM UTC

gm-conspiracy wrote: post #133

nwmcsween wrote: post #106

This has been my complaint from day one. Instead of docker have something like ports or pkgsrc and simply create tools that simplify sandboxing, like cgexec, a google kafel -> ebpf filter then all the package manager has to do is well package. Docker IMO is a mudball of concerns that need to be separated.

BSD jails?

jails are actually very mature if you compare them to docker. They have a well working security system and sane networking. (docker just does endless NAT abstraction, which is terrible for certain use cases. Not to mention it breaks a ton of useful networking features)

Re: The sad state of sysadmin in the age of containers (2015)

#209

Post by kazen44 » Wed, May 16, 2018, 6:13 PM UTC

endymi0n wrote: post #153

I think the only reasonable answer is "it depends". After the new updated base OS images for our systems after Meltdown and Spectre were out, it literally took us ten minutes of human and about an hour of machine time to recompile all of our containers, run the tests and deploy them to production on our Kubernetes cluster, replacing the old insecure ones. At the scale of our systems, any grumpy sysadmin would have sp…

> At the scale of our systems, any grumpy sysadmin would have spent at least several days untangling the dependencies and carefully restarting all servers in the correct order after some manually `sudo apt-get`ing (and probably forgetting a few of the lesser used systems).

if you are operating at suchs a large scale, your sysadmin should be automating things (not necesserly with docker mind you).

This automation has been possible on *nix for the better part of a decade by now.

Re: The sad state of sysadmin in the age of containers (2015)

#210

Post by seorphates » Wed, May 16, 2018, 6:13 PM UTC

spiderjerusalem wrote: post #149

ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

What fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?

They'll never die.

Some monkeys will just keep swinging branch to branch. Some monkeys check out the tree.