The sad state of sysadmin in the age of containers

I am old-school sysadmin and I still thinking in terms of what ldd command tells me, how shared libraries are implemented, how various dlopen based FFIs work, why I need this or that.

The other approaches, such as Java's (where we "abstract out an OS") in practice leading only to a bigger mess, because it boils down to the very same libc, libm, libffi and friends. JVM is an ordinary userlevel program, so it obeys to the restrictions and rules for any other userlevel program. This is the sad truth for Java zealots.

Basically, one cannot ignore an OS (at least when you still want to dlopen cand call the stuff instead of re-implementing it poorly) - it is just a wrong idea, leading to all these ridiculous FS-inside-JVM implementations and other messed up layers of unnecessary, redundant abstractions.

System administration is still hard and it (necessity to think, understand and analyze) cannot be eliminated by some bunch of shell or ruby scripts and wishful thinking.

System administration is as important as ever. Docker and other containers just simplify system administration across many different machines. The standard Unix user land tools are excellent and very flexible, but they are fucking god awful at configuration management. Docker solves the problem of "how do I make sure I have the same versions and configurations of everything on all 500 of my compute nodes without having to lock them down completely?" This question is meaningless if your base system image sucks, so you still need a proper sysadmin to build your Docker images.

Many places are rolling this type of sysadmin work up into DevOps. This scares graybeard sysadmins, because they see DevOps automating them out of a job. What they fail to see is that DevOps is a step up for them: it's an explicit admission that system administration is as important as software development, and needs to be integrated into the software development process and managed through whatever management processes and tools the core dev team uses.

The ultimate driver behind this is a shift in the way technology organizations are managed. A few years ago, you would have functional silos: development, operations, product, etc. that would all contribute to one or more products. Employees reported up through the functional lead, and incentives were doled out based on cost effectiveness. This didn't work well. So what started happening is that engineering executives began building product-focused silos instead. A development manager is no longer in charge of just software developers, but also QA, scalability and deployment. If the operations folks fuck up the deployment, the development manager gets chewed out about it. So the dev manager is going to bring as much of that under her control as she can.

Docker/Maven/etc. are the abstraction layer between the teams that manage the infrastructure (physical servers, VMWare pools, storage, network, etc) and the teams that manage the applications. This is no excuse for bad sysadmin practices; you still need good sysadmins in the DevOps role. But here's the kicker: DevOps often pays more than system administration! And if you're a SME in a very specific thing (say, Cassandra administration) you can be in a support role across a number of different teams, making sure their DevOps folks deploy Cassandra in a sane way.

(Yes, I realize all of this is centered on huge companies with massive engineering organizations. Small organizations have always required sysadmins to wear multiple hats, so none of this is new.)

> You cannot reasonably expect developers to package and maintain all their dependencies properly.

Why not? This is exactly what developers are expected to do. Every developer must manage dependencies for their application to work.

What you mean is they can't be reasonably expected to do it well.

In most situations, this is truly a trivial amount of investment relative to the overall cost of developing and maintaining an application.

There are ecosystems which make the task easier or more difficult/annoying, but that cost should be accounted for when choosing which development platform to use.

And before "curl | sh" it was download from freshmeat and run "tar xfz; cd; make install".

It's not really better. Fact is we are running huge and complicated frameworks with lots of dependencies. These technologies are new and evolve fast. Distros don't have enough volunteers to decouple this mess and thus fail to provide stable packages. There is a good chance nobody wants an old version of hadoop anyways.

Containers are a whole other problem. Always bothered me that no one cares about building these images themselves. The documentation is there, you can build your own docker/vagrant/... containers and vms. It's just nobody seems to care anymore? Sometimes I don't even know where these images come from, distro, community, ...?

Judging by the sheer number of frameworks we have, everyone who went through that process actually ended up writing one, or five.