Live data from Hacker News

The sad state of sysadmin in the age of containers

vitavonni.de

41–50 of 466 posts

Re: The sad state of sysadmin in the age of containers

#41
post #12
post #4

This bothers me as well. Even tasks as simple as adding a repository are now being "improved" with a curl | sudo bash style setup[1]. However, installing from source with make was (and remains) a mess. It may work if you're dedicated to maintaining one application and (part of) its stack. But even then it usually leads to out of date software and tracking versions by hand. Many people have this weird aversion to doin…

> Many people have this weird aversion to doing basic sysadmin stuff with Linux. What makes it weird is that it's really simple. Often easier than figuring out another deploy system. While I agree with the articles main points - the GNU build system is far from simple. Basically an arcane syntax limited to unix-based systems and 5 or 6 100+ page manuals to cover. It doesn't excuse it - but I think it's easy to see wh…

Maintaining autoconf/automake stuff is a pain. Using it is usually as simple as "configure;make;make install".

It doesn't do dependency management though, which is an externalised cost. But that's what rpm/deb do.

I see the attraction of containers and disk image based management. It's much less time consuming. But it's very much the opposite of ISO9001-style input traceability.

Re: The sad state of sysadmin in the age of containers

#42
post #19

Earlier quoted context omitted.

it's also still standard practice in most companies to either read the source code of open-source stuff before deploying it to production Eh, no.

I'd love a specific example? Because in many countries if they're publicly-traded or subject to other regulations (such as Basel, etc.) any company that didn't would be breaking the law.

Which laws are you referring to? Is this a strange interpretation of Sarbanes-Oxley or HIPAA?

Re: The sad state of sysadmin in the age of containers

#43
post #25
post #11

make is the least-auditable build tool imaginable. You don't have to obfuscate a Makefile, they come pre-obfuscated; you could put the "own me" commands right there in "plain" Make. Not to mention that it's often easier to tell whether a Java .class file is doing anything nefarious than whether a .c file is. How many sysadmins read the entire source of everything they install anyway? Maven, on the contrary, is the bi…

Makefiles don't really enter into it and getting software signed by the developer isn't that valuable or useful. The value of debian is not that they package (or repackage) everything into deb files but that they resolve versioning and dependancy conflicts, slip security fixes into old versions of libraries (when newer version break API/ABI), and make it possible to integrate completely disparate software into a syst…

> The value of debian is not that they package (or repackage) everything into deb files but that they resolve versioning and dependancy conflicts, slip security fixes into old versions of libraries (when newer version break API/ABI), and make it possible to integrate completely disparate software into a system.

Maven has exactly the same capabilities as deb does - you can depend on versions, depend on a range of possible versions, exclude things that conflict and so forth. And it puts even more emphasis on fully reproducible builds (with the aid of the JVM) - in that respect it's closer to nix than apt.

> But the sysadmin does feel responsible. He is responsible for the decisions you make, so if you seem to be making decisions that help him (like making it easy for you to get your software into debian) then he finds it easier to trust you. If you make him play whackamole with dependencies, and require a server (or a container) all to yourself, and don't document how to deal with your logfiles (or even where they show up)

Wow, self-important much? Too many sysadmins seem to forget that the system exists to run the programs, not the other way around.

> If you make him play whackamole with dependencies, and require a server (or a container) all to yourself, and don't document how to deal with your logfiles (or even where they show up), how or when you will communicate with remote hosts, how much bandwidth you'll use, and so on

On the contrary, maven makes the requirements much simpler. I have literally one dependency, a JVM, so it can run on any host you like (no need to worry about shared library conflicts with some other application). It needs to download one file (shaded jar) from our internal repo, and execute one command to run it. That's it.

> That's what Maven is. It's a surprise box that encourages shotgun debugging and using ausearch features to do upgrades.

No, it's just the opposite. All the dependencies and project structure are right there in declarative XML. It's what make should have been.

Re: The sad state of sysadmin in the age of containers

#44
post #12
post #4

This bothers me as well. Even tasks as simple as adding a repository are now being "improved" with a curl | sudo bash style setup[1]. However, installing from source with make was (and remains) a mess. It may work if you're dedicated to maintaining one application and (part of) its stack. But even then it usually leads to out of date software and tracking versions by hand. Many people have this weird aversion to doin…

> Many people have this weird aversion to doing basic sysadmin stuff with Linux. What makes it weird is that it's really simple. Often easier than figuring out another deploy system. While I agree with the articles main points - the GNU build system is far from simple. Basically an arcane syntax limited to unix-based systems and 5 or 6 100+ page manuals to cover. It doesn't excuse it - but I think it's easy to see wh…

I have never under stood why some many people are not ok with using the command line.

A few years back we had an issue where a mysql script was over the limit for phpmyadmin - my fairly experienced colleague he was unaware that you could log into the cli and use mysql from the cli.

Re: The sad state of sysadmin in the age of containers

#45
post #11

make is the least-auditable build tool imaginable. You don't have to obfuscate a Makefile, they come pre-obfuscated; you could put the "own me" commands right there in "plain" Make. Not to mention that it's often easier to tell whether a Java .class file is doing anything nefarious than whether a .c file is. How many sysadmins read the entire source of everything they install anyway? Maven, on the contrary, is the bi…

Signed packages isn't about just being signed.

I could sign anything I like, but that doesn't make it any more secure for you to curl it into /bin/bash.

Signatures are about who signs it, and that's not something mvn has solved at all. Mvn is a free-for-all of binary code that very well could own my system, wheras debian is a curated collection of software which the debian maintainers have signed as being compiled by their systems with no malign influence and having met at least some bar.

I'd trust foo.jar signed by debian over foo.jar signed by bobTheJavaBuilder@gmail.com anyday... and mvn only gives you the latter.

So yeah, sure, they're signed, but it doesn't actually matter if you don't take the time to hook into the chain of trust (and believe me, mvn does not ask you to trust your transitive jar dependency 50 down the line) or have a trusted third party (debian) do their own validations.

Re: The sad state of sysadmin in the age of containers

#46
post #40

Earlier quoted context omitted.

I want to work where you've been working. I've never, ever seen this happen.

I want to work there too, except if that means that I will be the one having to read through all open-source code before it is deployed.

And would you be liable for any missed bugs that cause production to break?

Re: The sad state of sysadmin in the age of containers

#47
post #7

So, asking the obvious question: what's the solution to that?

The obvious question is: what's the real problem with that?

A container is a container, as long as docker itself has not bug, the container can only harm the containers content.

Most problems exists in the custom created software in the container (e.g. web-services with bugs, backdoors, ....), this will be a problem for Docker, VMs, Real-Servers, whatever too.

The real problem is the interoperability of different container, if you link the whole data, without any audit, to another container, you can have a problem, but this problem is not docker specific.

Re: The sad state of sysadmin in the age of containers

#48
post #25
post #11

make is the least-auditable build tool imaginable. You don't have to obfuscate a Makefile, they come pre-obfuscated; you could put the "own me" commands right there in "plain" Make. Not to mention that it's often easier to tell whether a Java .class file is doing anything nefarious than whether a .c file is. How many sysadmins read the entire source of everything they install anyway? Maven, on the contrary, is the bi…

Makefiles don't really enter into it and getting software signed by the developer isn't that valuable or useful. The value of debian is not that they package (or repackage) everything into deb files but that they resolve versioning and dependancy conflicts, slip security fixes into old versions of libraries (when newer version break API/ABI), and make it possible to integrate completely disparate software into a syst…

You really nailed it. Among my duties are systems administration for a company that works with a lot of software development vendors. We have a user acceptance team that makes sure that we get what we ordered, that the QC stays at a high level. So functional problems, that's their deal. But they're not sysadmins, they can't easily see what developer choices make administering the servers more complicated, more fragile, more expensive, or more insecure. This shifts my job from the end of the process (here, run this!) to the beginning (hey guys, let's use these tools instead, it'll make everyone's lives easier).

As such I'm very pro containers as they will eliminate a ton of deployment effort and allow me to manage different environments much more easily. But it means that there needs to be a much bigger magnifying glass on the container contents early in the process as opposed to the moment of deployment.

Re: The sad state of sysadmin in the age of containers

#49
post #38

Except that Docker explicitly allows and encourages signing of core infrastructure containers.

Not even close. Docker now has some terribly attempts at signing images on their registry iirc (docker inc signs them for the docker client). There is no option for me, as a user, to build and sign my own image with my own pgp key afaik. My organization might already have a chain of trust, and docker is asking me to ignore that and just trust their signatures (which also only work on dockerhub as of docker 1.5... don…

Docker didn't fuck up when they hired the square guys. http://blog.docker.com/2015/03/secured-at-docker-diogo-monic...

I agree that security and provenance is a real issue in Docker. It is however being worked on, and it will be solved. Presumably we will end up with some sort of app-store like framework with proper signatures and verification.

Docker can't do everything at once. Give them a chance. The new version of the registry is a major step forward in this regard.

In the meantime, what you can do is take redhat's advice. Rather than using a registry to get your images, operate a download site which stores archives of docker images that you can import with `docker load`. You can then also store signatures and check them yourself.

Re: The sad state of sysadmin in the age of containers

#50
post #22

Earlier quoted context omitted.

> As far as I know, it's also still standard practice in most companies to either read the source code of open-source stuff before deploying it to production (binary or build) or get a support contract from someone else who has. Reading the source code might be the case if you are in a large enterprise that can afford to keep those programmers busy or needs to actually vet the code but I honestly doubt it. Just check…

I suspect this is why when many companies decide they want to use open source stuff they contract with companies like GitHub, who absorb the risk of using git, etc, rather than trying to vet the code themselves. I still have a hard time envisioning anyone in any position of authority in any credible company accepting the idea of installing open source software without vetting it one way or the other. If the shit did…

companies like GitHub, who absorb the risk of using git

Except they don't do that at all?

If a bug in git would cost your company a lot of money, why do you think GitHub would be accountable if you have some enterprise deal there?

I still have a hard time envisioning anyone in any position of authority in any credible company accepting the idea of installing open source software without vetting it one way or the other.

Well, that is an entirely different thing. People TEST stuff before deploying it. But that has little to do with code reading or mandatory support contracts.

Post reply on HN