Live data from Hacker News

The sad state of sysadmin in the age of containers (2015)

vitavonni.de

91–100 of 435 posts

Re: The sad state of sysadmin in the age of containers (2015)

#91
This is definitely a rant that obscures the underlying point: the introduction of _untrusted_ or _unreliable_ network resources, frequently hidden in a string of dependencies.

I'm baffled how often I see an someone throw this sort of craziness - "go fetch this thing from some random third party" - into very important places, such as the startup procedures of a container. It's something I see in a culture of the two person startup just trying to get something out the door. It's definitely "technical debt", and frequently, it won't get removed. Thus, you try to scale up to meet load, and all these new instances go time out on the same external resource that's randomly having problems... boom! At the worst possible time. Never mind the potential huge security gaps.

But the specific _tools_ aren't the issue here. It's the culture of "ship something now we'll deal with fallout later". A lot of people start using Docker and won't ever look at the Dockerfile, or, will add a Maven dependency and won't even check licenses or security updates for _any_ of the transient dependencies.

Cloud technologies and containerization make everyone just think "we can do things so fast now" and never, ever pay attention to details that can come back to bite you.

On the flip side, it's a good time to be in cybersecurity; because this cultural problem will never, ever, get solved. :)

Re: The sad state of sysadmin in the age of containers (2015)

#92

> The first internet worm spreading via flawed docker images? Good question, why don't we see exploits of all that implicit trust to the degree that, eg, the DOS shareware scene gave your PC visible virus infection, or the early internet gave us worms that would bog down the whole net? My attempt at an answer: Because the black hats aren't hobbyists anymore. Visibility is for amateurs.

Are you saying that every new black hat is immediately a professional? Or that there aren't any new black hats? In every other activity of human life, new amateurs appear as the older ones become professionals. Where are the visible exploits from the new amateurs?

Re: The sad state of sysadmin in the age of containers (2015)

#93
post #84
post #19

Earlier quoted context omitted.

Makefiles aren't scary. But they're also not particularly good. I use Rake (or Gulp, or whatever) because then I can use Ruby (or JavaScript, or whatever). Shell plumbing is fine for informal and small-scale stuff, and I make my code conform if somebody down the line (who may be me) wants to get out their duct tape, but the world is more complex than what /bin/sh can see. Shell is the lowest common denominator. Expec…

But they're also not particularly good. Million+ line codebases got errm made with make. In my experience people most sneering at it have far lesser demands on their tool...

Million+ line codebases were written entirely in COBOL.

The world advances.

Re: The sad state of sysadmin in the age of containers (2015)

#94
If a company has a threat model and a list of business goals then they would have at least a risk matrix and they might decide what to do: either go the slow way and build software they can trust or accept the risks, backdoors and all the rest. Most companies skip all of that and hope for the best. Not always a conscious decision.

Sometimes they get their unpatched servers encrypted by some ransomware, remember they don't have any backup, close shop and move on to the next business idea. I've seen that happen.

Re: The sad state of sysadmin in the age of containers (2015)

#95

Ugh make files. Kill me. Give me a bash script or give me death.

Whats wrong with Makefiles?; they're quick and easy ways to describe how to build simple and _not super_ complicated systems. I've found this to be a great resource, hope you find it useful: * http://gromnitsky.users.sourceforge.net/articles/notes-for-n...

It’s ironic. I love python but the syntax for make files was a pain. They’re rather cryptic. I guess they’re a staple time to conquer them once and for all.

Re: The sad state of sysadmin in the age of containers (2015)

#96
post #80
post #13

Earlier quoted context omitted.

There are over 2.9 million lines of code in Apache Hadoop alone, not counting dependencies. If you can't trust Apache, you can't trust Hadoop, regardless of whether or not you can compile it yourself.

There are nearly 10 million lines of code in libreoffice, and yet I can and have built it from source just by typing: 1. $ git clone git://anongit.freedesktop.org/libreoffice/core 2. $ apt-get build-dep libreoffice 3. $ ./autogen.sh && make Just because something has a large code base doesn't mean we shouldn't be able to build it from source ourselves.

Did you read all those lines yourself? Did you even confirm checksums matched before running them?

I think that's the parent's point. You can build from source, but how do you trust the source? Is it any more egregious to trust a prebuilt binary from a specific website than it is the raw source? If you can't trust the binary being hosted by the author/caretaker, can you really trust the source being hosted or maintained by the author/caretaker?

Re: The sad state of sysadmin in the age of containers (2015)

#97
post #5

> None of these "fancy" tools still builds by a traditional make command. Is there anything more "get-off-my-lawn" than "These tools don't use the thing I like!"

> "These tools don't use the thing I like!" Pretty much. I know several admins who appear to be joining a growing pool of luddites who rail against anything new. They're particularly butt-mad about anyone drawing more salary than them. "DevOps" is their favored totem to direct their ire at. I used to try and convince them otherwise, but it turned out to be a completely futile waste of time. At the end of the day, per…

Also means fat fat contracting gigs for us luddites to clean up after the move-fast-break-shit-gtfo kids.

Re: The sad state of sysadmin in the age of containers (2015)

#98
post #53
post #51

Earlier quoted context omitted.

Your second paragraph got to the heart of it. If we want to use some standard build toolchain, it needs to use a nice language and not feel obscure. I was explaining to someone a bash script I wrote, and he said "why not use Python". There were reasons but... he was right, Python would be much easier to use and maintain, and we have a lot more developers who know it. That said, Maven is incredibly suck-tastic.

Eh. It's not my favorite thing out there, but Maven's fine for what it is. It's designed for and explicitly for well-behaved Java artifacts. If your Java artifacts are not well-behaved, you're going to have a bad time--in my experience, most of those cases are doing things you probably shouldn't be doing. (You may be a wizard and have a reason to do them, for sure--but that's what writing Maven plugins is for. Or not…

Given the limitations of the platform, there really isn't a such a thing as a well-behaved JVM library that depends on other libraries, unfortunately. Oracle really dropped the ball by only serving their own needs with the module system.

Re: The sad state of sysadmin in the age of containers (2015)

#99

As a "major theme", the author takes: > Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It’s an incredible mess of dependencies, version requirements and build tools. And as the major introduction to the blog post: > I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths. Huh? Old-school sysadmins know how to keep syst…

I think the logic is, if we didn't rely on Containers and prebuilt VM's, Hadoop had to be easier to build to be useful.

The point everyone seems to be missing, and the one I think most important, is that we're no longer building from trusted sources.

Build systems just download and run random code from the internet without verifying that its the correct code, from the correct source.

Its a ticking time bomb.

Re: The sad state of sysadmin in the age of containers (2015)

#100

> Ever tried to security update a container? Wouldnt the approach for security updates be to replace the container?

Depends on the approach, you can use mutable or immutable containers. In fact, the OpenVZ VPSs that were at one time reasonably popular were just containers.
Post reply on HN