Live data from Hacker News

The sad state of sysadmin in the age of containers (2015)

vitavonni.de

221–230 of 435 posts

Re: The sad state of sysadmin in the age of containers (2015)

#221

Earlier quoted context omitted.

I believe you are talking about Ansible tower, the paid tool from RedHat that gives a centralized server. Ansible is not centralized. It configures servers with SSH and can operate from any user or host who has ssh access.

Yeah but when you run a playbook it's running from a single machine which is calling out via SSH

Yes. The host will run 100% CPU to handle the hundreds of SSH connections.

I've been re configuring 300 to 800 hosts many times a day, never had a problem. I think it would take a few thousands hosts for the performance to be noticeably slow and I am really not sure that other tools or systems could take it much better.

Re: The sad state of sysadmin in the age of containers (2015)

#222

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

What fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?

What makes you think there aren’t young people doing systems administration? Our last two hires in my most recent job were 23 and 27 respectively. Sure, they’re getting trained in the new hot cloud stuff... as the grumpy seniors figure it out first and set patterns... but they are still doing daily work with some rather ancient stuff.

Re: The sad state of sysadmin in the age of containers (2015)

#223

As a "major theme", the author takes: > Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It’s an incredible mess of dependencies, version requirements and build tools. And as the major introduction to the blog post: > I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths. Huh? Old-school sysadmins know how to keep syst…

I've seen this brewing for a while, and getting worse and worse. Back in the 80's and 90's, there were developers who would code their own sorting or hashing routines rather than linking in some external library to handle this "solved" problem. The perjorative term "Not Invented Here" (NIH) grew to describe those developers and they were shamed into reusing code whenever there was code to reuse. And in some cases (li…

Interesting take on the NIH term. I thought this was more an ego thing for the big tech companies. They love to reinvent existing things to look like geniuses

Re: The sad state of sysadmin in the age of containers (2015)

#224

Earlier quoted context omitted.

CFEngine is basic text manipulation, it's not comparable to the rest. Puppet and Chef was the first generation. I wouldn't recommend. All the companies and people I know using Chef migrated away from it after many disasters. Nowadays, it's only mentioned in interviews to find out if candidates have real world fire fighting experiences. Ansible is good. Used that for managing hundreds of machines at multiple jobs (som…

> Not sure about SaltStack. Never had the opportunity to try. I'd be a bit worried though on the long term prospect because I don't think they have much backing or user base. saltstack is a well thoughout solution in my opinion. It makes more logical sense and is less of a mumbled mess then either chef or puppet and has miles better performance then ansible. I know quite a few shops who use it. Its definitly smaller…

+1 for salt -- I wish it had better docs or examples of how to build out a larger system; it's hard to start with imho even if you know ansible well. The existing docs read like man pages without the helpful examples even.

At the last gig, I wrapped salt deploys with a small Slack bot, so users would fire deploys from Slack; you could see what was going out and who was pushing. It was a very very nice, simple, fast solution that should scale to hundreds of machines easily.

Re: The sad state of sysadmin in the age of containers (2015)

#225

Earlier quoted context omitted.

The author should have grabbed the .sdeb or the debian build scripts and tore them apart if they really wanted to make a point (if, upon examining the build, there was one to make). I mean there is a lot of cognitive load/disconnect we're talk about. As an ops guy, I can't look into every package. That's why I trust the package manager (apt-get, yum, whatever) and all the build maintainers who either volunteer or wor…

" Docker containers do need better security introspection and that's going to be a big deal going forward. " Exactly! And npm. And maven. And every damn package system for every damn programming language since package systems are now a requirement.

Yes, but shouldn't you have separate "build" and "deploy" container images? You should "build" a particular version once, "deploy" the result into a test environment, test it thoroughly, and then "deploy" to production, right?

This is not my job (yet). Please tell me if I'm wrong, because I'll need to do it in the next few months.

Re: The sad state of sysadmin in the age of containers (2015)

#226
post #198

Earlier quoted context omitted.

At the end of the day it comes down to the fact that businesses just simply don't care (Equifax etc). They like the idea of security and that's where it ends. In many places if you try to "do things right" you will get fired in two months for being too slow/strict and they will happily replace you with a clueless easily trusting person who "goes and fetches things from random third parties". Many times they get lucky…

> just simply don't care (Equifax etc). And, honestly, why should they? Security breaches have yet to hurt an actual company (they hurt users plenty, but not the organization that's actually responsible).

Data breaches are climbing in cost to organizations. Here's a claim, that in 2017, the average breach costs $3.62 million. https://www.scrypt.com/blog/average-cost-data-breach-2017-3-...

(I've seen similar claims in different ranges. Costs of breaches in the US are pretty high - over $7 million.)

Even Equifax probably wants the 3-4 billion in valuation it's lost since the breach.

The solution appears to be buying tools to avoid and respond to breaches quickly, instead of engaging and building in security awareness. (Microsoft's security development lifecycle comes to mind.)

IMO, both approaches are likely cost effective, though I have no numbers or research to back that up.

Re: The sad state of sysadmin in the age of containers (2015)

#227

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

What fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?

Been working since 86 in this industry.

You lose and gain and you should always be mindful of what is coming. Humility is good. I love you young guys, your ideas keep coming and they are mostly good.

Re: The sad state of sysadmin in the age of containers (2015)

#228

Earlier quoted context omitted.

BSD jails?

jails are actually very mature if you compare them to docker. They have a well working security system and sane networking. (docker just does endless NAT abstraction, which is terrible for certain use cases. Not to mention it breaks a ton of useful networking features)

I was asking a coworker the other day about using containers with a "shared" mapped filesystem subdirectory for UNIX file socket communication versus encapsulating everything in a TCP/IP network stack, but my coworker was concerned about the security risks of a mapped filesystem on the containers.

I sometimes feel like a crazy person.

Re: The sad state of sysadmin in the age of containers (2015)

#229
post #34

not to be pedantic but why should containers be patched for security issues? the entire point of containers (and orchestration systems) is the ability to push updates without downtime. Just update with a newer container....am I missing something? Also about container security - a strict process internally can easily help counter that (I believe Shopify had a nice talk about it at the Google Cloud Platform event in To…

How many people do you see setting up a deploy pipeline that includes pulling security updates into the base image and redeploying as needed? In my experience, it's much more common to see docker images that have been untouched for months with zero accountability of what exactly is running there.

That's my real concern: old, out of date images. How will we handle another OpenSSL-level vulnerability in 7 years, with bad code buried in containers that haven't been updated in 4, and for which the build infrastructure is no longer functional?

Re: The sad state of sysadmin in the age of containers (2015)

#230

Earlier quoted context omitted.

Would be interested to get your opinion on Puppet/Ansible/Chef/CFEngine/SaltStack

CFEngine is basic text manipulation, it's not comparable to the rest. Puppet and Chef was the first generation. I wouldn't recommend. All the companies and people I know using Chef migrated away from it after many disasters. Nowadays, it's only mentioned in interviews to find out if candidates have real world fire fighting experiences. Ansible is good. Used that for managing hundreds of machines at multiple jobs (som…

CFengine, at least version 3, was probably the furthest away from string manipulation (and I was given the impression that text file content manipulation was considered a bad idea with it). What killed it was promise theory, which is actually a great theory and works quite well but made writing the bundles painfully hard and also hard to maintain. Also during the early days of v3 it was probably lacking a ton of essential functions so even if you were trying to do things the right way you would bump into feature limitations. I think this put a lot of people off adopting it widely and why Chef and Puppet did so well.

Puppet and Chef is actually quite good and I still prefer it to Ansible for a number of reasons. I've certainly run it fine in environments of many thousands of servers, though I can understand that it can implode for some people at scale if they design their deployments in a certain way or structure their manifests/cookbooks a certain way. That said I've certainly seen Ansible fold on much smaller infrastructure, but that is also down to a number of factors that can be avoided or mitigated. Idempotency with Puppet is really strong which is something you want if not every single system in your environment is ephemeral, with Chef it's almost as good with that but not always with the first run, with Ansible you have to specifically consider and aim for it when writing code for it in your playbooks.

The fact that you get used to having Chef or Puppet run e.g. every half an hour is a good thing, where Ansible runs are more ad-hoc. This leads me to another thing that bothers me and that is where people think it's a situation of having to use e.g. Puppet or Ansible as if they're conflicting choices for the same tasks. They have a lot of things in common, but Puppet is more for managing and ensuring changes in an idempotent, non-conflicting way while Ansible is more for doing something a bit like that but more for ad-hoc or orchestration tasks. I think it's good to use both but also be sure what you use it for, since one can do a bit of what the other thing is good at but doesn't do it so well.

For example, I would consider using Ansible to do deployments and releases, rotate SSH keys, execute failovers, or even to install the Puppet agent for the first time. I would use Puppet to deploy and update monitoring agents and configuration, user access, ensure directory permissions, configure system things like rsyslog, logrotate, Postfix, ntp, etc.

Post reply on HN