Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The sad state of sysadmin in the age of containers (2015)

vitavonni.de

Post reply on HN
Print view View original

351–360 of 435 posts

Re: The sad state of sysadmin in the age of containers (2015)

#351

Post by nunez »

> None of these “fancy” tools still builds by a traditional make command. Every tool has to come up with their own, incomptaible, and non-portable “method of the day” of building.

100% this. Learning Make a few years ago was one of the best decisions I’ve ever made. It’s simple (until it’s not), straightforward and available on just about any Linux and UNIX installation (version compatibility aside).

Trying to make the case for POSIX-compatible Bash scripts has been tough, though.

I also agree re: Docker. While you can secure a container image with USER statements in the Dockerfile and knowing how to give containers just the capabilities that they need to do their job, it is WAY too easy to run them as root and give them privileged access to things. It should’ve been the other way around. Also, every container orchestration platform seems like a really elaborate hack.

Re: The sad state of sysadmin in the age of containers (2015)

#352

Post by emmelaich »

atsaloli wrote: post #315
kokey wrote: post #230

Earlier quoted context omitted.

CFengine, at least version 3, was probably the furthest away from string manipulation (and I was given the impression that text file content manipulation was considered a bad idea with it). What killed it was promise theory, which is actually a great theory and works quite well but made writing the bundles painfully hard and also hard to maintain. Also during the early days of v3 it was probably lacking a ton of esse…

I make my living as a CFEngine consultant. CFEngine runs every 5 minutes (it's lightweight enough to do that). The evolution was: CFEngine 1 ran once a day; CFEngine 2 ran once an hour; CFEngine 3 runs every 5 minutes. Self-healing infrastructure.

The concept of self-healing is a bit weird for me. Surely you want to investigate the cause before it heals?

Funny that we have tools like tripwire which have the opposite idea of the world.

My dream would be to have both functionalities in a single tool.

Bidirectionality! If you solve a problem on one machine you could pull that fix then push the same fix out to other machines as a preventative measure.

Some mix of git/osquery/augeas could do this.

Re: The sad state of sysadmin in the age of containers (2015)

#353

Post by nunez »

seorphates wrote: post #205
ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

Grumpykins here. I think the term "Modern devops" sort of nails it but not quite how you used it. Most departmental/enterprise sys admins/engineers of lore that had even the slightest necessity for life outside the box scaled anything resembling automation to its breaking points. Combined with knowing and serving the reasons for their existence - developers, users etc., and devops is nothing new - it is now simply th…

Agreed. DevOps has been a thing for a long time. The funny thing is that the core of the DevOps philosophy —- to unite development and ops through code —- is still a rarity. SO MANY big companies have entire departments for DevOps that are basically either developers doing release management sysadmins writing pseudo code for infrastructure while making it somehow less accessible to development teams.

Re: The sad state of sysadmin in the age of containers (2015)

#354

Post by nunez »

ci5er wrote: post #345
dozzie wrote: post #302

Earlier quoted context omitted.

It would be even nicer if people weren't advocating it as a configuration management system, only as what it is: a deployment system.

Who advocates it as a configuration management system? I've not seen that...

Everyone compares it to chef and puppet

Re: The sad state of sysadmin in the age of containers (2015)

#355

Post by nunez »

I think a lot of stuff that I used to need to know —- OS kernel internals, hardware specifics, relatively deep network knowledge —— is less useful in the age of public cloud, containers and immutable infrastructure. There isn’t really a need to tune a kernel for performance or do deep troubleshooting to root cause OS issues and ensure uptime; if something seems off, kill the machine or container and let auto scaling or the container orchestrator take care of it. If something wrong with cloud networking, call AWS, as the most we can do is prove that it’s on them. Etc.

While I still know these things, I haven’t had to employ them in a while. I will probably use those even less as I start getting into management. That’s a bit saddening.

But the new stuff I’ve learned over the years, namely, designing systems like I’d design software, is amazing. Applying TDD to infrastructure isn’t something I thought I’d be doing, but here we are, and we have “modern DevOps” tooling (and lots of other things) to thank for that.

Re: The sad state of sysadmin in the age of containers (2015)

#356

Post by ci5er »

nunez wrote: post #354
ci5er wrote: post #345

Earlier quoted context omitted.

Who advocates it as a configuration management system? I've not seen that...

Everyone compares it to chef and puppet

Ansible? I could see using it to build a container that then got "orchestrated", I guess, but ... hmmm. I've never really looked at them as doing the same thing. (Nor have I looked at either Chef/Puppet for CM. Maybe I'm just stupid... My ex- certainly thinks that I am...)

Re: The sad state of sysadmin in the age of containers (2015)

#357

Post by nunez »

brightball wrote: post #160
ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

Would be interested to get your opinion on Puppet/Ansible/Chef/CFEngine/SaltStack

Ansible all of the way. Chef and Puppet have too much overhead in comparison. Ansible is agentless. You can either use a centralized server for deployments or you can have every instance configure itself. Also, Ansible is YAML based, which is a strength and a weakness.

Chef is a runner up. Love their community and Chef is pretty straightforward once you learn the lingo.

Puppet doesn’t really work for modern Git development workflows (Hiera and r10k are duct tape) and testing Puppet is kludgey. Also, most of the docs you’ll find for it stopped getting updated in 2015 or so.

Re: The sad state of sysadmin in the age of containers (2015)

#358

Post by Spivak »

kazen44 wrote: post #202
Animats wrote: post #179

If you thought having to deal with old COBOL programs was a problem, it gets much worse. When there are 10 year old containers in production, and the parts to rebuild them are long gone, then you have a real problem. There's a reason that Google has internal systems which can and do rebuild everything from source.

This is actually a very important point that gets ignored a lot. What will happen if public container images are no longer available for building your application? Your in a for a world of hurt if a major dependency of your application doesn't work anymore because someone pulled the image.

The same thing you do with any 3rd party dependency -- save a copy of the source. Pull from DockerHub all you want but keep the images and their sources in a private registry and deploy your actual services from that.

Re: The sad state of sysadmin in the age of containers (2015)

#359

Post by Intermernet »

segmondy wrote: post #265
gm-conspiracy wrote: post #243

Earlier quoted context omitted.

A good sysadmin would not look like they are doing much work (everything is humming along and can self-heal minus physical problems), but a good devops person is constantly busy.

Great sysadmins get fired unless they learn to pretend to look busy. They automate everything till they have nothing to do.

Logging is your friend here. You can spend days scrolling through logs, doing an occasional grep and making disapproving noises occasionally. Bonus points for developing some graphs for the next meeting.

Re: The sad state of sysadmin in the age of containers (2015)

#360

Post by Spivak »

secabeen wrote: post #229
claviola wrote: post #141

Earlier quoted context omitted.

How many people do you see setting up a deploy pipeline that includes pulling security updates into the base image and redeploying as needed? In my experience, it's much more common to see docker images that have been untouched for months with zero accountability of what exactly is running there.

That's my real concern: old, out of date images. How will we handle another OpenSSL-level vulnerability in 7 years, with bad code buried in containers that haven't been updated in 4, and for which the build infrastructure is no longer functional?

This really isn't that different from having some pre-built statically liked app still kicking on your system with the source and/or build tooling long gone.

There aren't really easy answers here. You can't fix bad software with more tooling.

Post reply on HN