Live data from Hacker News

The sad state of sysadmin in the age of containers

vitavonni.de

311–320 of 466 posts

Re: The sad state of sysadmin in the age of containers

#311
post #295

I agree that many of these convenient setups are embarrassingly sloppy, but it's the sysadmin's responsibility to insist on production deployments being far more rigorous. No one can tell you how to build hadoop? Well, figure it out. Random Docker containers being downloaded? Use a local Docker repo with vetted containers and Dockerfiles only. I don't even allow vendor installers to run on my production systems. My e…

I think more disconcerting is the rise of "sysadmins" who think they're qualified sysadmins because they know how to bash and docker.

This is hardly a new problem- and in many ways, I'm not sure it's a problem at all compared to the company cultural issues brought up by skywhopper.

Whether it's programming or system administration, you're always going to have new people getting excited about the sudden power they've learned. Being able to make computers do things opens up this whole new world, and when people find themselves in that world they may end up overestimating their skills and underestimating how much they need to grow. What they fail at understanding they make up with in enthusiasm, and with experience they become more knowledgable about what they don't know.

If we waited until they were "qualified" for jobs they would never get the experience to become qualified. At the same time there is more than enough room in the current job market to support people of lower skillsets, and for some companies that's considered an investment (junior people tend to turn to senior people over time).

This is where it becomes a company culture issue. If a company is smart they'll have a few senior people making sure things are held to the right standard, and a few junior people who can get things done but need some guidance and direction. However, lots of companies (especially the smaller ones who may be more constrained by budget) go for the cheaper route and would rather hire someone junior as their main support. The problem isn't that the sysadmins aren't qualified sysadmins, it's that they're junior system admins who have been hired for the wrong job. Companies that fail to value experience tend to suffer as a result.

Re: The sad state of sysadmin in the age of containers

#312

Earlier quoted context omitted.

Have you even been in a project where the developers didn't know how to build it? It's a strange situation, with huge environments being passed from one computer to another, and treasured with more care than the code itself.

This happened to me about a decade ago. A very smart sysadmin in the company created an acronis image for machine deployments. They very carefully documented everything they changed, and how to recreate it. Then someone else created an image from one of the imaged machines without documenting what they changed. This happened a couple dozen or so times until the image pretty much was a mess of hand installed binaries,…

Well fuck me. I just spent two weeks fiddling with Vagrant and Docker and finally got everything up and humming only to come into this thread. Going to refrain from slapping the SysAdmin title on myself for now.

Re: The sad state of sysadmin in the age of containers

#313
post #237

Earlier quoted context omitted.

I agree that the "just curl this into bash" instructions are nightmare - on any platform. I think a lot of this is a result of what I like to call the "Kumbaya approach to project/team management": This is where you have a team (either for a single project or a team at a consulting agency, etc) that is effectively all development-focused staff, possibly with some who dabble in Infrastructure/Ops. In this environment,…

Am dealing with this situation right now. Apparently wget -qO- https://get.docker.com/ | sh as root[1] is the "supported" way of installing Discourse [1]: https://github.com/discourse/discourse/blob/master/docs/INST...

No, that's Discourse install instructions quick, hand-wavy way of telling you to install Docker if you don't already have it. If your cloud environment already has Docker installed, you can skip that step.

Are you really trying to say that the instructions for installing Docker should be considered in-scope for a guide to install Discourse on a cloud server?

They've included a short snippet that will get you a Docker, in the way recommended by Docker, for whatever your base system is. Many production systems do not move at the pace of Docker development, so it's not practical to run Docker from your distribution's package archive. Some distros will not have distributed packaged Docker releases at all.

What's wrong with these instructions? If you are really "dealing with this" right now, it is worth noting that something like 20 or more supported platforms have specific Docker installation instructions from the Docker website.

https://docs.docker.com/installation/#installation

From a quick sample of those instructions, only the Ubuntu instruction page uses the wget|sh method, and it's using an SSL connection to Docker's own website to add an apt source with signatures in the supported way. This way should work on any Debian-based or Yum-based distro, and writing the instructions like this must likely save Discourse from getting a lot of "How do I docker" issues and e-mails from their clueless users.

So, would you prefer that part just says "installing Docker is out of scope" or should the Discourse developers go through every distro and cloud system and document the specific instructions for that? To do that would completely defeat the purpose of even using Docker at all.

Re: The sad state of sysadmin in the age of containers

#314

Earlier quoted context omitted.

I agree that the "just curl this into bash" instructions are nightmare - on any platform. I think a lot of this is a result of what I like to call the "Kumbaya approach to project/team management": This is where you have a team (either for a single project or a team at a consulting agency, etc) that is effectively all development-focused staff, possibly with some who dabble in Infrastructure/Ops. In this environment,…

There are many more depressing examples of this at http://curlpipesh.tumblr.com

Funny tumblr but makes me care-confused.

I understand that curl pipe sh could have security problems but I also don't see it as that much different than the "normal" and "ok" way of doing things. I would consider something like the below pretty normal.

  wget https://whatever.io/latest.tgz
  tar xzf latest.tgz
  cd whatever-stable
  ./configure && make
  sudo make install
Because of familiarity, we aren't going to be too worried about what we are doing. If we are on a secure system (like a bank or something) then we've probably already gone through a bunch of hoops (source check, research) and we mitigate it like anything else.

What is so different about

  curl https://whatever.io/installer.sh | sudo bash
We didn't check the md5s in the first example, so yolo, we don't care about the content of the tarball we just `make install`-ed. We're assuming the webserver isn't compromised and that https is protecting the transfer. Is it because the tarball hit the disk first? Does that give us a warm fuzzy? Is it because "anything could be in installer.sh!!!?! aaaaah!". Well, anything could be in Makefile too right? Anything could be in main.c or whatever.

I agree that curl sh | sudo bash makes my spidey sense tingle. But if I really cared, I would read the source and do all the normal stuff anyway. So I think it's some kind of weird familiarity phase we're all in.

Re: The sad state of sysadmin in the age of containers

#316
This 1 page poorly titled wrong rant is the #2 story on this site?

"Ever tried to security update a container?" lol. you are doing it wrong.

"Essentially, the Docker approach boils down to downloading an unsigned binary, running it, and hoping it doesn't contain any backdoor into your companies network." nope https://blog.docker.com/2014/10/docker-1-3-signed-images-pro...

"»Docker is the new 'curl | sudo bash'«" no it's not. most intelligent companies are building their own images from scratch.

People that care about what's in their stack take the time to understand what's in there & how to build things.

Re: The sad state of sysadmin in the age of containers

#317
post #300
post #269

Earlier quoted context omitted.

>> No one can tell you how to build hadoop? Well, figure it out. I get the impression that several people working on debian couldn't work this one out!

Hadoop is insane. The elephant is fitting. Is it really the best choice, or has someone done something cleaner in golang or c++11?

> Is it really the best choice, or has someone done something cleaner in golang or c++11?

What does the language have to do with the program?

Hadoop is what it is because it's a complex problem with a fittingly complex solution. Simply re-writing it in your pet language won't somehow make it "better".

Re: The sad state of sysadmin in the age of containers

#318

Is it a coincidence that all the technologies the OP complains about are Java (Hadoop, Apache Bigtop, Maven, ivy, sbt, HBaseGiraphFlumeCrunchPigHiveMahoutSolrSparkElasticsearch)?

Node/npm should get a mention. On a simple static website I've seen, a couple of grunt tasks end up pulling in over 14,000 files. And a hugely nested directory structure.

Part of it is the ... interesting ... idea that individual functions should come in their own module. Some npm modules are literally 6 lines of code. But they get packaged up just like everything else. There's no concept of having a stdlib or something. (Apparently node/v8/minifiers aren't smart enough to do a good job if you use a stdlib.)

Re: The sad state of sysadmin in the age of containers

#319
post #300
post #269

Earlier quoted context omitted.

>> No one can tell you how to build hadoop? Well, figure it out. I get the impression that several people working on debian couldn't work this one out!

Hadoop is insane. The elephant is fitting. Is it really the best choice, or has someone done something cleaner in golang or c++11?

http://www.pachyderm.io is modern alternative.

Re: The sad state of sysadmin in the age of containers

#320
post #312

Earlier quoted context omitted.

This happened to me about a decade ago. A very smart sysadmin in the company created an acronis image for machine deployments. They very carefully documented everything they changed, and how to recreate it. Then someone else created an image from one of the imaged machines without documenting what they changed. This happened a couple dozen or so times until the image pretty much was a mess of hand installed binaries,…

Well fuck me. I just spent two weeks fiddling with Vagrant and Docker and finally got everything up and humming only to come into this thread. Going to refrain from slapping the SysAdmin title on myself for now.

Docker is awesome, but you shouldn't be using blind base images. Use Dockerfiles, they're self-documenting.
Post reply on HN