The sad state of sysadmin in the age of containers • Hacker News Board

Re: The sad state of sysadmin in the age of containers

#341

Post by thaumaturgy » Wed, Apr 22, 2015, 5:58 PM UTC

jetpks wrote: post #324

skywhopper wrote: post #232

I agree that many of these convenient setups are embarrassingly sloppy, but it's the sysadmin's responsibility to insist on production deployments being far more rigorous. No one can tell you how to build hadoop? Well, figure it out. Random Docker containers being downloaded? Use a local Docker repo with vetted containers and Dockerfiles only. I don't even allow vendor installers to run on my production systems. My e…

> it's the sysadmin's responsibility to insist on production deployments What decade are you from? No startups are hiring sysadmins to do any kind of work anymore. They're hiring "dev-ops" people, which seems to mean "Amateur $popularLanguage developer that deployed on AWS this one time." That's the whole problem with the dev-ops ecosystem. None of these dev-ops people seem to have any ops experience.

> No startups are hiring sysadmins to do any kind of work anymore.

Then maybe people should be willing to work for more grown-up businesses.

HN tends to get a distorted view of what's important in the tech industry. The tech industry is way, way, way bigger than startups, and there are still plenty of companies that recognize the value of good sysadmins.

Let the startups learn their lesson in their own time.

Re: The sad state of sysadmin in the age of containers

#342

Post by dikaiosune » Wed, Apr 22, 2015, 6:02 PM UTC

thaumaturgy wrote: post #341

jetpks wrote: post #324

Earlier quoted context omitted.

> it's the sysadmin's responsibility to insist on production deployments What decade are you from? No startups are hiring sysadmins to do any kind of work anymore. They're hiring "dev-ops" people, which seems to mean "Amateur $popularLanguage developer that deployed on AWS this one time." That's the whole problem with the dev-ops ecosystem. None of these dev-ops people seem to have any ops experience.

> No startups are hiring sysadmins to do any kind of work anymore. Then maybe people should be willing to work for more grown-up businesses. HN tends to get a distorted view of what's important in the tech industry. The tech industry is way, way, way bigger than startups, and there are still plenty of companies that recognize the value of good sysadmins. Let the startups learn their lesson in their own time.

The alternative is that many of the startups don't learn this in their own time, and they go on to become bigger, more successful companies who can set the tone and shift the market. Of course, if they're actually able to succeed by doing so, then that says something too. Although the trend of many data breaches certainly wouldn't decline in that case.

Re: The sad state of sysadmin in the age of containers

#343

Post by dba7dba » Wed, Apr 22, 2015, 6:03 PM UTC

tedivm wrote: post #311

ksml wrote: post #295

Earlier quoted context omitted.

I think more disconcerting is the rise of "sysadmins" who think they're qualified sysadmins because they know how to bash and docker.

This is hardly a new problem- and in many ways, I'm not sure it's a problem at all compared to the company cultural issues brought up by skywhopper. Whether it's programming or system administration, you're always going to have new people getting excited about the sudden power they've learned. Being able to make computers do things opens up this whole new world, and when people find themselves in that world they may…

Amen to that. When I see some of the job desc in job postings for DevOps/Sysadmin, I wonder. Is there really someone out there will all the skills that are asked for?

Re: The sad state of sysadmin in the age of containers

#344

Post by tracker1 » Wed, Apr 22, 2015, 6:04 PM UTC

stephenr wrote: post #32

blfr wrote: post #4

This bothers me as well. Even tasks as simple as adding a repository are now being "improved" with a curl | sudo bash style setup[1]. However, installing from source with make was (and remains) a mess. It may work if you're dedicated to maintaining one application and (part of) its stack. But even then it usually leads to out of date software and tracking versions by hand. Many people have this weird aversion to doin…

I agree that the "just curl this into bash" instructions are nightmare - on any platform. I think a lot of this is a result of what I like to call the "Kumbaya approach to project/team management": This is where you have a team (either for a single project or a team at a consulting agency, etc) that is effectively all development-focused staff, possibly with some who dabble in Infrastructure/Ops. In this environment,…

It's easy enough to download a given/checked version of the script at http://foo.com/ubuntu/install and have that copied and run inside your docker image... for that matter, it's usually adding a given repository to your repo manager, then installing a given package from that software's corporate sponsors.

I don't think the problem is as rampant as it's made out to be in TFA... that said, most people don't look at said script(s), so it's entirely possible something could have been slipped in. For that matter, I think the issues outlined in the article relate more to overly complicated Java solutions (the same happens in the .Net space) that are the result of throwing dozens of developers some with more or less experience than others at a project, and letting a lot of code that isn't very well integrated slide through whatever review process does or doesn't exist.

Re: The sad state of sysadmin in the age of containers

#345

Post by tracker1 » Wed, Apr 22, 2015, 6:05 PM UTC

DannoHung wrote: post #267

stephenr wrote: post #32

Earlier quoted context omitted.

I agree that the "just curl this into bash" instructions are nightmare - on any platform. I think a lot of this is a result of what I like to call the "Kumbaya approach to project/team management": This is where you have a team (either for a single project or a team at a consulting agency, etc) that is effectively all development-focused staff, possibly with some who dabble in Infrastructure/Ops. In this environment,…

I think part of this is because there aren't any trusted, fully open source, artifact repositories that work with the various package indices out there. Like, most of the way deployment should work is that you come up with some collection of packages that need to be installed and you iterate through and install them. Bob's your uncle. Thing is, all the packages you need live out in the wild internet. Ideally, you'd j…

With docker, as referenced in TFA... you can simply vet a base image, and use that for your application... upgrades? create a new/updated base image and test/deploy against that.

Re: The sad state of sysadmin in the age of containers

#346

Post by j_s » Wed, Apr 22, 2015, 6:07 PM UTC

dysinger wrote: post #316

This 1 page poorly titled wrong rant is the #2 story on this site? "Ever tried to security update a container?" lol. you are doing it wrong. "Essentially, the Docker approach boils down to downloading an unsigned binary, running it, and hoping it doesn't contain any backdoor into your companies network." nope https://blog.docker.com/2014/10/docker-1-3-signed-images-pro... "»Docker is the new 'curl | sudo bash'«" no i…

It appears this was finally changed mid-March, but after initial release in December image signing initially worked as follows:

Docker’s report that a downloaded image is “verified” is based solely on the presence of a signed manifest, and Docker never verifies the image checksum from the manifest. An attacker could provide any image alongside a signed manifest.

https://news.ycombinator.com/item?id=8788770

https://titanous.com/posts/docker-insecurity

https://github.com/docker/docker/issues/9719

edit: add hn discussion, github issue.

Re: The sad state of sysadmin in the age of containers

#347

Post by pjmlp » Wed, Apr 22, 2015, 6:12 PM UTC

getsat wrote: post #331

Alupis wrote: post #317

Earlier quoted context omitted.

> Is it really the best choice, or has someone done something cleaner in golang or c++11? What does the language have to do with the program? Hadoop is what it is because it's a complex problem with a fittingly complex solution. Simply re-writing it in your pet language won't somehow make it "better".

Go and modern C++ are both quite a bit more terse than Java. They also produce binaries which don't necessarily require a runtime to be available on every server (just ABI compatibility). (I have no horse in this race, I am just writing what I think the grandparent comment was referring to)

> They also produce binaries which don't necessarily require a runtime to be available on every server

Just like Java[0]. It is just a matter of choosing the right compiler for the use case at hand.

[0] - http://www.excelsiorjet.com/ (one from many vendors)

Re: The sad state of sysadmin in the age of containers

#348

Post by nasalgoat » Wed, Apr 22, 2015, 6:13 PM UTC

ksml wrote: post #295

skywhopper wrote: post #232

I agree that many of these convenient setups are embarrassingly sloppy, but it's the sysadmin's responsibility to insist on production deployments being far more rigorous. No one can tell you how to build hadoop? Well, figure it out. Random Docker containers being downloaded? Use a local Docker repo with vetted containers and Dockerfiles only. I don't even allow vendor installers to run on my production systems. My e…

I think more disconcerting is the rise of "sysadmins" who think they're qualified sysadmins because they know how to bash and docker.

As a hiring manager, it's very easy to filter these people out at the interview stage.

Being a system administrator requires a very specific personality type that has little to do with experience and more to do with attitude and critical thinking.

Sadly, people are right that startups are skipping past admins, thinking they're not needed anymore. Then later they need to hire one to clean up the giant mess.

Re: The sad state of sysadmin in the age of containers

#349

Post by swills » Wed, Apr 22, 2015, 6:15 PM UTC

eranation wrote: post #252

As a Java / Hadoop / Spark / Scala fan, all I can say is, it's a little embarrassing, not sure how the Java ecosystem around hadoop became so sloppy (I witness it first hand on a daily basis). I wish more people who are concerned with security / ease of build would turn into contributing to maven, sbt, ivy and the hadoop project. Instead of hating the Java ecosystem, why not join it and make it better? Hadoop is ubiq…

Why add all the missing pieces to the other tools (maven, etc) when the OS has the tool (rpm, deb, etc) with all these pieces already?

I think that is the reaction that most people have when they see maven or similar tools.

Re: The sad state of sysadmin in the age of containers

#350

Post by Aloha » Wed, Apr 22, 2015, 6:16 PM UTC

dba7dba wrote: post #343

tedivm wrote: post #311

Earlier quoted context omitted.

This is hardly a new problem- and in many ways, I'm not sure it's a problem at all compared to the company cultural issues brought up by skywhopper. Whether it's programming or system administration, you're always going to have new people getting excited about the sudden power they've learned. Being able to make computers do things opens up this whole new world, and when people find themselves in that world they may…

Amen to that. When I see some of the job desc in job postings for DevOps/Sysadmin, I wonder. Is there really someone out there will all the skills that are asked for?

I'm reasonably certain there isnt - not for the payband offered.