Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The sad state of sysadmin in the age of containers (2015)

vitavonni.de

Post reply on HN
Print view View original

171–180 of 324 posts

Re: The sad state of sysadmin in the age of containers (2015)

#171

Post by mitchell_h »

0xFFFE wrote: post #40
buzzkillington wrote: post #25

Looked at the Debian wiki the OP linked to: >Debian currently does not include Hadoop packages. There are a number of reasons for this; in particular the Hadoop build process will load various dependencies via Maven instead of using distribution-supplied packages. Java projects like this are unfortunately not easy to package because of interdependencies; and unfortunately the Hadoop stack is full of odd dependencies…

I recently went as far as setting up Bigtop on a VM to build the Deb packages, I gave up after 2 days. I would rather suffer through 2 root canal procedures back to back without anesthesia.

On a bet a couple years ago someone gave me a week to compile and stand up most of the core hadoop services distributed by horton. 4 days in I lost my shit and desk flipped.

Re: The sad state of sysadmin in the age of containers (2015)

#172

Post by spion »

eeZah7Ux wrote: post #103
lmm wrote: post #77

You've got to look at this in a context where platform package managers like apt are simultaneously 1) platform-specific 2) jealous, insisting that every language has to conform to their way of doing things and 3) fundamentally not very good, having very limited ability to do things like install packages for a single user or install multiple versions of the same package. Platform package managers like maven have been…

> jealous, insisting that every language has to conform to their way of doing things Not at all. The large majority of packaging systems have reasonable and similar requirements for upstreams, like not bundling dependencies or not hardcoding paths. > fundamentally not very good, having very limited ability to do things like install packages for a single user or install multiple versions of the same package That's com…

The design results with flawed user experience and the users will move past the restrictions one way or another. It's up to system package managers to realize why the old way of doing things is simply not good enough; otherwise, people will just move on.

I'm sad that this is not happening since in the process we're also losing the good bits of system package managers as per this article.

Re: The sad state of sysadmin in the age of containers (2015)

#173

Post by regularfry »

Annatar wrote: post #63
the_duke wrote: post #58

Earlier quoted context omitted.

You missed the bigger point. Just look at the popular images on Docker hub. A lot of them involve messy build steps, including downloading binaries or source tarballs without verification. It's often hard to know which dependencies a Docker image has, and therefore hard to track vulnerabilities and redeploy fixed images. A lot of docker containers end up either running for a long time, or get rebuilt and redeployed o…

"But we need to figure out the tooling and ecosystem story to build, verify and update container deployments securely." Already done: Solaris zones. Available in the SmartOS distribution near you. Combine with OS packaging, imgadm and vmadm commands for maximum impact.

I suspect there's a similar story around LXC available.

Re: The sad state of sysadmin in the age of containers (2015)

#174

Post by adamtulinius »

dsr_ wrote: post #97
hvidgaard wrote: post #74

Earlier quoted context omitted.

If you can build on your distro of choice, you can rely on the same security audit and process. Keep track of the packages you link against and include in your image, and just rebuild your images when a new security update comes along. Then you replace the running one.

See that "keep track" bit? It's expensive unless you build a tool to do it for you -- which is what I said previously.

You don't have to keep any more track than you do with your OS; it's just a different button you hit when updating containers.

Or don't keep track at all: Just rebuild and redeploy often enough.

Re: The sad state of sysadmin in the age of containers (2015)

#175

Post by dfox »

outadoc wrote: post #85
_wldu wrote: post #49

Standard Makefiles are really under appreciated today. They are simple and can be used with most any language, version controlled and have been around for decades.

They don't solve any dependency management problems.

Most ports like systems actually use makefiles to solve cross-package dependency management. (Whether that is good approach and whether writing what is essentially package manager as hairy ball of makefiles is good idea are another questions)

Re: The sad state of sysadmin in the age of containers (2015)

#176

Post by GordonS »

Avamander wrote: post #158
GordonS wrote: post #125

Earlier quoted context omitted.

Whether it's `sudo make install` or `curl | bash`, in both cases you have the opportunity to first inspect the code that will be executed.

You don't have the opportunity in the latter case. https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...

Sorry, I don't follow.

If I have a makefile, I can inspect it and see what it does.

If I have shell script (or indeed a makefile!) that calls `curl | bash`, I can inspect that shell script and see the URL that is used with curl, and then inspect the contents that the URL returns.

Re: The sad state of sysadmin in the age of containers (2015)

#177

Post by apta »

buzzkillington wrote: post #117
zajio1am wrote: post #104

Earlier quoted context omitted.

> 2) jealous, insisting that every language has to conform to their way of doing things Well, programming languages clearly should not be in business of software distribution. That creates unnecessary tight coupling between language, build system and distribution, causes proliferation of language-specific package managers incompatible with a platform way of doing things.

Something Java and the Java ecosystem is terrible at. I've had project where we needed all of 7,8,9 and 10 [0] for the data stack: Hadoop, Scala, Spark, HDFS, zookeeper, pyspark. Good luck setting that up on a single machine, containers all the way. The java path nightmares I still have. I'm probably one of the few people in the word who know this much about the JVM without ever having written any Java. [0] We should…

> I've had project where we needed all of 7,8,9 and 10 [0] for the data stack: Hadoop, Scala, Spark, HDFS, zookeeper, pyspark.

I suspect that a big reason was due to using Scala, which is notorious for not maintaining binary backward compatibility across minor releases.

Re: The sad state of sysadmin in the age of containers (2015)

#178

Post by javabean22 »

alfozan wrote: post #36
buzzkillington wrote: post #25

Looked at the Debian wiki the OP linked to: >Debian currently does not include Hadoop packages. There are a number of reasons for this; in particular the Hadoop build process will load various dependencies via Maven instead of using distribution-supplied packages. Java projects like this are unfortunately not easy to package because of interdependencies; and unfortunately the Hadoop stack is full of odd dependencies…

Who still uses Hadoop anyway? https://spark.apache.org/ https://www.iguazio.com/data-science-post-hadoop/

Author doesn't know what they are talking about. Amazon EMR is just Hadoop in the could.

Re: The sad state of sysadmin in the age of containers (2015)

#179

Post by dsr_ »

adamtulinius wrote: post #174
dsr_ wrote: post #97

Earlier quoted context omitted.

See that "keep track" bit? It's expensive unless you build a tool to do it for you -- which is what I said previously.

You don't have to keep any more track than you do with your OS; it's just a different button you hit when updating containers. Or don't keep track at all: Just rebuild and redeploy often enough.

Supply chain attackers love this.

Re: The sad state of sysadmin in the age of containers (2015)

#180

Post by dirkf »

ekidd wrote: post #144
jwr wrote: post #19

I recently had a similar discussion with people using npm for building a CSS framework library. I tried to explain the concept of getting a pre-downloaded tarball and using "make" (or similar) to produce target artifacts from source files in a deterministic, repeatable and reliable manner, without relying on any third-party servers being available and without pulling in dependencies that might have changed. It seems…

> without relying on any third-party servers being available and without pulling in dependencies that might have changed. There are two different issues here: 1. Not pulling in changed dependencies. This is what "lock files" are for: To limit builds to known version of every dependency. npm was terrible about this for a long time. Most other language package managers are better. 2. Not relying on third party servers…

At work we also cache all 3rd party dependencies locally. Not because the 3rd party servers might be down but to be sure that one/two/... years from now we can still recreate the same software we delivered at that time to a customer. Our build machines typically didn't even have internet access. If a dependency was not available offline (due to developer error or whatever) we would know quickly.

In certain industry branches this is even mandatory if you want to be seriously considered as a supplier. If a build tool does not support reproducible builds in such a way (both fixing dependency version and getting it from a cache somehow) or makes it difficult then it is considered as a hobby toy that has no place in the workplace.

Even for small businesses I would advocate to take this seriously from the beginning. It's not that hard and will save you headaches later on when suddenly reproducible builds become important.

Post reply on HN