[flagged]
The sad state of sysadmin in the age of containers (2015)
131–140 of 324 posts
Re: The sad state of sysadmin in the age of containers (2015)
#132Re: The sad state of sysadmin in the age of containers (2015)
#133Earlier quoted context omitted.
Truth be told the whole front-end ecosystem relies on those servers being available, so in their absence any development everywhere would grind to a halt - as it almost did during the left-pad scandal. While I agree that this is not exactly the sanest approach, within the ecosystem there's no incentive to work differently. Also, like someone else mentioned - dependency hell only got worse over time - setting up a new…
There’s a pretty straightforward solution to this problem which is to run your own NPM server even if it’s only a caching proxy. For some orgs, the uptime provided by the third party servers is sufficient.
Having been at 3 companies that did this, I can say with certainty that our caching proxies (artifactory in all of them) were much more likely to go down than the public repos were.
Re: The sad state of sysadmin in the age of containers (2015)
#134Looked at the Debian wiki the OP linked to: >Debian currently does not include Hadoop packages. There are a number of reasons for this; in particular the Hadoop build process will load various dependencies via Maven instead of using distribution-supplied packages. Java projects like this are unfortunately not easy to package because of interdependencies; and unfortunately the Hadoop stack is full of odd dependencies…
Who still uses Hadoop anyway? https://spark.apache.org/ https://www.iguazio.com/data-science-post-hadoop/
Spark and dependencies are just as much of a tire fire and you’ll often want Hadoop as well.
Re: The sad state of sysadmin in the age of containers (2015)
#135I recently had a similar discussion with people using npm for building a CSS framework library. I tried to explain the concept of getting a pre-downloaded tarball and using "make" (or similar) to produce target artifacts from source files in a deterministic, repeatable and reliable manner, without relying on any third-party servers being available and without pulling in dependencies that might have changed. It seems…
18yo JS/Ruby/Go dev here. `make` is certainly known to us; it's just that platform-specific build systems (`mage` for Go, `rake` for Ruby, `gulp` for JS etc.) are simply more sane to set up per languages than using `make` for all languages and allows for efficient code reuse.
You have make call the platform specific build systems.
Re: The sad state of sysadmin in the age of containers (2015)
#136Earlier quoted context omitted.
You missed the bigger point. Just look at the popular images on Docker hub. A lot of them involve messy build steps, including downloading binaries or source tarballs without verification. It's often hard to know which dependencies a Docker image has, and therefore hard to track vulnerabilities and redeploy fixed images. A lot of docker containers end up either running for a long time, or get rebuilt and redeployed o…
Just build your own. It's not that hard. Before containers we would script the install process using Ansible. With containers we script the install process using a Dockerfile. If you know how to do one, you know how to do the other. Just don't be lazy and use unofficial images created by people you don't know or trust. Just like you wouldn't pull any random role from ansible-galaxy. Or pull a random guy off the stree…
You can even avoid dockerfiles by using Ansible:
https://docs.ansible.com/ansible/latest/scenario_guides/guid...
Re: The sad state of sysadmin in the age of containers (2015)
#137Before containers, I had sysadmins who ran our AIX cluster. Three nodes, not a single one configured like the other. There was zero consistency where it counted and these jokers would go on a terminal and manually configure Websphere and other components and screw it up constantly. We even tried to give them a terminal tool where you could type in one window and it would mirror on other boxes, but they refused to use it.
Puppet, Chef, Ansible, all led the way to something better. Docker and Kubernetes feel so far advanced from what we had in 2012.
Re: The sad state of sysadmin in the age of containers (2015)
#138Earlier quoted context omitted.
Sell talk and buzzwords. Either author has no idea that Hadoop is ecosystem and Spark depends on it or deliberately mix Hadoop and Kubernetes, which aren't much related. And good luck running Spark without Hadoop ;)
Spark doesn’t have a hard dependency on Hadoop. Spark doesn’t have a storage engine, but you don’t necessarily need one.
Spark still depends on Hadoop for a lot:
- Using Parquet files = parquet-mr which is tied to Hadoop MR https://github.com/apache/spark/tree/master/sql/core/src/mai...
- Using S3 instead of HDFS = Hadoop S3a connector
Even if you don't run HDFS and YARN, you aren't escaping Hadoop. And if some configuration goes wrong, and you'll probably need to look into the Hadoop conf files.
The original comment was about the mass of libraries that Hadoop brings in. Spark isn't a solution that allows you to leave the mess. If you try to dockerize spark, you'll still see that you have 300 MB size images full of JARs that came from wherever.
Re: The sad state of sysadmin in the age of containers (2015)
#139Earlier quoted context omitted.
The "one package at a time" model assumes you care about per-instance uptime and bandwidth costs. The "reset the world" model assumes rebuilds and reboots are cheap enough not to have to. I don't think either is necessarily wrong; there's not a huge distance between `apt-get install unattended-upgrades` and a daily (weekly, whatever) container rebuild except that you need to cron the latter. Unless you're using a dif…
You need to make the decisions, keep track, and do the work. If you've got the tools made for you, excellent. Otherwise you also have to build and maintain the tools -- which is my point.
There are no more decisions to make one way or the other. If you're keeping a VM image updated with new packages, that's either a) automated with something like unattended-upgrades, or b) a manual scan of DSA's with decisions as to whether they're important enough to apply and to which boxes they need to be applied. It you're rebuilding a container image, that's either a) a 2am cron job to do a blind rebuild that captures the latest package versions anyway, or b) a manual scan of DSA's with decisions etc etc...
The 2am cron job, if you go that way, is, assuming you've got an automated build anyway, a one-line bit of config somewhere. It's not something I'd usually think the words "build and maintain" would apply to.
Re: The sad state of sysadmin in the age of containers (2015)
#140Ever tried to security update a container? The whole point of using a container is that you can destroy it and build a new one easily. The new one should be built using up-to-date packages with security patches applied (and tested, obvs). Using the 'pets versus cattle'[1] analogy, patching a container feels like you're treating it like a pet. You should just kill it and get a new one instead. [1] https://thenewstack.…
The whole point of having a distro with a good reputation is that you can incrementally outsource your trust to them, one package at a time. OpenSSL needs a fix? One package update per machine, and if you have lots of machines, you update that. With VMs, you can use the same tools you use with any other machine. With containers, you are responsible for figuring out which ones need which packages and how to re-build t…
I don't understand why you think it's hard to tell your CI/CD pipeline to run itself on a schedule. I find this totally consistent with my tooling (I have CI/CD in place for many different reasons, this is one of them) and I don't know how this CI/CD cron makes me "packaging maintainer and infrastructure maintainer".
"With containers, you are responsible for figuring out which ones need which packages and how to re-build them."
Yeah, you need to know how to rebuild your world in ANY CASE. It's not an argument against containers that they require an approach that is the best practice.