The sad state of sysadmin in the age of containers (2015) • Hacker News Board

Re: The sad state of sysadmin in the age of containers (2015)

#141

Post by turk73 » Mon, Dec 09, 2019, 2:13 PM UTC

_wldu wrote: post #49

Standard Makefiles are really under appreciated today. They are simple and can be used with most any language, version controlled and have been around for decades.

Make is possibly the worst of the breed. I would not want to go back to messing with Makefiles when so much has happened in build as code. Just look at how Maven revolutionized builds with the concept of convention over configuration. Make is purely configuration. The whitespace issues in Makefiles used to drive me batty, which alone was a reason to not use Make.

Re: The sad state of sysadmin in the age of containers (2015)

#142

Post by colinjfw » Mon, Dec 09, 2019, 2:13 PM UTC

jwr wrote: post #19

I recently had a similar discussion with people using npm for building a CSS framework library. I tried to explain the concept of getting a pre-downloaded tarball and using "make" (or similar) to produce target artifacts from source files in a deterministic, repeatable and reliable manner, without relying on any third-party servers being available and without pulling in dependencies that might have changed. It seems…

Not alien to me, likely younger.

Re: The sad state of sysadmin in the age of containers (2015)

#143

Post by PaulHoule » Mon, Dec 09, 2019, 2:14 PM UTC

Uh... I've built Hadoop from source, it is not that bad.

Re: The sad state of sysadmin in the age of containers (2015)

#144

Post by ekidd » Mon, Dec 09, 2019, 2:14 PM UTC

jwr wrote: post #19

I recently had a similar discussion with people using npm for building a CSS framework library. I tried to explain the concept of getting a pre-downloaded tarball and using "make" (or similar) to produce target artifacts from source files in a deterministic, repeatable and reliable manner, without relying on any third-party servers being available and without pulling in dependencies that might have changed. It seems…

> without relying on any third-party servers being available and without pulling in dependencies that might have changed.

There are two different issues here:

1. Not pulling in changed dependencies. This is what "lock files" are for: To limit builds to known version of every dependency. npm was terrible about this for a long time. Most other language package managers are better.

2. Not relying on third party servers to be available. Personally, I've mostly worked with Ruby's bundler and Rust's cargo, and in over 10 years, I've lost maybe two days of work because of package server outages. That's less than I've lost due to S3 outages, less than I've lost due to broken backup systems, and less than I've lost due to complex RAID failures. For the clients and employers in question, this was acceptable downtime.

In the rare cases where one day of noticeable build downtime every 5 years is unacceptable, then it's usually possible to just "vendor" the dependencies, usually by running a one-line command.

For many small to midsize businesses (and many growing startups), a small risk of brief outages is an acceptable engineering trade-off.

Re: The sad state of sysadmin in the age of containers (2015)

#145

Post by CathedralBorrow » Mon, Dec 09, 2019, 2:14 PM UTC

zeveb wrote: post #128

cuillevel3 wrote: post #111

> I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths. Sadly these ways of olde didn't scale. They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'. > And since nobody is still able to compile things from scratch, everybody just downloads precompiled binaries from random websites. Often wit…

> They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'. Those old ways of doing things were indeed slower than they needed to be in many cases, and I'm glad that we sped them up. I don't believe that we needed to throw the baby away with the bathwater though. The problem now is that we build systems which are unmanageable and unmanaged. 'Throw it in a con…

> The problem now is that we build systems which are unmanageable and unmanaged

Are all the systems out there in production actually unmanageable and unmanaged?

Re: The sad state of sysadmin in the age of containers (2015)

#146

Post by duelingjello » Mon, Dec 09, 2019, 2:15 PM UTC

Yes, it's worse than that on a couple of fronts. Cloudera, Amazon and others benefit financially from situations of over-complicated messiness, while average SWEs, SAs and others are left with wasted time, money and lack of trust. And worse, lax engineering and operations practices introduce attack surfaces to fraudsters, thieves, organized crime and state-level actors to backdoor, destroy, steal, extort and monetize infrastructure of organizations large and small.

Re: The sad state of sysadmin in the age of containers (2015)

#147

Post by cuillevel3 » Mon, Dec 09, 2019, 2:19 PM UTC

zeveb wrote: post #128

cuillevel3 wrote: post #111

> I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths. Sadly these ways of olde didn't scale. They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'. > And since nobody is still able to compile things from scratch, everybody just downloads precompiled binaries from random websites. Often wit…

> They could only do it for a handful of servers, change request took months and the systems were not to be 'touched'. Those old ways of doing things were indeed slower than they needed to be in many cases, and I'm glad that we sped them up. I don't believe that we needed to throw the baby away with the bathwater though. The problem now is that we build systems which are unmanageable and unmanaged. 'Throw it in a con…

After a few decades it's no longer a baby ;)

Software is becoming more complex and moving faster than ever. 'Controlling your dependencies' was always an illusion and a trade-off at best. Often a trade-off against security, inheriting the folder of JAR files from your predecessor and such.

I think of containers and all these new tools as designs which are supposed to help us manage existing complexity. They don't create it, it's already there in the requirements and real-world deployments. The friction we feel, is that some of those tools are not very good (yet), but maybe better than the previous generation (sometimes). On the negative side, there is a lot of nostalgia ("ah, do you remember bare metal...") and unwillingness to change and learn.

Re: The sad state of sysadmin in the age of containers (2015)

#148

Post by CathedralBorrow » Mon, Dec 09, 2019, 2:21 PM UTC

weberc2 wrote: post #93

Tade0 wrote: post #35

Earlier quoted context omitted.

Truth be told the whole front-end ecosystem relies on those servers being available, so in their absence any development everywhere would grind to a halt - as it almost did during the left-pad scandal. While I agree that this is not exactly the sanest approach, within the ecosystem there's no incentive to work differently. Also, like someone else mentioned - dependency hell only got worse over time - setting up a new…

There’s a pretty straightforward solution to this problem which is to run your own NPM server even if it’s only a caching proxy. For some orgs, the uptime provided by the third party servers is sufficient.

Maybe it's obvious and straightforward enough that other people have come up with the same idea, tried it and not found it to be worth the effort?

Re: The sad state of sysadmin in the age of containers (2015)

#149

Post by oauea » Mon, Dec 09, 2019, 2:22 PM UTC

mkesper wrote: post #136

oauea wrote: post #76

Earlier quoted context omitted.

Just build your own. It's not that hard. Before containers we would script the install process using Ansible. With containers we script the install process using a Dockerfile. If you know how to do one, you know how to do the other. Just don't be lazy and use unofficial images created by people you don't know or trust. Just like you wouldn't pull any random role from ansible-galaxy. Or pull a random guy off the stree…

You can even avoid dockerfiles by using Ansible: https://docs.ansible.com/ansible/latest/scenario_guides/guid...

We tried that about a year or two ago with `ansible-docker`. At the time it didn't leverage the docker cache so it was quite slow. Looks like that got replaced with `ansible-bender` which according to the readme does use the docker cache!

Thanks for mentioning this, will definitely have to give this another go.

Re: The sad state of sysadmin in the age of containers (2015)

#150

Post by marcosdumay » Mon, Dec 09, 2019, 2:24 PM UTC

lmm wrote: post #89

marmaduke wrote: post #86

Earlier quoted context omitted.

No I don't. I use Debian derivatives because I can apt get all my stuff without thinking hard, because those package maintainers have done the hard work.

Well, clearly not, since you can't install Hadoop. Before you blame that on Hadoop, remember that it only requires some really quite basic things from its package manager, which apt is nevertheless completely unable to do: use libraries in the normal recommended way, play nice with Java, work cross-platform.

Whatever. (Great practices those, being stuck in a decade old JVM and requiring root to build! But well, whatever.)

If I can not install on Debian in a clean way, I will just try to avoid using the software (and everything that comes on the ecosystem). It may be unavoidable, but if there is an alternative, it will be preferred. If one alternative appears after I am dealing with it for some time, it is still preferred (because the work with out of distro software never ends).