Ask HN: Who operates at scale without containers?

The gulf between "parallel ssh and a bunch of bash scripts" and "Ansible/Salt/etc" is a huge one.

I wrote an access layer framework that abstracts cli command delivery to nodes, has a configuration layer where env / cluster / datacenter / rack / individual node configuration settings can be applied in a cascading/overriding fashion.

I use this to do adhoc stuff (using the groovy shell) as well as write backup and restore and other automation programs. They can run from the IDE and be debugged, they can run from CLI.

I have a crude directory tree organized "workflowish" organization to spawning cassandra, kafka, zookeeper, elasticsearch, etc clusters and tearing them down.

It works better than parallel. ssh and bash scripts by far, but I'm the only one that knows it. Is it "better" than just using Ansible or Salt or Kubernetes? Uh... don't know. Ansible got thrown out as messy (it became a massive monorepo). Our stateless servers haven't even fully jumped to k8s. Salt has too many servers.

I'll try to open source it, but I doubt it will get traction. It will be useful for certain operations in Cassandra / Kafka / etc in that it is framework agnostic and isn't trying to take over the world, so we'll see.

Yes there are. Ultimately you want to put files on a server and start a process. The disposition of the scaffolding is a matter of taste. Aside from containers and virtualisation, there are two common alternatives I've seen:

1. deploy code from a tag or branch in a revision control system (perforce/git/hg etc), with OS-level dependencies and provisioning handled by one of the so-called "infrastructure as code" tools e.g. puppet/chef/salt/ansible etc.

2. package code for deployment using OS-native packages (e.g. .debs), with OS-level dependencies handled natively, and provisioning in a pre/post-install package script.

How they are similar:

* Both still allow use of userspace partitioning, whether it's a dumb chroot, jail, zone or whatever.

* Both still need some kind of workload scheduling, and a release tool.

* Both approaches are from the school of "choose boring technology".

* Compared to containers, both approaches tend to leave resources underutilized, unless the scheduler's bin packing is very good.

* Both are slightly more coupled to OS release cycles than a container or virtualisation approach to distribution.

For scaled entities the glue between the parts, any dashboards, and often the scheduler, are likely to be bespoke. Some PaaS vendors were explicitly designed around the first type (e.g. Engineyard, Opsworks) but like Mesos they're half dead now. I've seen one very large brand try to run this inside their CI/CD pipeline, which you can shoehorn in, but it doesn't fit well due to conceptual mismatch.

How they differ, in my experience:

The first is really easy to get going with, since it's barely a conceptual step beyond installing locally by hand. Your release tool may ultimately be a wrapper around git. It suffers from long deploy times (generally due to local compiles) and can be brittle, in part because config management scripts are an afterthought for many developers. Unwinding a broken deploy is (usually) horrible. Developers often like this style, but the ops tech debt builds rapidly.

The second kind (native packages) is more easily managed and certainly deploys faster, and has a lower attack surface. You need a more sophisticated and probably centralized build service+package repository (a wise architect will use whatever the OS vendor uses). Unwinding a broken deploy is (usually) easy. Developers often complain about this style because they're forced to consider operational concerns, although I personally think that's an excellent forcing function at work. If you like the idea of shipping a compiled statically-linked binary to bare metal, this is probably the best way to enable it, short of baking an entire machine image.

If it wasn't obvious from the remarks, I am personally quite fond of the second option.

This isn't a great example to cite, since traditional circular replication in MySQL is a massive anti-pattern... it's incredibly fragile and pretty much has no valid use-case.

That's especially true today when other options like Galera or Group Replication are available. But even 10-15+ years ago, mentioning circular replication was a great way to give a DBA an aneurysm. (Well, I suppose that's arguably a use-case, if you profoundly dislike your company's DBAs...)