Etcd, or, why modern software makes me sad

Disclaimer: I'm was part of the Apache Aurora PMC before the project went into the attic.

Not sure what you mean by declarative infrastructure but eventual reconciliation has been implemented in at least one other project (now retired) called Apache Aurora[1] which runs on Apache Mesos[2].

Twitter ran (and probably still runs) a combo of these to huge scale and great success so I don't think it's fair to say that there's nothing else around that does that :).

K8s is great, but we should be careful not to rewrite history.

[1] http://aurora.apache.org/

[2] http://mesos.apache.org/

It took me a bit to understand what you were saying here:

> I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms.

...but I think I got it now. I was arguing with the wrong thing here.

Yes, there's a reason that FTP and RTP do what they do. (Though in FTP's case it had more to do with TCP-port parity gender; multiple data sockets per control socket was just a convenient outgrowth of the existing architecture.)

But crucially, HTTP/{2,3} are about accelerating existing traffic; and existing HTTP "apps" (the HTML + JS + CSS that contain an implicit dependent-resource digraph that gets turned into flow-requests by the browser) have no indication in them for what's going to be a "control-plane" request vs. a "data-plane" request. HTTP itself is an oblivious carrier re: these use-cases. So HTTP/{2,3} must continue that trend, and be oblivious carriers as well. To do otherwise would force a rewrite of all HTTP "apps" to add such indications.

However, in HTTP/1.1, you got the benefit of this purpose-isolated signalling "for free" anyway, because you'd probably be making your data requests from a CDN origin, while making control requests to an API-subdomain origin. Those are separate origins, so they got separate (keepalive, pipelined) TCP sockets.

In HTTP/{2,3}, you can choose to shove everything into one socket by putting both your control-plane and your data-plane into the same origin.

But, crucially, you don't have to. Nobody's stopping your HTTP "app" from just having separate control and data origins; and so ending up with two HTTP/{2,3} sockets open, one for control signals, the other for BLOB data.

And, of course, existing sites previously engineered for HTTP/1.1 won't shove everything into a single socket, since they've already separated the control origin from the data origin. (They've likely got several data origins, so they'll likely want to re-engineer with upstream proxying to compact those down into one data origin; but all they have to do while doing this is to choose not to also compact the control origin in there as well. The choice to not change anything is, in this case, the right choice!)

And also keep in mind that most websites aren't, like, photographer's portfolios or direct video hosts. There are few websites directly embedding same-origin BLOBs. There are many websites embedding "Binary Not-so-large OBjects", like blogs that embed same-origin 500px-wide medium-quality optimized-JPEG images. And those websites don't have such a large disparity between control signalling and data signalling that they have any need to separate them—those images aren't much larger than the average compressed-JSON XHR response!

And because of that, the "default" advice for HTTP/{2,3} transitioning (that you should follow if you're 99% of websites) can just be "put everything on one origin"; whereas "split control signals to a different origin, thereby forcing it to a separate socket with separate TCP window scaling parameters" can just be one of those arcane PageSpeed things you learn from an infrastructure consultant, a bit like choosing what type of select(2)-alike kernel ABI your webserver should use to receive packets.)

Having an additional layer of restriction (be it an alloation and workload scheduler or whatever) is additional security. The fact that there are vulnerabilities that bypass layers of security is irrelevant. The power button on the machine is a vulnerability to your software. The fact the restrictions are possible with other tools is irrelevant. Grouping functionality into a singular paradigm has utility.

To be fair, more security by additional tooling adds a vulnerability in human error, but that's true of all the tools.

First time I’m aware of that reading the footnotes provides at least as much meat and interest as the article itself. There’s even footnotes in the footnotes.

I don't really use k8s nor etcd, but one sentence in the article really stood out to me:

> I would go so far as to say that Kubernetes is the worst thing to happen to system administration since systemd.

I know it's popular to shit on systemd, but as a casual user, I honestly don't get it. I've written about a dozen systemd unit files over the years, and each time it's been a pleasure. Write a few lines describing what to launch when, one command to make systemd aware of your new file, one command to launch it.

In contrast, I've written a couple sysvinit files, and the experience was horrible every time. Having to manage pid files and logging myself, setting up the proper symlinks, etc. It'd always end up being a ~100 line script that'd be 99% copy-paste.

There might be some use-case where systemd is worse, but for my (admittedly simple) use-case, it's far superior.