Knightmare: A DevOps Cautionary Tale (2014)

Part of the solution is at the level of attitude, just one more productive than "don't fuck up"

To create a contrived example, say someone reads your note on replacing bash scripts and decides they agree with the principle.

They go into work tomorrow, their fellow engineer agrees on the technical merit, and they reimplement a bunch of bash scripts in Rust with a suite of tests bigger than anyone imagined, and life is great.

... fast forward a few months from now and suddenly a state the bash scripts were hiding flares up and everyone is lost, and type safety didn't help.

A shared culture of "conservation of value" can help in a lot of ways there. That's the attitude that creation of value is always uncertain, so you prioritize potential future value lower than currently provided value:

- instead of looking at technical merit of the new, we prioritize asking: What specific shortcomings the old way have? What can we improve downstream so that the value those systems provide is protected from invalid states we're worried about this tool generating?

- does switching the language reduce the number of people who can work on it? Do we reduce the effect surface area of the team providing value to it? When hair is on fire do we know the sysops guy won't balk?

- when it goes down, with a culture of "conservation of value", your plan A is always rolling back, there's no back and forth on if we can just roll this one fix. If you cause the company to lose a million dollar trade, it's already codified that you made the right decision

Obviously these are all extensions of a contrived example, but to me culture is heavily utilized as a way to guide better engineering.

I think these days people tend to think in terms of culture that affirms, as a reaction to cultures that block anyone from accomplishing anything: to me a good engineering culture is one that clashes with what people want to do just enough to be mildly annoying.

As expected it seems many here, even you couldn't figure out what the '/s' means even as I preempted it in advance before anyone comes and tries to claim it anyway.

So even putting the '/s' to denote sarcastic intent doesn't work on HN. Can't even take a joke here.

Wild west times ! It's worth noting, that things changed a lot in trading systems since then.

When I started working in this domain (2009), it was pretty crazy how unreliable those systems were, on all sides - banks, brokers, exchanges. Frequently you needed to make sure over the phone, what quantities got executed etc.

I remember when the Italian exchange was rolling out their systems, at some point we did "tests" on a mix of production and UAT - if my memory is correct, we were just changing IPs to which to connect for order passing to test for the upcoming release, after the market closed. We couldn't just test in their UAT environment, since it was so bugged and half down most of the time.

And let's not even talk about Excel spreadsheets with some VBA code that would make chatGPT swear, that were pricing instruments with volumes traded with a lot of zeros.

It's very different nowadays, in part thanks to stories like this one. Most things are automated, and there is much less cowboy's attitude.

There are mandatory kill switches, a lot of layers of risk / trading activity monitorings (on your side, on exchange side), and really a lot of hard learned lessons incorporated into the systems. That's also part of the reason why people sometime tend to be naive about how hard it is to build a good trading system - the strategies are sometimes now really smart - it's mostly about how to avoid getting killed by something that's outside of usual conditions.

Early theory about the cause (incorrect, as it turns out):

Nanex ~ 03-Aug-2012 ~ The Knightmare Explaned - https://news.ycombinator.com/item?id=4337359 (no comments)

I worked in fintech for a few years. I'll never again work on software that's responsible for trading, you could offer $1M/year and I wouldn't take it. By far the most stress I've ever experienced at a job.