NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

The difference is that one asserts authority over the other.

Worked in HFT for a few years. The reason why most markets are not open 24 hours is more human, and just historical - aligned with people's 9-5 workday. There are also pre open and post close sessions of trading but it's much less liquid. Futures markets are open almost 24 hours. Even there, it's down for some time daily. Personally I think it's actually inertia that keeps existing markets this way - the systems of the exchanges and participants were designed with the assumption that they will have daily downtime, so it's hard to change. It's also dependant on how banking and settlement works - a lot of stuff happens after the trading ends. Batch processes run as different institutions settle their trades between each other, etc etc.

Now, as a result, there needs to be a way to set the opening price and closing price, like a bootstrap process. A smaller version of this process actually happens every time a stock gets halted and resumed.

An exchange has an order book - orders of things people want to buy and sell at different prices. During normal operation the buy and sell orders don't overlap in the order book - if two people want to buy and sell at the same overlapping price, they just get matched by the exchange at that moment. Unmatched orders stay in the order book data structure until a matching order comes along. The "price" you see in charts is just the midpoint between the highest buy and lowest sell price in the order book.

Now, if the order book is empty, what the heck is the price? That's what the opening auction needs to solve. The way it works is that people can start placing orders ahead of the opening bell, but they won't get matched until the open. So before the open, the order book is getting filled with orders, but crucially the _orders will overlap_. This "crossed" order book is a no no during normal trading, but ok before the opening auction. When the auction comes, a price is picked which maximizes the amount of orders filled (it's more nuanced than that, but bear with me). Imagine you pick a price in the overlapping region of the order book - every buy order that has a higher price than that will match with every sell orders that has a price lower than that. They will get matched and executed at the opening price, and BAM, you have an uncrossed order book, full of orders.

If the auction doesn't happen, and you just open the stock, then all hell breaks loose. Many things can go wrong here. Firms connected to the exchange may have code that assumes a book is not crossed (or at least not as crossed as it would be during an auction) causing wild behavior. The exchange itself could start matching orders haphazardly in the overlapping region, causing those "price swings" that the article talked about.

Can't imagine the panic that day haha.

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings.

In a week or so there will be a comprehensive internal post mortem, and every engineer in the company will read it because that's why they work there. "The staffer" will not be named, nor will they be fired. The process will be changed. The systems will be changed. You probably haven't heard of Pillar, but the NYSE in your head was replaced by some pretty amazing, distributed, low latency systems. The culture is to over-engineer, over-provision, plan for black swans. And test. That it works. Test that it scales. Test that backups work. Test, test, test. firmitatis, utilitatis, venustatis. This failure was due to daily testing.

Sometimes things still fail. That's true anywhere. In most places your failures don't make the papers, and accidents are swept under the rug. That doesn't happen at NYSE for obvious reasons. They're not building large language models (that I know of), or self driving cars (pretty sure on this one), but they're a modern, cutting edge, "soft" real-time engineering shop. If you haven't looked already, you might find something interesting there: https://www.ice.com/careers

I don't imply that there is a solution.

We will simply create a second system (B) to monitor the first system (A). Now we have two systems to maintain. System B will not be capable of steering A by itself. So we still need to know how to diagnose and repair A, and we also need to know about B too. Maybe system B can talk to a Prometheus/Grafana stack (if it's up). And that can put alerts into Slack (which we ignore because there's always alerts in Slack). And after standup we can take turns looking at graphs with consternation.

> Stuff like this will happen more and more. We treat software driven systems rather recklessly.

That sentence is where I go when I hear the word 'automation'.

It's very easy to talk about completely automated systems and LOTO and you need these when you have under-skilled staff. The NYSE likely does NOT have under trained staff. If you have LOTO systems etc, what do you do when a sensor fails and you can't figure out why your method for checking whether the other system is running incorrectly thinks it is. Do you allow the stock market to simply not open?

What if multiple sensors fail or it's an ambiguous situation like say you are deciding whether or not to fail over a power circuit and it's a brownout but not a complete power failure? What if there is a systemic problem and it's likely the backup power source is going to brown out too? At some point you need highly skilled individuals, like say trained airline pilots flying a plane who have the authority to override systems immediately without having to jump through hoops.

This is especially true for mission critical systems. Many of the mission critical systems we rely on are NOT built on the cloud, i.e. other people's computers because you want to be really careful about what hardware you are using, precisely how your data center is setup and want to make sure things like a noisy neighbor do not impact you.

Like it or not, these highly trained individuals are going to make mistakes every now and then. A failure like this once every decade or so really isn't so bad. The individual who made this error is likely not a "grunt". I suspect the individual in question will not necessarily suffer any major consequences as a result of this unless it wasn't a mistake but a flagrant disregard for the rules like say bringing a bottle of water into a data center that then spilled or something.

Have you built a mission critical, distributed system that hasn't failed for 10 years? It's a lot harder than it looks. That's how often the NYSE has a problem like this, about once a decade. A lot of things that work in theory, don't work for the edge cases and things that lead to problems once a decade or so are extreme edge cases.

In the grand scheme of things a mucked up opening auction is a minor problem and anyone who did not take the precaution of sending a limit order and sent a market on open order despite it being standard practice to essentially always use limits and go hurt badly will be made whole.