Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

Post reply on HN
Print view View original

151–160 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#151

Post by JBlue42 »

anonred wrote: post #127
ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

>If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars

Maybe they aren't as smart as they think they are? Or they find that there are interesting problems to solve in fintech? Problems they can tackle and see resolved in a realistic time frame vs 'tangible' (?) self-driving cars or chat bots.

I know AI encompasses a far larger range of things but right now, what problems is it solving? Artists, writers, and others can do that work. What do self-driving cars resolve beyond continuing the dominance of car culture in a world that could have better public transit and safer infrastructure?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#152

Post by quantgenius »

yourapostasy wrote: post #139
quantgenius wrote: post #110

Earlier quoted context omitted.

It's very easy to talk about completely automated systems and LOTO and you need these when you have under-skilled staff. The NYSE likely does NOT have under trained staff. If you have LOTO systems etc, what do you do when a sensor fails and you can't figure out why your method for checking whether the other system is running incorrectly thinks it is. Do you allow the stock market to simply not open? What if multiple…

> If you have LOTO systems etc, what do you do when a sensor fails... It pretty much boils down to: it depends upon what the business wants to prioritize; operating margin or resiliency. There is an entire subfield investigating the statistical foundations of resiliency, and the general case of N-modular redundancy is in practice implemented as triple modular redundancy in most commercial systems that want to spend i…

You don't get to major failures once a decade (or less) on systems this complex without understanding and in fact being on the cutting edge (likely ahead of what you read in journal articles written by academics) of the statistical foundations of resiliency, n-modular redundancy etc.

In real-life outside of a journal article, it's a lot harder than just deciding whether you want to prioritize operating margin or resiliency at 5000 feet.

In real life when these sorts of edge cases happen, you have to understand in minutes or sometimes seconds the tradeoffs in terms of costs to your own company and your customers of one of n specific possible failure modes and risk-manage so you minimize the probability of the catastrophic outcomes. This sometimes may involve increasing the probability of low cost bad outcomes. You can't reason about this stuff before hand. If you could, you would have designed your system to not fail in that manner.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#153

Post by JumpCrisscross »

wrldos wrote: post #23
mellosouls wrote: post #22

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.

> individual's mistake can take out your business

It didn’t.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#154

Post by dylan604 »

godshatter wrote: post #150
anonred wrote: post #127

Earlier quoted context omitted.

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

Wouldn't working with systems that keep the largest stock exchange for the largest economy in the world running where a simple mistake can cause "mayhem" when the market opens be considered more "tangible" than working in AI or on self-driving cars? It just doesn't have as much street cred as working on those particular projects in the tech community.

Not necessarily. If you're the type that's into finance, then sure, that might get you out of bed in the morning. I'm not into finance and kind stand the culture that surround finance. Yes, it's big and touches every single one of us, but doesn't mean I want to embrace it and go to work in it every day.

If I can take that same skill set and apply it to something with a much better culture surrounding it that affects people in a positive way, then I would definitely choose that over finance any day of the week and twice on Sunday.

At the end of the day, if the NYSE did not exist, the world would continue to turn. It's just not that big of a deal to a heck of a lot of people.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#155

Post by jonpo »

Its nearly always the damn humans. I find it awful that we are often assigning blame to some "technology error" when its the damn humans pulling the strings all the time. all those times the market shut accidentally. or the market opens at the wrong time. or that time someone accidentally deletes all the GTC orders in order to "save some disk space". that time someone tests opening the market at the weekend and puts in the wrong date. Sometimes we are just trying to test that the things work and so we take awful risks like adding test orders, or failing over to test that backup versions of the trading infrastructure still work. All these things add human execution risk.

That said I find the US market structure is unfair Charles Schwab does protest too much. Retail orders never seem to get near the central order book. there is no direct market access. brokers just sell your order to whomever MM pays them for the spread in return for a kickback. this should be a fantastic fair multiplayer game, but instead its pay to win mobile crap with vested interests milking their customers.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#156

Post by psychlops »

This and other exchanges need to be running 24/7, in large part to level the trading field for retail investors. The backend should be handled invisibly behind the scenes. There should further be an exchange version of a Netflix chaos monkey running constantly to ensure such a critical infrastructure is robust.

The fact that these systems do not exist is an exchange problem, not a "staffer".

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#157

Post by jrochkind1 »

khold_stare wrote: post #104
jrochkind1 wrote: post #87

> That misled the exchange’s computers to treat the 9:30 a.m. opening bell as a continuation of trading, and so they skipped the day’s opening auctions that neatly set initial prices. I didn't even know about this process. I don't know much about trading, but it surprises me that there is a separate process for setting prices at the start of trading, and that if it's missed, chaotic prices result. Is this related to…

Worked in HFT for a few years. The reason why most markets are not open 24 hours is more human, and just historical - aligned with people's 9-5 workday. There are also pre open and post close sessions of trading but it's much less liquid. Futures markets are open almost 24 hours. Even there, it's down for some time daily. Personally I think it's actually inertia that keeps existing markets this way - the systems of t…

Very helpful and clear, thank you.

> Now, as a result, there needs to be a way to set the opening price and closing price, like a bootstrap process. A smaller version of this process actually happens every time a stock gets halted and resumed.

So this suggests that if you did have a hypothetical exchange that ran 24/7... and something unusual happened to make trading halt completely (which always is going to happen occasionally, whether 9/11 level or more frequently)... you would still need to have that "bootstrap" process in place to re-start trading.

But if you normally ran 24/7, you'd have a process that you maybe had never used, or hadn't used in years!

This maybe provides another justification that isn't just historical for having exchanges shut down every day. So you are at least testing the bootstrap process daily, you don't have a bootstrap process you're going to need in an emergency (the worst time to have further problems) that has actually just been sitting around unused for years!

(Reminding me of making sure you test your backup and continuity processes regularly, right? And the irony here is that it's the backup/continuity processes which are alleged to have caused the issue here! but still, you need the backup/continuity processes...)

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#158

Post by bagacrap »

psychlops wrote: post #156

This and other exchanges need to be running 24/7, in large part to level the trading field for retail investors. The backend should be handled invisibly behind the scenes. There should further be an exchange version of a Netflix chaos monkey running constantly to ensure such a critical infrastructure is robust. The fact that these systems do not exist is an exchange problem, not a "staffer".

why do retail traders "need" to exist?

Retail traders realistically have only luck to rely on to beat hedge funds and banks. What they do is akin to gambling, which is on net quite negative for those who participate in it and heavily regulated. Retail traders don't serve any purpose in our society. They don't help with efficient allocation of capital and anyone who might be an actual savant in trading can join or start a firm rather than staying independent and unlicensed.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#159

Post by bob1029 »

ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

I've thought about getting into this... The stuff they work on is so incredible to me.

Here's a quote from their Pillar product page:

> Up to a 95% Reduction in Latency: The roundtrip latency on NYSE Pillar order entry sessions via Pillar matching engines has been reduced from ~592μs to ~32μs for FIX and from ~96μs to ~26μs for Binary, getting client orders into the market much faster. With a 92% improvement in the 99th percentile latency results, clients can also have more confidence in improved performance consistency regardless of market conditions.

Reading stuff like this makes my current work feel stupid by comparison.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#160

Post by barneygale »

anonred wrote: post #127
ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

You answered your own question. The only motivation is greed.
Post reply on HN