Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

141–150 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#141

Earlier quoted context omitted.

I know that. I am referring to you and I, the reader accepting the word of the anonymous source. Combined with the fact that they apparently did not ask for a comment from the NYSE before publishing this. Or if they did, they neglected to mention it.

We aren’t accepting the word of the anonymous source. We’re accepting Bloomberg’s word that the source is reliable.

Bloomberg's word: https://news.ycombinator.com/item?id=19526348

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#142
There is so much stupidity in the process they describe, I have no faith it will be fixed. a manual daily DR test that clearly wasn't followed by a test or checklist or double checked by another person, and leaving the DR up broke prod?? literally none of those things should have happened.

I know the world is held together with duct tape, but it's embarrassing when you see the tape fall off.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#143

Earlier quoted context omitted.

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

You're not making it easy to get any answers, if you cut out 2 of the main reasons for people in general to change jobs.

I consider these things table stakes when choosing a job.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#144
post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

There’s also geography. You go far enough East and you have mostly public sector or defense jobs. A little smattering of insurance data processing. And then fintech.

illinois.edu has one of the top rated CS programs, but once you graduate there are not a lot of options but to move to one of the coasts, or move back/to Chicago and try your luck there. Second City has a good deal of fintech.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#145

Earlier quoted context omitted.

Why couldn’t have been Jim’s fault? The world is run by Jims.

As others have said, if Jim's fuckup can have such large consequences, then there should have been backstops for Jim who would have shared the blame. Nothing against Jim, it's just he's being used as a distraction to avoid talking about the real issues.

Manager: "I hope nobody asks why Jim was allowed to have so much power with no oversight or validation"

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#146
post #11

A lot of stocks have minute bar of wide range prices in the first few minutes of continuous trading. This seems like the incident in what caused Knight Capital fiasco, in which the system repeatedly buy on the ask and sell on the bid very fast, thus pushing the high price high and low price low. In the opening usually the market maker will be more weary of the risk they cannot hedge directly and thus will be less wil…

Doesn't each trade require both a buyer and a seller, who buy and sell at an agreed on price? Presumably both parties would be satisfied with the trade so it isn't clear to me what all the fuss is about.

If I had put in a market buy/sell on open order, I'm accepting the market price, but expecting the market price to be set by an opening auction. I don't know if a market on open order would have been cancelled or just executed shortly after the bell; you could argue for both treatments and it usually doesn't come up, so it might not be mentioned in retail brokerage documentation.

Personally, I always do limit orders, but I would consider market on open/close as reasonable options. But I don't think this is typical, a lot of orders are market orders against whatever limit order is at the top of the book. Normally, that's ok, but it gets weird when things get weird, as seen here.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#147

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

We have a Slack channel where we are expected to announce all of our production changes.

Some updates are highly regimented, but a couple of the more operational teams have discretion to deploy things outside of that process, and most teams can flip feature toggles whenever they want.

Point is that sometimes people will comment, or even veto changes. We have a major customer visiting today, or the sales team is at a conference. Don’t touch anything or you might break something.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#148

> That misled the exchange’s computers to treat the 9:30 a.m. opening bell as a continuation of trading, and so they skipped the day’s opening auctions that neatly set initial prices. I didn't even know about this process. I don't know much about trading, but it surprises me that there is a separate process for setting prices at the start of trading, and that if it's missed, chaotic prices result. Is this related to…

FYI, there's a similar auction for closing, too. The closing price isn't just a race for the last trade under the buzzer; there's a process where at some number of minutes before close, you can put in orders for close or realtime, and then magic happens.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#149
post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

I worked for NYSE’s parent, ICE, and I have to agree with this. While there were many things I didn’t like about working there, the tech and the management weren’t involved in those things. A similar problem to this happened while I was working there, but it was on Endex and not NYSE. Management spoke with the responsible party, but he wasn’t fired and no punitive actions were taken against him. The blame game also wasn’t played. The team just decided to provide more eyes on the process, change the interface of the tools a bit, and move on. The company itself did face hefty fines for the screw up tho. Ultimately, the issues at ICE/NYSE are due to a highly bureaucratic structure and to onerous regulations forcing parts of that structure to exist. Given those two problems, I think ICE. does extremely well.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#150
post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

Wouldn't working with systems that keep the largest stock exchange for the largest economy in the world running where a simple mistake can cause "mayhem" when the market opens be considered more "tangible" than working in AI or on self-driving cars? It just doesn't have as much street cred as working on those particular projects in the tech community.
Post reply on HN