Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

Post reply on HN
Print view View original

131–140 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#131

Post by horsawlarway »

DrBazza wrote: post #125
zaroth wrote: post #107

Earlier quoted context omitted.

But it’s absurd. Companies are not supposed to only ever hire . Some things are cyclical and you need more people for some amount of time, and then you find you need less. It’s not always predictable/seasonal like farming or holiday rush. Is it wrong for a company to respond to market effects? That there was a layoff isn’t necessarily a sign a company did anything wrong… I think how they actually do the layoff certai…

It’s not hiring though. It’s overhiring. I’ve forgotten which FAANG it is. But one of them still has more employees than last year even after layoffs. It’s offensive.

It's a response to extreme demand during covid. When - you know - online service usage was at all time highs because everyone was stuck inside and doing things online.

It was likely the right call to hire then, just like it might be the right call to reduce headcount now.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#132

Post by ynniv »

throwaway122095 wrote: post #128
ngz00 wrote: post #126

Earlier quoted context omitted.

I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.

As somebody who worked with them as a client, I can confirm this. There is currently a spec-level bug with their core Pillar engine and it was essentially bounced between several different teams and ultimately ignored as nobody's problem.

Unlike all of the "modern tech company" problems which are never ignored and only solved when someone's problem goes viral on social media.

They're a big company, some groups are better than others, some customers get more attention than others.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#133

Post by khold_stare »

johnbcoughlin wrote: post #124
khold_stare wrote: post #104

Earlier quoted context omitted.

Worked in HFT for a few years. The reason why most markets are not open 24 hours is more human, and just historical - aligned with people's 9-5 workday. There are also pre open and post close sessions of trading but it's much less liquid. Futures markets are open almost 24 hours. Even there, it's down for some time daily. Personally I think it's actually inertia that keeps existing markets this way - the systems of t…

Matt Levine suggested that the chaos after opening was mainly due to market orders executing at ridiculous prices. Like, a limit buy for half the "real price" is the first buy order to get in the door, and that gets matched with a market sell order. Does that track with your understanding?

Yes! I almost forgot about market orders because trading firms never use market orders for this exact reason - you have no control over the price if things go bad. Most flash crashes are exacerbated by runaway market orders and stop orders for example.

A buy market order would try to match with the "best price" which in a deeply crossed book would mean matching with a really low priced sell order. Exchanges match orders in price-time priority. Similar is true for a market sell order - would match at an extreme high price.

Besides the midpoint of the order book, another metric for a "current price of the stock" people use, is the "last trade price". In the situation above you would get "swings" in the price because market orders would be trading very high and very low if they alternate between buying and selling. The data structure on the exchange itself isn't "swinging", it's just the overlapping region being slowly eroded by market orders. The "last trade price" metric looks really insane in this situation.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#134

Post by misja111 »

anonred wrote: post #127
ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

You're not making it easy to get any answers, if you cut out 2 of the main reasons for people in general to change jobs.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#135

Post by mynameisvlad »

throwaway122095 wrote: post #128
ngz00 wrote: post #126

Earlier quoted context omitted.

I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.

As somebody who worked with them as a client, I can confirm this. There is currently a spec-level bug with their core Pillar engine and it was essentially bounced between several different teams and ultimately ignored as nobody's problem.

So basically like any other medium to large company? This doesn't sound unique in the slightest.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#136

Post by ynniv »

ngz00 wrote: post #126
ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.

Having been in the industry for a couple decades, and having worked at both, they're not all that different. Some groups are going to be better than others in the same company. Some companies are floating on venture money today, and might disappear tomorrow. Most technologies constantly cycle. Our experiences working at the same company were different.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#137

Post by ynniv »

jacquesm wrote: post #129
ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Just naming a 'staffer' though seems to already be a way to apportion blame to a segment of the employees, insulating management from what was done. Named or not doesn't really matter, clearly blame is being assigned.

I think those are Bloomberg's words, or their paraphrasing of the grapevine. The high level people that I knew there weren't petty, and everyone was of the opinion that it didn't matter who clicked the button: we were all in the same boat.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#138

Post by randomdata »

j33zusjuice wrote: post #109
lotsofspots wrote: post #38

Earlier quoted context omitted.

Oh, they take full responsibility, it always says so in the mails they send out. It's just that taking responsibility doesn't appear to actually result in anything happening.

Their punishment is in bearing the shame of having been wrong. That’s the price of leadership.

What shame is there in being wrong? Being wrong is the ideal state, paving a path to gaining an education, which is a source of pride and a benefit.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#139

Post by yourapostasy »

quantgenius wrote: post #110
yourapostasy wrote: post #76

Earlier quoted context omitted.

In this specific case, I don't think that's necessarily the outcome. Our industry has yet to accept a universally-acknowledged equivalent of a lockout/tagout (LOTO) interlock. There is no need for a bureaucracy if we have cryptographically-enforced multisig Shamir secret sharing keys where a LOTO prevents (in this case) a system from spinning up while another system (the backup system apparently in this case) is runn…

It's very easy to talk about completely automated systems and LOTO and you need these when you have under-skilled staff. The NYSE likely does NOT have under trained staff. If you have LOTO systems etc, what do you do when a sensor fails and you can't figure out why your method for checking whether the other system is running incorrectly thinks it is. Do you allow the stock market to simply not open? What if multiple…

> If you have LOTO systems etc, what do you do when a sensor fails...

It pretty much boils down to: it depends upon what the business wants to prioritize; operating margin or resiliency. There is an entire subfield investigating the statistical foundations of resiliency, and the general case of N-modular redundancy is in practice implemented as triple modular redundancy in most commercial systems that want to spend in this vector.

> Like it or not, these highly trained individuals are going to make mistakes every now and then.

Absolutely, and here is where the organization's no-blame learning culture swings into action for the well-led teams.

> It's a lot harder than it looks.

We all know this, and we can all help each other get better to deliver ever increasing value to our customers by sharing what works for the context we deployed within!

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#140

Post by ynniv »

anonred wrote: post #127
ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

I'm not recruiting for them, just sharing my experience. I included their careers link for people who might be interested because I know they're always looking for good engineers.
Post reply on HN