Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

161–170 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#161

Earlier quoted context omitted.

I know that. I am referring to you and I, the reader accepting the word of the anonymous source. Combined with the fact that they apparently did not ask for a comment from the NYSE before publishing this. Or if they did, they neglected to mention it.

We aren’t accepting the word of the anonymous source. We’re accepting Bloomberg’s word that the source is reliable.

Some of us aren't "accepting" anything. We're just reading about a potential cause of an incident and speculating about how it could happen to us or could have been prevented. Just because we're reading this article and commenting here doesn't mean we just believe everything that we read. The post-mortem will come out soon enough and we'll read that and comment again.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#162
post #23

Earlier quoted context omitted.

This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.

The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.

This is because the CEO's core job is to raise stock price. Nothing else. They hired in covid and profits & share price spiked due to the economic state at the time. Now the economic state has changed so they fire employees and the stock goes up. By that metric, the CEO will get a bonus at the end of the year. CEO does not get a bonus for not laying people off. Employees are not humans once you get to the csuite. An employee can be a person but multiple employees are just numbers on a ledger. They just send out "I'm sorry" emails to placate the masses and to get good media, no one really cares if the lower level people are upset. You only count once you get to a certain level.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#163

Earlier quoted context omitted.

Wouldn't working with systems that keep the largest stock exchange for the largest economy in the world running where a simple mistake can cause "mayhem" when the market opens be considered more "tangible" than working in AI or on self-driving cars? It just doesn't have as much street cred as working on those particular projects in the tech community.

Not necessarily. If you're the type that's into finance, then sure, that might get you out of bed in the morning. I'm not into finance and kind stand the culture that surround finance. Yes, it's big and touches every single one of us, but doesn't mean I want to embrace it and go to work in it every day. If I can take that same skill set and apply it to something with a much better culture surrounding it that affects…

>if the NYSE did not exist, the world would continue to turn

This is startlingly ignorant of the complex machine that is the modern economic system. If something like the NYSE was to shut down today it would be pandemonium.

There is a difference between 'I don't understand how something works' and 'I don't understand how something works, so it is worthless'. The former is healthy and the first step to understanding, the latter is ignorant, and the first step to getting more ignorant.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#164
post #126
post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.

Blame cultures and process cultures are both problems in different ways. Blame cultures don't care about individual accountability, only that someone suffers. Process cultures only care that no one suffers, not that individuals are accountable. Both have some misguided notion that something other than personal accountability can lead to good results. Misattributed blame and suffering does not deter poor performance or mistakes. Not even correctly aimed punishments are very good at that. Accountability isn't about punishment, it is about limiting power to the level of responsibility demonstrated. Rules and procedures don't prevent poor performance, they can in fact entrench and guard it, and they only mildly impact mistakes. Best practice can mitigate mistakes to the same extent or better (due to easier adaptability), but people keep trying to turn them into rules, and that has to be fought. If you followed all the rules but didn't get the job done, you still shouldn't be handed the same task again, but not out of blame.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#165
post #4

Earlier quoted context omitted.

More automation -> more code -> more things to go wrong

The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.

Knight Capital will forever haunt fintech engineers... https://www.henricodolfing.com/2019/06/project-failure-case-...

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#166

Earlier quoted context omitted.

As somebody who worked with them as a client, I can confirm this. There is currently a spec-level bug with their core Pillar engine and it was essentially bounced between several different teams and ultimately ignored as nobody's problem.

So basically like any other medium to large company? This doesn't sound unique in the slightest.

I would think that the company being a securities exchange would factor into the analysis. Don't you?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#167
post #166

Earlier quoted context omitted.

So basically like any other medium to large company? This doesn't sound unique in the slightest.

I would think that the company being a securities exchange would factor into the analysis. Don't you?

How does them being a securities exchange in any way affect the analysis of their software engineering practices? They're not some special snowflake, they can suffer the same software engineering and business process issues as other companies.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#168

Earlier quoted context omitted.

Not necessarily. If you're the type that's into finance, then sure, that might get you out of bed in the morning. I'm not into finance and kind stand the culture that surround finance. Yes, it's big and touches every single one of us, but doesn't mean I want to embrace it and go to work in it every day. If I can take that same skill set and apply it to something with a much better culture surrounding it that affects…

>if the NYSE did not exist, the world would continue to turn This is startlingly ignorant of the complex machine that is the modern economic system. If something like the NYSE was to shut down today it would be pandemonium. There is a difference between 'I don't understand how something works' and 'I don't understand how something works, so it is worthless'. The former is healthy and the first step to understanding,…

Relax, Ayn Rand cum True Believer complex.

The current state of business within the current iteration of how people interact with one another isn’t some necessity.

Yes, the world may fall apart for a relatively brief moment in the grand scheme of things — but then life will go on.

The first step to understanding this is to drop the superiority complex.

Very little is actually needed to keep the world turnin.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#169
post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Curious why the link to ICE.COM?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#170
post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

I've thought about getting into this... The stuff they work on is so incredible to me. Here's a quote from their Pillar product page: > Up to a 95% Reduction in Latency: The roundtrip latency on NYSE Pillar order entry sessions via Pillar matching engines has been reduced from ~592μs to ~32μs for FIX and from ~96μs to ~26μs for Binary, getting client orders into the market much faster. With a 92% improvement in the 9…

> Reading stuff like this makes my current work feel stupid by comparison.

It makes our economic system seem stupid. Jesus, we're not calculating astrophysics or quantum mechanics. A made-up system should not require or depend upon this kind of speed or precision. Maybe we should chill.

Reminds me of those pro StarCraft players who keep unnecessarily clicking the mouse to keep their APM (actions per minute) stat high.

Post reply on HN