Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

Post reply on HN
Print view View original

201–210 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#201

Post by SoftTalker »

Johnny555 wrote: post #196
ngz00 wrote: post #126

Earlier quoted context omitted.

I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.

I used to work for a small startup, and postmortems were truly no blame - engineers would talk about exactly what happened and wouldn't hesitate to put the blame on their mistakes. But as the company grew, the postmortems became more about blame since now you're not blaming an engineer, but an entire team so singling them out isn't personal. The postmortems were no longer a single engineer describing what happened in…

As organizations become larger they become more political. It's unavoidable.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#202

Post by bostonsre »

jacquesm wrote: post #129
ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Just naming a 'staffer' though seems to already be a way to apportion blame to a segment of the employees, insulating management from what was done. Named or not doesn't really matter, clearly blame is being assigned.

Yea, it sounds like an issue with process and automation. It shouldn't have been possible for the staffer to make a mistake that would cause this.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#203

Post by tgtweak »

Kon-Peki wrote: post #195
tgtweak wrote: post #182

Earlier quoted context omitted.

I don't think it's that they "left it running" like you would leave a backup app running... they literally left the entire disaster-recovery site up and running and live. Cermak (referred to in the article as the "backup") is an entire datacenter, hosting a running copy of the exchange to be used in a failover scenario. You'd have to have more than 1 person involved to forget that DR is still active when completing t…

> Cermak (referred to in the article as the "backup") is an entire datacenter Well, they are a tenent at the Cermak data center. It’s a truly massive building with huge amounts of connectivity and colo opportunities. Probably also the only 100+ year old data center building on the US register of historic places, lol (it’s a former catalog printing facility, built to hold insanely heavy printing presses on 8 or 9 real…

Yeah my point is only that it's a full DR site, not a "backup" that was left running as the article pointed out (and as a lot of commenters are insinuating).

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#204

Post by [deleted] »

thisarticle wrote: post #32
wrldos wrote: post #23

Earlier quoted context omitted.

This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.

The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.

[deleted]

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#205

Post by SpeedilyDamage »

DrBazza wrote: post #125
zaroth wrote: post #107

Earlier quoted context omitted.

But it’s absurd. Companies are not supposed to only ever hire . Some things are cyclical and you need more people for some amount of time, and then you find you need less. It’s not always predictable/seasonal like farming or holiday rush. Is it wrong for a company to respond to market effects? That there was a layoff isn’t necessarily a sign a company did anything wrong… I think how they actually do the layoff certai…

It’s not hiring though. It’s overhiring. I’ve forgotten which FAANG it is. But one of them still has more employees than last year even after layoffs. It’s offensive.

Offensive? I'm... honestly, baffled. How could one tech company's ability to hire many more people actually offend you?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#206

Post by kube-system »

I spent an hour trying to figure out why my new stock purchase had disappeared from my account. I had an order placed for opening on Tuesday morning, and I guess I was affected by the trade cancellations. Which is totally weird, because they showed up in my account on Tuesday morning after opening.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#207

Post by zmmmmm »

NoboruWataya wrote: post #50
mellosouls wrote: post #22

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

They're not mutually exclusive. A staffer leaving a backup system running may well have been the proximate cause of the issue but, if true, it was also likely a management/system issue as you say. The article is a bit strange in that it doesn't attribute the fact in the headline to any source. I don't see anything from the NYSE saying "it's all that guy's fault". On the contrary it says: > [NYSE execs] plan to examin…

yes ... if an organisation has a critical process where a single human making a mistake can cost it $millions then it has a management / process level issue not an issue with the human. Humans make mistakes. Apart from other obvious issues with it, creating a context where individual mistakes lead to horrific outcomes will create a toxic and horrifically stressful workplace - I would actively avoid working in a situation like that myself.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#208

Post by usefulcat »

DrBazza wrote: post #125
zaroth wrote: post #107

Earlier quoted context omitted.

But it’s absurd. Companies are not supposed to only ever hire . Some things are cyclical and you need more people for some amount of time, and then you find you need less. It’s not always predictable/seasonal like farming or holiday rush. Is it wrong for a company to respond to market effects? That there was a layoff isn’t necessarily a sign a company did anything wrong… I think how they actually do the layoff certai…

It’s not hiring though. It’s overhiring. I’ve forgotten which FAANG it is. But one of them still has more employees than last year even after layoffs. It’s offensive.

So if they "under hire", should they step down for that too?

Maybe they should step down any time they fail to accurately predict the future?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#209

Post by mynameisvlad »

sandworm101 wrote: post #172
mynameisvlad wrote: post #167

Earlier quoted context omitted.

How does them being a securities exchange in any way affect the analysis of their software engineering practices? They're not some special snowflake, they can suffer the same software engineering and business process issues as other companies.

>> They're not some special snowflake But they are. The consequence of a one-day or one-hour shutdown on their system is exponentially worse than most any other. I would expect them to have more rigorous systems, including more rigorous attention to development. Comparing the NYSE to any other business is like calling Fort Knox just like any other bank vault.

No company or organization is immune to bad business practices.

Them being a securities exchange does not somehow provide immunity from developing rigorous systems which have oversights, or make bureaucracy magically go away.

Likewise, the impact of an outage being more extreme does not mean the people there are infallible. Things slip through. Especially random customer requests being bounced around from team to team, the thing in question.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#210

Post by usefulcat »

helsinkiandrew wrote: post #9
bmitc wrote: post #3

Seems so weird to not have automated checks for something that seems to be described as "someone left the light on" and also not have the exchange automatically initiate itself. However, it still isn't that clear what the problem was. Were prices not "real" or correct? Stuff like this will happen more and more. We treat software driven systems rather recklessly.

Matt Levine at Bloomberg gives a good explanation: https://www.bloomberg.com/opinion/articles/2023-01-25/nyse-f... Basically at the market open all the requests to buy and sell get matched at the same "open" auction price, then (a second later) the orders get sent to the order book, where the price can go up and down based on size. Because the system didn't think there was an opening, there wasn't the opening auction…

https://archive.is/LovhH
Post reply on HN