Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

21–30 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#22
These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me.

Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#23

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#24
If the trades are being cancelled, are they going to correct the chart data? Right now it looks very misleading on the daily[0], weekly, monthly, quarterly, yearly etc for large caps that trade quite steadily otherwise. I do understand that this would be a challenging effort as that data already flowed to and was stored by all the broker-dealers, but I think it should be done.

[0] https://www.dropbox.com/s/6jdmgkdyei9xqz0/mcd.png?dl=0

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#26
post #5

I am short a couple dozen naked Feb 3 calls on a lot of affected tickers and almost had a heart attack when looking at one of them on my phone. Thankfully I was not in front of my computer at the time because I have no idea how my broker was managing my margin at open.

The LULD breakers saved everyone from more chaos here because effectively as soon as the market opened, all these symbols were halted.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#27
post #4
post #3

Seems so weird to not have automated checks for something that seems to be described as "someone left the light on" and also not have the exchange automatically initiate itself. However, it still isn't that clear what the problem was. Were prices not "real" or correct? Stuff like this will happen more and more. We treat software driven systems rather recklessly.

More automation -> more code -> more things to go wrong

This employee left the backup system running. There's obviously some automation but what is the solution?

Process changes that people have to remember or more systems to prevent the issue. So I don't get your statement related to this article

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#29
post #4

Earlier quoted context omitted.

More automation -> more code -> more things to go wrong

The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.

Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who watches the readings through a centralized system.

It's clearly not optimized for efficient use of personnel, but the personnel complement will have been designed to provide sufficient people at all times and the cost of getting it wrong can be very large indeed.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#30
post #23

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.

Exactly, management shouldn't allow this kind of situation to occur by designing the xomoanies processes such that there are checks and balances
Post reply on HN