Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

11–20 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#11
A lot of stocks have minute bar of wide range prices in the first few minutes of continuous trading. This seems like the incident in what caused Knight Capital fiasco, in which the system repeatedly buy on the ask and sell on the bid very fast, thus pushing the high price high and low price low. In the opening usually the market maker will be more weary of the risk they cannot hedge directly and thus will be less willing to take on positions, leading to wild swings.

Still this report (and the previous statement) does not give enough detail on why a backup system misoperation resulted this. Also, critical large systems like exchange rarely have a single point failure. Usually there will be a sequence of issues along the event chains leading to this. Thus one "failed to properly shutdown" caused all this is a bit incredible. We will need more explanation.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#15
post #10

Earlier quoted context omitted.

The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.

I've not seen it put so eloquently.

To err is human, to really foul things up requires a computer

- William E. Vaughan https://quoteinvestigator.com/2010/12/07/foul-computer/

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#16
post #7
post #3

Seems so weird to not have automated checks for something that seems to be described as "someone left the light on" and also not have the exchange automatically initiate itself. However, it still isn't that clear what the problem was. Were prices not "real" or correct? Stuff like this will happen more and more. We treat software driven systems rather recklessly.

The auction is meant to find a stable price and can have some wild prices coming in, because no matching happens at the point of entry, the market will naturally find a level before trading commences. In this case, those wild prices were matching, resulting in crazy trades no-one would have expected. You'd be surprised how many manual processes there are in places like this. It's a combination of legacy systems / pro…

When I worked on a trading platform, I spent many a happy Sunday night waiting for the Australian market to open and watch the first orders go through successfully.

We had hundreds of jobs and upgrades happening over each weekend. It definetly needed an eye casting over it regardless of the automation.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#18

Backup system connected to prod, that somehow reminds of the Knight Trading debacle. Someone there apparently connected some test code to their prod and they blew up the company in under an hour.

I mean - isnt that how all DR is essentially configured? You need it "somehow" connected to Prod depending on the failover, config, system etc. And in many of these complex systems DR can be on a subsystem etc - not an "all or nothing" approach?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#20
post #15
post #10

Earlier quoted context omitted.

I've not seen it put so eloquently.

To err is human, to really foul things up requires a computer - William E. Vaughan https://quoteinvestigator.com/2010/12/07/foul-computer/

“A computer lets you make more mistakes faster than any other invention, with the possible exceptions of handguns and tequila.”—Mitch Ratcliffe
Post reply on HN