Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

Post reply on HN
Print view View original

81–90 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#81

Post by JackFr »

WJW wrote: post #29
bongobingo1 wrote: post #8

Earlier quoted context omitted.

The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.

Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who…

Working on the repo desk of a large Japanese bank in New York in the 90’s. There was a big (both in font size and magnitude) number that was on the upper left of the blotter system that ran on every traders desk, which represented the total we had to borrow that day to fund the banks trading book. There would be a number below it which would represent how much we had borrowed so far.

It was “too important to automate” so a trading assistant keyed it in every morning. One morning he typed the wrong number and the mistake was in the billions digit.

At 2:45 “the cage” called the repo desk and said “You know you guys are still short a billion, right?”

There was then a flurry of activity as traders got on the phone to try to borrow a billion dollars in in fifteen minutes, while also trying to not let on we were kind of over a barrel. The head of fixed income prepared his explanation to the Fed about why we needed to borrow a few hundred million overnight.

The number got automated in our next release, and the open procedure was changed to the trading assistant verifying the number against the “cage” report.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#82

Post by logifail »

thisarticle wrote: post #32
wrldos wrote: post #23

Earlier quoted context omitted.

This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.

The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.

> The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring

The list of managers stating that "they were taking responsibility" and then immediately stepping down was always fairly short.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#83

Post by credit_guy »

mellosouls wrote: post #22

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

It does not sound like cover up to me.

It was simply the explanation of what happened. I didn't get any hint that the said "staffer" will be fired or otherwise punished.

Is there a problem with the system that did not have enough safeguards to let this happen. For sure, but then no system is perfect. This glitch does not happen every day. From memory, I remember a NASDAQ glitch at Facebook's IPO. Let's say there are 2 or 3 glitches like that for major exchanges in one decade. How can you design a system that prevents bugs that show up once a decade?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#84

Post by gjvc »

lr1970 wrote: post #71

From the article: > Meanwhile, market professionals and day traders are rattled and waiting for the exchange to elaborate on what it publicly called a “manual error” involving its “disaster recovery configuration”. Oh, I love it -- a disaster caused by "disaster recovery configuration" :-)

Oh, I love it -- a disaster caused by "disaster recovery configuration" :-)

People install failover configurations to minimise time-to-repair or time-to-resume service (and some customers' contracts will demand this). This is at the expense of another layer of stuff to go wrong, and raising the possibility that it fails over when it shouldn't, causing brief but embarrassing outages.

It's possible in some such situations that, on the balance of probabilities, introducing mechanisms like this cause more disruption over time than they were intended to protect against, and that this is more widespread than often considered. Still, their operational cost must be borne in order to satisfy the clause in the customers' contracts.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#86

Post by xwolfi »

benjaminwootton wrote: post #16
qrybam wrote: post #7

Earlier quoted context omitted.

The auction is meant to find a stable price and can have some wild prices coming in, because no matching happens at the point of entry, the market will naturally find a level before trading commences. In this case, those wild prices were matching, resulting in crazy trades no-one would have expected. You'd be surprised how many manual processes there are in places like this. It's a combination of legacy systems / pro…

When I worked on a trading platform, I spent many a happy Sunday night waiting for the Australian market to open and watch the first orders go through successfully. We had hundreds of jobs and upgrades happening over each weekend. It definetly needed an eye casting over it regardless of the automation.

I am working at one now and nothing has changed. Automations are so many we recruit an army of people just checking if they ran, while knowing how to replace them if they didnt (or, more accurately, who to call at night to fix it asap).

And the AU orders going through is a good sign, but it's far from guaranteeing a free monday, as Japan, Korea or Shanghai can fuck it up, each in their own little ways. Hong Kong is the best, low regulatory crap, invested regulator, high volume low latency traffic everyday (relative to the region), I cant recall a time it broke.

Once, someone fat fingered an excel import at close, and we lost our trading license for that entire country for 18 months. And we're not small. But the amount mismatched at settlement was super tiny. High attack surface, low holistic understanding (it works despite us, we honestly have no clue sometimes), heavy consequences on screwup.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#87

Post by jrochkind1 »

> That misled the exchange’s computers to treat the 9:30 a.m. opening bell as a continuation of trading, and so they skipped the day’s opening auctions that neatly set initial prices.

I didn't even know about this process. I don't know much about trading, but it surprises me that there is a separate process for setting prices at the start of trading, and that if it's missed, chaotic prices result.

Is this related to how stock markets aren't really ever open 24 hours? Do they need that reset to function in stable way?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#88

Post by papito »

I sweat over "idiot-proofing" the smallest systems, while multi-billion dollar operations don't seem to care enough.

Like the S3 being blown away with a simple change in the early days, or GitHub running a test suite with production settings. It's like the FIRST thing I think about when starting a project.

https://github.blog/2010-11-15-today-s-outage/

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#89

Post by papito »

WJW wrote: post #29
bongobingo1 wrote: post #8

Earlier quoted context omitted.

The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.

Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who…

I will argue that the corporate world is the exact opposite of the training and discipline of a submarine crew. Most of the time I wonder how the businesses even survive the chaos and mismanagement, let alone make money.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#90

Post by itsoktocry »

thisarticle wrote: post #32
wrldos wrote: post #23

Earlier quoted context omitted.

This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.

The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.

>See: not a single CEO stepping down for over hiring.

Wait, what? You think a CEO should step down because their management over-hired a relatively small proportion of employees and had to do some layoffs?

Post reply on HN