Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

Post reply on HN
Print view View original

191–200 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#191

Post by dylan604 »

kasey_junk wrote: post #187
dylan604 wrote: post #154

Earlier quoted context omitted.

Not necessarily. If you're the type that's into finance, then sure, that might get you out of bed in the morning. I'm not into finance and kind stand the culture that surround finance. Yes, it's big and touches every single one of us, but doesn't mean I want to embrace it and go to work in it every day. If I can take that same skill set and apply it to something with a much better culture surrounding it that affects…

All of my “culture” experience working in finance were uniformly better than pure tech. The movie portrayals don’t match my experiences at all and I saw a lot more bad behavior in the tech companies I worked for. Heck I saw more people working for the intellectual challenge of it in trading than I did in SV style tech firms where money drove nearly every decision. It’s really hard for me to buy that SV style tech com…

Oh, don't get me wrong. I pretty much abhor SV/VC culture too. It's why I don't have one inkling of a notion to work on either coast for the "big" corps.

It's not an either or, I can hate both ;-) I'm a big boy and get to make up my own mind on the matter.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#192

Post by hadlock »

hackernewds wrote: post #169
ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

Curious why the link to ICE.COM?

ICE owns NYSE and several other exchanges. ICE stands for Intercontinental Exchange. ICE is the IT administrator for these exchanges. I helped sell some router management software to them a while ago. ICE is fairly new, NYSE used to be independent. That changed sometime after 2008.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#193

Post by shanebellone »

sandworm101 wrote: post #172
mynameisvlad wrote: post #167

Earlier quoted context omitted.

How does them being a securities exchange in any way affect the analysis of their software engineering practices? They're not some special snowflake, they can suffer the same software engineering and business process issues as other companies.

>> They're not some special snowflake But they are. The consequence of a one-day or one-hour shutdown on their system is exponentially worse than most any other. I would expect them to have more rigorous systems, including more rigorous attention to development. Comparing the NYSE to any other business is like calling Fort Knox just like any other bank vault.

I disagreed with you until: "...like calling Fort Knox just like any other bank vault."

Interesting point that teeters on false equivalence. I think AWS or Azure might make for a better analogy. Your point identifies the inherent risk of actually operating a platform business. A bank vault is (mostly) synonymous with Cloud, in this context. If a vault is robbed or a cloud goes offline, losses extend beyond the business which inherently compounds the severity of downtime.

Linear loss vs. parabolic loss.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#194

Post by blantonl »

sandworm101 wrote: post #172
mynameisvlad wrote: post #167

Earlier quoted context omitted.

How does them being a securities exchange in any way affect the analysis of their software engineering practices? They're not some special snowflake, they can suffer the same software engineering and business process issues as other companies.

>> They're not some special snowflake But they are. The consequence of a one-day or one-hour shutdown on their system is exponentially worse than most any other. I would expect them to have more rigorous systems, including more rigorous attention to development. Comparing the NYSE to any other business is like calling Fort Knox just like any other bank vault.

No they aren't.

There's far more critical snowflakes out there... FAA Airspace management, a medical radiation device, avionics in an aircraft, and facebook.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#195

Post by Kon-Peki »

tgtweak wrote: post #182
bmitc wrote: post #3

Seems so weird to not have automated checks for something that seems to be described as "someone left the light on" and also not have the exchange automatically initiate itself. However, it still isn't that clear what the problem was. Were prices not "real" or correct? Stuff like this will happen more and more. We treat software driven systems rather recklessly.

I don't think it's that they "left it running" like you would leave a backup app running... they literally left the entire disaster-recovery site up and running and live. Cermak (referred to in the article as the "backup") is an entire datacenter, hosting a running copy of the exchange to be used in a failover scenario. You'd have to have more than 1 person involved to forget that DR is still active when completing t…

> Cermak (referred to in the article as the "backup") is an entire datacenter

Well, they are a tenent at the Cermak data center. It’s a truly massive building with huge amounts of connectivity and colo opportunities. Probably also the only 100+ year old data center building on the US register of historic places, lol (it’s a former catalog printing facility, built to hold insanely heavy printing presses on 8 or 9 really tall floors, so it has no problems with densely-packed server racks)

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#196

Post by Johnny555 »

ngz00 wrote: post #126
ynniv wrote: post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.

I used to work for a small startup, and postmortems were truly no blame - engineers would talk about exactly what happened and wouldn't hesitate to put the blame on their mistakes.

But as the company grew, the postmortems became more about blame since now you're not blaming an engineer, but an entire team so singling them out isn't personal. The postmortems were no longer a single engineer describing what happened in his code, but were team leads talking on behalf of teams. They were all about shifting blame from your own team and talking about why a service from another team led to the problem, even if your team could have (and should have) been able to work around it without melting down.

I'm no longer at the company, but Postmortems are much more useful when they really are no-blame because you can get to the real root of the problem, but I don't know if that's possible in a large company.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#197

Post by btown »

shanebellone wrote: post #193
sandworm101 wrote: post #172

Earlier quoted context omitted.

>> They're not some special snowflake But they are. The consequence of a one-day or one-hour shutdown on their system is exponentially worse than most any other. I would expect them to have more rigorous systems, including more rigorous attention to development. Comparing the NYSE to any other business is like calling Fort Knox just like any other bank vault.

I disagreed with you until: "...like calling Fort Knox just like any other bank vault." Interesting point that teeters on false equivalence. I think AWS or Azure might make for a better analogy. Your point identifies the inherent risk of actually operating a platform business. A bank vault is (mostly) synonymous with Cloud, in this context. If a vault is robbed or a cloud goes offline, losses extend beyond the busine…

But if a cloud goes offline, there is damage to the economy linear to the length and breadth of the outage. Sure, there are losses to businesses serviced by the cloud's users, but they'll bounce back, even if a day-long outage was so severe as to temporarily ground flights and halt supply chains.

If a stock exchange executes trades at incorrect prices, even for a short amount of time, all of a sudden you're in a kind of non-linear sigmoid regime, where investor confidence can suddenly tip into panic selling and recessions can be triggered. Thankfully, that didn't happen here, but it could have. If you're going to give a company that power, you should better hope that they're held to higher standards than most dysfunctional tech organizations!

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#198

Post by shanebellone »

btown wrote: post #197
shanebellone wrote: post #193

Earlier quoted context omitted.

I disagreed with you until: "...like calling Fort Knox just like any other bank vault." Interesting point that teeters on false equivalence. I think AWS or Azure might make for a better analogy. Your point identifies the inherent risk of actually operating a platform business. A bank vault is (mostly) synonymous with Cloud, in this context. If a vault is robbed or a cloud goes offline, losses extend beyond the busine…

But if a cloud goes offline, there is damage to the economy linear to the length and breadth of the outage. Sure, there are losses to businesses serviced by the cloud's users, but they'll bounce back, even if a day-long outage was so severe as to temporarily ground flights and halt supply chains. If a stock exchange executes trades at incorrect prices, even for a short amount of time, all of a sudden you're in a kind…

"If a stock exchange executes trades at incorrect prices, even for a short amount of time, all of a sudden you're in a kind of non-linear sigmoid regime, where investor confidence can suddenly tip into panic selling and recessions can be triggered."

This is false equivalence and slippery slope.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#199

Post by evanpw »

WiSaGaN wrote: post #62
itronitron wrote: post #37

Earlier quoted context omitted.

Doesn't each trade require both a buyer and a seller, who buy and sell at an agreed on price? Presumably both parties would be satisfied with the trade so it isn't clear to me what all the fuss is about.

I don't know what happened in current case. In the Knight Capital case, KC clearly didn't intend to send those erroneous orders. And if those trades were not annuled, KC would not be able to settle those, since the trade loss were larger than the collateral KC put up.

The trades were not annulled, because NYSE ruled them not "clearly erroneous". Which is why it is was an existential mistake, not just an embarrassing one.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#200

Post by evanpw »

herpderperator wrote: post #24

If the trades are being cancelled, are they going to correct the chart data? Right now it looks very misleading on the daily[0], weekly, monthly, quarterly, yearly etc for large caps that trade quite steadily otherwise. I do understand that this would be a challenging effort as that data already flowed to and was stored by all the broker-dealers, but I think it should be done. [0] https://www.dropbox.com/s/6jdmgkdyei…

The same feed that publishes trades also publishes trade busts, so it's up to whoever's consuming it downstream to take care of.
Post reply on HN