Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

171–180 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#171

Earlier quoted context omitted.

>if the NYSE did not exist, the world would continue to turn This is startlingly ignorant of the complex machine that is the modern economic system. If something like the NYSE was to shut down today it would be pandemonium. There is a difference between 'I don't understand how something works' and 'I don't understand how something works, so it is worthless'. The former is healthy and the first step to understanding,…

Relax, Ayn Rand cum True Believer complex. The current state of business within the current iteration of how people interact with one another isn’t some necessity. Yes, the world may fall apart for a relatively brief moment in the grand scheme of things — but then life will go on. The first step to understanding this is to drop the superiority complex. Very little is actually needed to keep the world turnin.

Exactly. Our current implementations of resource rationing isn't some fundamental of reality, or even needed by human society just 1-2 centuries ago.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#172
post #166

Earlier quoted context omitted.

I would think that the company being a securities exchange would factor into the analysis. Don't you?

How does them being a securities exchange in any way affect the analysis of their software engineering practices? They're not some special snowflake, they can suffer the same software engineering and business process issues as other companies.

>> They're not some special snowflake

But they are. The consequence of a one-day or one-hour shutdown on their system is exponentially worse than most any other. I would expect them to have more rigorous systems, including more rigorous attention to development. Comparing the NYSE to any other business is like calling Fort Knox just like any other bank vault.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#173
post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

I've thought about getting into this... The stuff they work on is so incredible to me. Here's a quote from their Pillar product page: > Up to a 95% Reduction in Latency: The roundtrip latency on NYSE Pillar order entry sessions via Pillar matching engines has been reduced from ~592μs to ~32μs for FIX and from ~96μs to ~26μs for Binary, getting client orders into the market much faster. With a 92% improvement in the 9…

[dead]

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#174

Tuesday news blackout; Thursday "it was all Jim's fault"... Right. Smells like horse shit. Most of what comes out of the profession of "journalism" does too, lately; but this smells strongly .

and all the trades are being reversed basically. The whole thing stinks.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#175

Earlier quoted context omitted.

> It's clearly not optimized for efficient use of personnel I disagree. When the sub sinks, all these people die. That's far more inefficient use of personnel. It feelf like you're putting more emphasis on the material cost ("the cost ... can be very large indeed") than on what actually matters.

It feels like a wartime-vs-peacetime priority problem. In wartime you would care about efficiency, build the largest number of subs staffed with the minimum crew. And since training crew quickly becomes the bottleneck you would probably go for the highest degree of automation that doesn't impact production times too much. In peacetime, efficiency isn't as important. What is important is the bad PR of losing one of yo…

Losing a ship (boat?) during peacetime is a PR disaster.

Losing one during wartime can cost you the war.

An inefficient one that probably won’t sink due to combat damage can stay in the fight long enough to matter.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#176
post #126

Earlier quoted context omitted.

I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.

Blame cultures and process cultures are both problems in different ways. Blame cultures don't care about individual accountability, only that someone suffers. Process cultures only care that no one suffers, not that individuals are accountable. Both have some misguided notion that something other than personal accountability can lead to good results. Misattributed blame and suffering does not deter poor performance o…

[dead]

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#177
post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

I've thought about getting into this... The stuff they work on is so incredible to me. Here's a quote from their Pillar product page: > Up to a 95% Reduction in Latency: The roundtrip latency on NYSE Pillar order entry sessions via Pillar matching engines has been reduced from ~592μs to ~32μs for FIX and from ~96μs to ~26μs for Binary, getting client orders into the market much faster. With a 92% improvement in the 9…

wonder how they measure this and is this smart engineering from the exchange or just new fast network gear.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#178

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

It's also true that in systems like this there exist many single points of failure. There's a reason decentralized systems are seeing a rebirth.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#179
post #106

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or…

do they still do UDP packet loss replay over email?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#180
post #75

Earlier quoted context omitted.

So two people are more reliable as a automated system you wanna say? That's totally wrong....

Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. I did last year, and my company is in the process of de-automating certain processes that can endanger that company if they go wrong. There are many things in tech that are too important to automate. I'd even posit that the more experience you have in tech, the more you've seen how things go wrong, and the more yo…

>Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right.

Much much much more due to human error......but hey maybe you are the worst programmer ever..but even then i would say your programs are more reliable then a human.

Post reply on HN