Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

bloomberg.com

Post reply on HN
Print view View original

211–220 of 233 posts

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#211

Post by smcin »

bigpeopleareold wrote: post #70
mellosouls wrote: post #22

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

I had a project that was really important once. I made a tiny mistake that had a big consequence - a couple of hours of potential lost revenues from our customers. I fixed my mistake with both my boss and CEO nearby. I said after I pushed the fix that I really need more resources around it. That little light of "yeah, this is important" that should have flickered didn't. :) I will not be surprised if nothing gets fix…

But how much actual lost customer revenue? Also, did the customer even notice or not?

You're reminding me of the difference between engineers and non-technical managers; to many of the latter something's only a problem if/when the customer or senior mgmt are on the phone complaining about it. Until then it's all naysaying engineers being too pessimistic about process and risk.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#212

Post by LarryMullins »

sandworm101 wrote: post #172
mynameisvlad wrote: post #167

Earlier quoted context omitted.

How does them being a securities exchange in any way affect the analysis of their software engineering practices? They're not some special snowflake, they can suffer the same software engineering and business process issues as other companies.

>> They're not some special snowflake But they are. The consequence of a one-day or one-hour shutdown on their system is exponentially worse than most any other. I would expect them to have more rigorous systems, including more rigorous attention to development. Comparing the NYSE to any other business is like calling Fort Knox just like any other bank vault.

> like calling Fort Knox just like any other bank vault.

Main difference being that most bank vaults aren't actually empty. ;)

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#213

Post by yibg »

dylan604 wrote: post #154
godshatter wrote: post #150

Earlier quoted context omitted.

Wouldn't working with systems that keep the largest stock exchange for the largest economy in the world running where a simple mistake can cause "mayhem" when the market opens be considered more "tangible" than working in AI or on self-driving cars? It just doesn't have as much street cred as working on those particular projects in the tech community.

Not necessarily. If you're the type that's into finance, then sure, that might get you out of bed in the morning. I'm not into finance and kind stand the culture that surround finance. Yes, it's big and touches every single one of us, but doesn't mean I want to embrace it and go to work in it every day. If I can take that same skill set and apply it to something with a much better culture surrounding it that affects…

Will the world stop turning if people stopped working on self driving cars or AI?

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#214

Post by WJW »

UncleEntity wrote: post #175
wongarsu wrote: post #57

Earlier quoted context omitted.

It feels like a wartime-vs-peacetime priority problem. In wartime you would care about efficiency, build the largest number of subs staffed with the minimum crew. And since training crew quickly becomes the bottleneck you would probably go for the highest degree of automation that doesn't impact production times too much. In peacetime, efficiency isn't as important. What is important is the bad PR of losing one of yo…

Losing a ship (boat?) during peacetime is a PR disaster. Losing one during wartime can cost you the war. An inefficient one that probably won’t sink due to combat damage can stay in the fight long enough to matter.

Submarines are referred to as boats rather than ships due to naval tradition.

As a former naval officer I can only say that technically losing any vessel could be the one that loses a war, just as any soldier lost could be the straw that breaks the camel's back. But if your navy is so rickety that the loss of a single vessel is enough to lose then the main deficiency was in planning rather than any specific warship loss. Losing one in peacetime should never happen but is not unheard of even in modern times. See eg the Kursk or the Fitzgerald.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#215

Post by dragonwriter »

TeMPOraL wrote: post #33
WJW wrote: post #29

Earlier quoted context omitted.

Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who…

This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general).

> This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general

The canon explanation is that automation-in-charge was experimented with and went really badly, though periodically they try something approaching it again.

https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(...

(AI, human genetic engineering, and a number of other areas of technology are affected by variants of this issue in the Trek canon.)

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#216

Post by icedchai »

DrBazza wrote: post #125
zaroth wrote: post #107

Earlier quoted context omitted.

But it’s absurd. Companies are not supposed to only ever hire . Some things are cyclical and you need more people for some amount of time, and then you find you need less. It’s not always predictable/seasonal like farming or holiday rush. Is it wrong for a company to respond to market effects? That there was a layoff isn’t necessarily a sign a company did anything wrong… I think how they actually do the layoff certai…

It’s not hiring though. It’s overhiring. I’ve forgotten which FAANG it is. But one of them still has more employees than last year even after layoffs. It’s offensive.

Why is it offensive? Over-hiring has been a thing since at least the first dot-com boom. One's managerial power is directly proportional to how many "reports" they have under them. I worked at one company that raised a decent A round. We immediately rented another office down the street, spent close to 2 million on renovations, then filled it with anyone who could spell HTML. The B round was even larger, so the cycle continued (until late 2001 or so.)

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#217

Post by DontchaKnowit »

nubb wrote: post #177
bob1029 wrote: post #159

Earlier quoted context omitted.

I've thought about getting into this... The stuff they work on is so incredible to me. Here's a quote from their Pillar product page: > Up to a 95% Reduction in Latency: The roundtrip latency on NYSE Pillar order entry sessions via Pillar matching engines has been reduced from ~592μs to ~32μs for FIX and from ~96μs to ~26μs for Binary, getting client orders into the market much faster. With a 92% improvement in the 9…

wonder how they measure this and is this smart engineering from the exchange or just new fast network gear.

Just an educated guess (I'm in the same industry, have worked on some networking related stuff) But I think it is probably mostly network hardware and architecture. You can only improve so much from the code, the networking is where all the latency comes from.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#218

Post by DontchaKnowit »

Razengan wrote: post #170
bob1029 wrote: post #159

Earlier quoted context omitted.

I've thought about getting into this... The stuff they work on is so incredible to me. Here's a quote from their Pillar product page: > Up to a 95% Reduction in Latency: The roundtrip latency on NYSE Pillar order entry sessions via Pillar matching engines has been reduced from ~592μs to ~32μs for FIX and from ~96μs to ~26μs for Binary, getting client orders into the market much faster. With a 92% improvement in the 9…

> Reading stuff like this makes my current work feel stupid by comparison. It makes our economic system seem stupid. Jesus, we're not calculating astrophysics or quantum mechanics. A made-up system should not require or depend upon this kind of speed or precision. Maybe we should chill. Reminds me of those pro StarCraft players who keep unnecessarily clicking the mouse to keep their APM (actions per minute) stat high…

Absolutely agree. but "liquid markets are important" or something. rolls eyes

This is just another step in the endless journey of widening the gap between your average Joe and someone with access to high level financial services.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#219

Post by jacquesm »

bostonsre wrote: post #202
jacquesm wrote: post #129

Earlier quoted context omitted.

Just naming a 'staffer' though seems to already be a way to apportion blame to a segment of the employees, insulating management from what was done. Named or not doesn't really matter, clearly blame is being assigned.

Yea, it sounds like an issue with process and automation. It shouldn't have been possible for the staffer to make a mistake that would cause this.

Precisely. It is never just one error. At a minimum two and if you really stare at this sort of thing long enough it isn't rare at all to discover a whole chain of them. The only difference with all the times that it went right is that this time everything was aligned 'just so'.

Re: NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

#220

Post by DontchaKnowit »

barbishkoolaid wrote: post #168
idiotsecant wrote: post #163

Earlier quoted context omitted.

>if the NYSE did not exist, the world would continue to turn This is startlingly ignorant of the complex machine that is the modern economic system. If something like the NYSE was to shut down today it would be pandemonium. There is a difference between 'I don't understand how something works' and 'I don't understand how something works, so it is worthless'. The former is healthy and the first step to understanding,…

Relax, Ayn Rand cum True Believer complex. The current state of business within the current iteration of how people interact with one another isn’t some necessity. Yes, the world may fall apart for a relatively brief moment in the grand scheme of things — but then life will go on. The first step to understanding this is to drop the superiority complex. Very little is actually needed to keep the world turnin.

I actually think both you and the guy you are arguing with are half correct.

The real answer here, in my opinion, is that yes there would be pandemonium, and then yes, the world would go on without it, but then something else just like it will pop up. And that is because a liquid market for financial assets (whether that is securities, options on securities, futures, etc) will always be a massive benefit to the ability of businesses to conduct business, and the ability of individuals to preserve and increase wealth.

Post reply on HN