181–190 of 233 posts
> Question: Can I connect to both the production and the DR site at the same time?
Answer: No, only one site is available at a time. When the primary site is up, the DR site is down; and when the DR site is activated, the primary site is down.
I think they need to update these docs to say /should/ be down
Seems so weird to not have automated checks for something that seems to be described as "someone left the light on" and also not have the exchange automatically initiate itself. However, it still isn't that clear what the problem was. Were prices not "real" or correct? Stuff like this will happen more and more. We treat software driven systems rather recklessly.
You'd have to have more than 1 person involved to forget that DR is still active when completing these failover exercises and tests off-hours.
Tuesday news blackout; Thursday "it was all Jim's fault"... Right. Smells like horse shit. Most of what comes out of the profession of "journalism" does too, lately; but this smells strongly .
and all the trades are being reversed basically. The whole thing stinks.
There is so much stupidity in the process they describe, I have no faith it will be fixed. a manual daily DR test that clearly wasn't followed by a test or checklist or double checked by another person, and leaving the DR up broke prod?? literally none of those things should have happened. I know the world is held together with duct tape, but it's embarrassing when you see the tape fall off.
The NYSE DR guide [1] says that if DR is active, production is not. It's not a distant reach to consider that some of these clients have a deadman switch doing a healthcheck poll on DR and switching to it when it see's that it is "up". If they've built their systems in such a way that when it detects the DR site active it uses that, then it makes sense that having both "online" would cause some havoc. I'm sure the complexity of the entire exchange is fairly significant, and having "two" copies of it running in parallel with both able to accept and execute trades would be a scenario that can cause some unintended consequences. Fundamentally, an exchange is "atomic" and transactional and cannot be meaningfully distributed to two sites that are that far away. The replication in place is likely master/slave with a switch to make the slave primary. Anyone who has toyed with master-master replication on less complicated databases knows the issues that can come up with split writes. Imagine that at the scale of a system as large as the NYSE.
[1] https://www.nyse.com/publicdocs/support/DisasterRecoveryFAQs...
Earlier quoted context omitted.
Wouldn't working with systems that keep the largest stock exchange for the largest economy in the world running where a simple mistake can cause "mayhem" when the market opens be considered more "tangible" than working in AI or on self-driving cars? It just doesn't have as much street cred as working on those particular projects in the tech community.
Not necessarily. If you're the type that's into finance, then sure, that might get you out of bed in the morning. I'm not into finance and kind stand the culture that surround finance. Yes, it's big and touches every single one of us, but doesn't mean I want to embrace it and go to work in it every day. If I can take that same skill set and apply it to something with a much better culture surrounding it that affects…
Earlier quoted context omitted.
Wouldn't working with systems that keep the largest stock exchange for the largest economy in the world running where a simple mistake can cause "mayhem" when the market opens be considered more "tangible" than working in AI or on self-driving cars? It just doesn't have as much street cred as working on those particular projects in the tech community.
Not necessarily. If you're the type that's into finance, then sure, that might get you out of bed in the morning. I'm not into finance and kind stand the culture that surround finance. Yes, it's big and touches every single one of us, but doesn't mean I want to embrace it and go to work in it every day. If I can take that same skill set and apply it to something with a much better culture surrounding it that affects…
The movie portrayals don’t match my experiences at all and I saw a lot more bad behavior in the tech companies I worked for.
Heck I saw more people working for the intellectual challenge of it in trading than I did in SV style tech firms where money drove nearly every decision.
It’s really hard for me to buy that SV style tech companies are a better place to work when for the last 2 decades the business models that have been front and center are panopticon style tracking to sell ads and legal arbitrage.
These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.