71–80 of 233 posts
> Meanwhile, market professionals and day traders are rattled and waiting for the exchange to elaborate on what it publicly called a “manual error” involving its “disaster recovery configuration”.
Oh, I love it -- a disaster caused by "disaster recovery configuration" :-)
Earlier quoted context omitted.
This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.
The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.
Earlier quoted context omitted.
That wasn't an exchange "malfunction" in the sense that the exchange did not do what it was supposed to, was it?
Do you really think they will take responsibility for billions lost/made that day? No, that's a big liability. Anyone trading that day knew it was a big glitch at open. Some names, BLUE CHIPS, were down 40-50%! What shocked us all is that they actually allowed the trades to stand. What was different about this week was that the SEC actually tried to do their jobs for once and the exchange had to address it ie come up…
These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.
At some point you need to strike a balance between freedom/flexibility and stupid proofing.
HN goes real hard on the "people are idiots and we should design things that no matter what buttons get mashed it all works out fine" side of things but in the financial world the balance is struck a little further on the "train our employees to not be idiots" side of things.
Furthermore, it's usually better optics to blame things on people because people can easily and cheaply alter their behavior cheaply (per incremental change). If you blame the outage on systems it raises questions of when it will be fixed and how much $$.
As an aside, it was almost certainly not individual error. At places like NYSE you pretty much always have 2-3 people who should be in a position to catch a mistake like this.
Earlier quoted context omitted.
>More automation -> more code -> more things to go wrong More people -> more entropy -> much more things to go wrong
One person might be inattentive or drunk, but it's less likely that two people are. So you institue a two-person rule. And if that's not good enough, add a third person to double-check. Maybe a supervisor to observe the people doing all of the above, to catch any mistakes or negligent behavior. Also have them write down the steps they have taken, and have somebody else read through that to verify. Just keep adding pe…
Earlier quoted context omitted.
This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.
If you have processes where there is nothing an employee can do to affect the outcome of the company you successfully built a legacy bureaucracy that is waiting to be disrupted.
Of course, once you take advantage of such a representation at scale by deploying tremendously more complex infrastructures, you then have to deal with the dependency network meta challenge lest you inadvertently fall into dependency hell. While towards there lies NP-hard problems, they're still computable to a reasonable degree and I dare say a more robust situation than doing it all by hand like we do today.
The real challenge is the vast majority of devops staff today would really dislike reasoning about such a representation when it blows up in their faces, and I can't blame them for that kind of reaction.
These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.
> These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. At some point you need to strike a balance between freedom/flexibility and stupid proofing. HN goes real hard on the "people are idiots and we should design things that no matter what buttons get mashed it all works out fine" side of things but in the financial world the balance is struck a little furthe…
That's exactly the point that is being made here. Either the message being put out by the NYSE claiming this was an error by one individual is true -- in which case, NYSE leadership is to blame for setting up a process that allows catastrophic consequences for a single individual's error, OR the message being put out by the NYSE is a fabrication designed to redirect blame at some scapegoat, in which case NYSE leadership is to blame for putting out a false or misleading statement.
[Edit: It seems I misunderstood -- attributing this to an individual was done by reporters and rumors, not by a formal statement from NYSE.]
Earlier quoted context omitted.
This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.
The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.
Earlier quoted context omitted.
The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.
There's plenty of companies replacing their CEOs. Just today Toyota announced theirs.
Earlier quoted context omitted.
More automation -> more code -> more things to go wrong
This employee left the backup system running. There's obviously some automation but what is the solution? Process changes that people have to remember or more systems to prevent the issue. So I don't get your statement related to this article