251–260 of 294 posts
In this context it's more useful to think in terms of production principles. The principle that was poorly followed was defence in depth. There was no line of defence after the deployment.
Earlier quoted context omitted.
The goal with automation is that the number of unidentified corner cases reduces over time. A manual runbook is a game of, "I did step 12, I think I did step 13, so the next step is 14." that plays out every single time you do server work. The thing with the human brain is that when you interrupt a task you've done a million times in the middle, most people can't reliably discern between this iteration and false memo…
1. Print the checklist/runbook out on paper with actual empty boxes next to the steps. 2. Laminate the printed checklist and put it in a big folder. 3. Every time you run the checklist, use a sharpie to mark the checkbox after you've done the step. 4. When you are done with the entire process, use whiteboard cleaner to wipe out the checks again and put the checklist back in the big folder with all the other checklist…
Earlier quoted context omitted.
It would be a serious issue for in person transactions like shops, supermarkets, gas stations, etc Imagine Walmart or Costco or Chevron centralised payment services went down for 30+ mins. You would get a lot of lost sales from those who don’t carry enough cash to cover it otherwise. Maybe a retailer might have a zapzap machine but lots of cards aren’t imprinted these days so that’s a non starter too.
Not just lost sales. I've seen a Walmart lose all ability to do credit card sales and after about 5 minutes maybe 10% of people waiting just started leaving with their groceries in their cart and a middle finger raised to the security telling them to stop.
This incident highlights a problem that is often overlooked in the debate about feature branches versus feature toggles. I've worked with both feature branches and feature toggles, and while long lived feature branches can be painful to work with what with all the conflicts, they do have the advantage that problems tend to be uncovered and resolved in development before they hit production. When feature toggles go wr…
Earlier quoted context omitted.
How are they taking on loads of risk? Risk has a particular meaning in investment and "well, a bug can blow up my company" isn't part of that meaning. Simply creating risky (in the colloquial meaning) things is not itself a reason to deserve money.
When you make markets you are literally paid the spread to assume the risk of holding the position.
Earlier quoted context omitted.
> Actually, we made way less of Knight's $400m than we could have because our risk systems kept shutting strategies down because what was happening was "too good to be true". Aren't a lot of trades undone anyway by the authorities after such severe market hiccups?
Normally trades are undone or amended in price if they are executed far away (say 10%+) from what is determined to be reasonable market prices. And when amended, they get amended to a price that's still in the same direction, so the market taker still loses a little bit compared to the fair price. KCG traded in such liquid instruments and in such a way that it didn't move the market that much. They lost a hundred dol…
Technically KCG didn't exist yet. NITE, aka Knight Capital Group lost ~$460 million. Getco bought NITE forming KCG (Beating out Virtu's bid which later purchased KCG) and now is called Virtu Financial (VIRT).
Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.
I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.
It's a set of failures in a system that all had to happen for the failure to occur.
I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…
> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill sw…
Isn't the problem that pulling the plug on a trading bot doesn't just have opportunity costs, but may also leave you with open positions that, depending on the kind of trades you're doing and the way the market is moving, could be arbitrarily expensive to unwind?
Earlier quoted context omitted.
1. Print the checklist/runbook out on paper with actual empty boxes next to the steps. 2. Laminate the printed checklist and put it in a big folder. 3. Every time you run the checklist, use a sharpie to mark the checkbox after you've done the step. 4. When you are done with the entire process, use whiteboard cleaner to wipe out the checks again and put the checklist back in the big folder with all the other checklist…
Regarding checklist people like Hollnagel, Wears, Braithwaite, Dekker, ... have done a bit on investigation (Hollnagel mostly on Healthcare, Dekker started on air industry but spread from there). Read the "Safety-I vs Safety-II" paper or the "When a checklist is not enough: How to improve them and what else is needed" paper
I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…
Which really means is a failure of leadership for being so incompetent as to allow such an intensely risky situation to exist.
Imagine a company where the engineering culture hires macho programmers who love bitmasked flags and manual memory management, and who think memory safe languages and json are for sissies.
Imagine they hire lots of graduates who've never worked elsewhere and teach them that their way is best, and everyone who says otherwise doesn't understand real performance. Those 'industry best practices' are written by javascript folks who think a 2 second pageload is fast, and a 200ms pageload is instant, don't you know?
And imagine when they make experienced hires, they look for people who have experience with bitmasked flags and manual memory management - which is reasonable enough, they gotta be able to code review that stuff and coach junior employees on working with it. But has the side effect experienced hires won't rock the boat.
Now, the nontechnical bosses can ask anyone from the highest engineering leadership to the must junior of peons, and they'll all agree that bitmasked flags are the right way of doing things.
Is it so unreasonable for nontechnical bosses to trust the consensus of their engineers on matters of engineering?