From the blameful-postmortem:
Q. Why did we click `Send $900M?` A. Not sure. Tom felt all clicky-clicky, so he clicked it.
Q. Why did we hire Tom? A. Also not sure. [Action Item: Fire Tom]
Citi’s $900M Misfire Happened During Software Switch
21–30 of 55 posts
Re: Citi’s $900M Misfire Happened During Software Switch
#22Somewhere in your company hiring queue is a resume with:
Strengths: Architecting risk management systems
Weaknesses: Sometimes I click on things to see what happens
Re: Citi’s $900M Misfire Happened During Software Switch
#23Earlier quoted context omitted.
Actually, banks risk management is easy. Most in-bank or between banks transfers are reversible and usually a non-issue. That why the risk management probably says something like this: Risk: Incorrect transfer of funds to customer in another bank Mitigation: Manual review of all funds transfer above 5 million dollars Mitigation: Besides litigation issues, lost funds are easily recovered by asking the receiving bank S…
Bank transfers between banks are not reversible.
Do you mean when the mortal clients of the banks are doing them, or when the banks are doing them?
I imagine bank CEOs know each other and can even call each other and say "Oh sorry old chap, that was a mistake!".
Re: Citi’s $900M Misfire Happened During Software Switch
#24Earlier quoted context omitted.
At least Citibank is trying to upgrade their ancient systems, but it sure looks as if previous or current CEOs failed to exercise due diligence. Doesn't look like good risk management at all.
Citi is one of them banks that spend a lot of their $$$$$$ in IT. They jokingly say that they are an IT company with a banking license. Anyone related can please pitch in with a TA account. How bad/frequent are their Software Errors?
I mean...if this were any other business, someone clicking the wrong button and then someone else not catching this error would maybe mean someone's account was accidentally closed, or someone receiving a free pizza, or whatever.
The IT systems within banks are more or less the same as IT systems anywhere. Just as advanced, just as crappy. The difference is that if there is a human error with banking software, you're not sending free pizza, you accidentally pay ouy $900,000,000.
Re: Citi’s $900M Misfire Happened During Software Switch
#25Earlier quoted context omitted.
At least Citibank is trying to upgrade their ancient systems, but it sure looks as if previous or current CEOs failed to exercise due diligence. Doesn't look like good risk management at all.
Actually, banks risk management is easy. Most in-bank or between banks transfers are reversible and usually a non-issue. That why the risk management probably says something like this: Risk: Incorrect transfer of funds to customer in another bank Mitigation: Manual review of all funds transfer above 5 million dollars Mitigation: Besides litigation issues, lost funds are easily recovered by asking the receiving bank S…
Nothing about bank risk management is easy. You haven’t accounted for a whole host of risks with this simple analysis, including the most important one: customer retention. The high-value customers sending high-value interbank transfers won’t be impressed that our systems let their wire go out the door incorrectly. Even if I can totally reverse the transaction (and it’s not nearly as easy or guaranteed as you’re assuming) I still have to tell the client about it in most cases. The client will, correctly, think “what if they can’t get it back next time this happens? I’ll find a bank that doesn’t have these kinds of issues.” And aside from crimes, losing a high-value client is perhaps the worst offense you can commit in banking.
Regardless of the dollar amount or outcome I also have to tell the audit committee, the board, the auditors, and all of my regulators. And exactly none of those groups would let me put your write-up along with the conclusion “risk accepted” in front of them.
Re: Citi’s $900M Misfire Happened During Software Switch
#26reminds me of that missile alerting system gif..
https://giphy.com/gifs/emibob-ads-missile-warning-system-xUL...
Re: Citi’s $900M Misfire Happened During Software Switch
#27I'm put on a banking project (as external) which already flushed down the toilet around ~$500M. Based on my experiences of the meetings and meetings about meetings, I totally understand how the incompetence lead to this clusterfuck.
My question to my boss was rather: "but _where_ do these banks get this huge amount of money from? I guess it's not from the $5 account fees." He answered that although he is in the banking business for decades, he still doesn't know.
These 100s of Millions of losses are not necessarily threatening core business. I find it amusing.
Re: Citi’s $900M Misfire Happened During Software Switch
#28I'm put on a banking project (as external) which already flushed down the toilet around ~$500M. Based on my experiences of the meetings and meetings about meetings, I totally understand how the incompetence lead to this clusterfuck. My question to my boss was rather: "but _where_ do these banks get this huge amount of money from? I guess it's not from the $5 account fees." He answered that although he is in the banki…
> My question to my boss was rather: "but _where_ do these banks get this huge amount of money from? I guess it's not from the $5 account fees."
They create it.
https://www.bankofengland.co.uk/quarterly-bulletin/2014/q1/m...
Re: Citi’s $900M Misfire Happened During Software Switch
#29I'm put on a banking project (as external) which already flushed down the toilet around ~$500M. Based on my experiences of the meetings and meetings about meetings, I totally understand how the incompetence lead to this clusterfuck. My question to my boss was rather: "but _where_ do these banks get this huge amount of money from? I guess it's not from the $5 account fees." He answered that although he is in the banki…
> My question to my boss was rather: "but _where_ do these banks get this huge amount of money from? I guess it's not from the $5 account fees." They create it. https://www.bankofengland.co.uk/quarterly-bulletin/2014/q1/m...
This is a very misunderstood article. The money they "create", i.e loaned or paid out, has to be funded by a deposit or similar borrowing. Making sure they can fund all their commitments is what liquidity managers and treasury departments do, it's why regulators subject banks to annual ILAAPs (Internal Liquidity Adequacy Assessment Process), it's why banks have liquidity risk and modelling teams to manage any "gap" risk banks are running in this respect.
If banks could simply create money then they'd never go bust. The only exception is the Central Bank, which can create new money that is it uses to buy assets of the same value, supporting prices and improving liquidity in the financial system.
Re: Citi’s $900M Misfire Happened During Software Switch
#30This is the key part: "But the employee didn’t select the correct system options -- instead allowing the loan to be repaid in full with interest. Colleagues who are supposed to catch such errors didn’t." Saved you a click.
> ...Colleagues who are supposed to catch such errors didn’t.
This might well be a case of to be the hard problem of shared responsibility becoming someone else's responsibility.
There is no perfect solution - even the implicit death penalty does not prevent avoidable airplane crashes (AF 447, PIA 8303...) - but it seems that things could have been done better here, as indicated by the preceding sentences:
"After Revlon repurchased part of the debt, a Citigroup employee was supposed to manually adjust the share of the loan the remaining lenders still owned ahead of interest payments scheduled to be sent out this month."
It is asking for trouble to have a process that allows you to start a task having no immediate, irreversible consequences, but which presents few or no barriers to accidentally executing one that does. Were the "colleagues who are supposed to catch such errors" notified of the actual transaction that was about to be performed, or only of the one that was intended?