i remember this. for at least a week people couldn't access their money. it was chaos. the bank lost lots of money and customers due to this botched transfer.
Not your keys, not your coins.
UK bank fined £49M over IT system meltdown
11–20 of 84 posts
Re: UK bank fined £49M over IT system meltdown
#12The actual report by the FCA: https://www.fca.org.uk/publication/final-notices/tsb-bank-pl...
Quite thorough report, some points that stand out from the summary:
>SABIS was TSB’s principal outsourced provider
>SABIS relied extensively on 85 third parties (TSB’s fourth parties) to deliver the systems required for the migration and the operation of the platform, which required it to act as a service aggregator.
It amazes me the sheer complexity of a retail bank software system and I suspect most of it is due to legacy systems, legal requirements and lack of regular spring cleaning.
Re: UK bank fined £49M over IT system meltdown
#13I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot.
At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable.
I never claimed to have worked with big data.
Re: UK bank fined £49M over IT system meltdown
#14Sounds like classic A type personalities with zero technical chops deciding how long a technical project should take to further their own agenda:
> The Migration Programme experienced delays from the outset and fell behind the IMP timings. While progress had been made, on 20 September 2017 the firm decided that the Migration Programme would have to be re-planned. However, nine days after it had resolved to re-plan, and before it had concluded its re-planning exercise, TSB publicly announced it would now migrate in Q1 2018.
Re: UK bank fined £49M over IT system meltdown
#15i remember this. for at least a week people couldn't access their money. it was chaos. the bank lost lots of money and customers due to this botched transfer.
Not your keys, not your coins.
Nobody actually lost any money. The "coins" were still mine (I was affected) and heavy banking regulation in the UK probably reduced the harm to effectively zero for the vast majority of retail customers.
I received compensation for the harm caused and got all of my money back.
Re: UK bank fined £49M over IT system meltdown
#16I have a bank account with TSB and got compensation as a result of this mix-up.
Some rather personal experiences of the fiasco:
– Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid extensions block.
– This was introduced at the time of the switchover, and until that point the IT system used looked identical between Lloyds, TSB and Halifax / BOS systems (I have accounts with some of those)
– The online browser-based system was telemetry and JS heavy, replacing a far leaner page
– I was unable to log in during the time of the fiasco, mostly due to 403 errors or timeouts. Often the page would just hang as an async request wasn't answered.
– Once I did manage to log in, I was amazed to see another person's account details (!!!), replete with (their) name and statement.
– I was unable to use online banking to pay bills or check my balance – I could see someone else's account in detail but was too honest to do anything with that knowledge. I can't remember if my card stopped working but I was effectively forced to make other arrangements for quite an extended period of time.
Re: UK bank fined £49M over IT system meltdown
#17The 250+ page analysis of the incident was an excellent insight into how large IT projects fail: https://www.tsb.co.uk/news-releases/slaughter-and-may/slaugh... money quote: > This situation has all the hallmarks of business management strong-arming the IT organization into an unrealistic timeline. When business leaders push for overly-aggressive timelines, or regulators ask for multiple competing risk frameworks and…
The report is by Slaughter & May, one of the more delightful company names in the City of London.
My understanding was that they’re a law firm, perhaps they’ve also branched into IT consultancy?
Re: UK bank fined £49M over IT system meltdown
#18Earlier quoted context omitted.
That probably has very little to do with it. The immediate cause of all the problems was that Lloyds TSB was forcibly split up in order to try and increase competition and the Lloyds half kept the IT department, and when the TSB half tried to move over to the existing IT platform of their new parent company everything broke.
True as far as it goes, but "everything broke" as a predictable result of poor decisions they made, such as moving everyone over in one go.
> such as moving everyone over in one go.
It's the core banking system of a big bank. Handling consistent state between the old and new systems while progressively migrating customers would probably have been extremely complex. They also get to have a maintenance window (few people will complain if they get warned their bank and all transactions won't work for 2 hours in the middle of the night on Monday). A "big bang" migration makes more sense, if everything is properly prepared and tested, which it wasn't.
Re: UK bank fined £49M over IT system meltdown
#19I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.
Just wondering if the migration disaster at this scale can be avoided using modern cluster and orchestration technology like Kubernetes?
Re: UK bank fined £49M over IT system meltdown
#20The 250+ page analysis of the incident was an excellent insight into how large IT projects fail: https://www.tsb.co.uk/news-releases/slaughter-and-may/slaugh... money quote: > This situation has all the hallmarks of business management strong-arming the IT organization into an unrealistic timeline. When business leaders push for overly-aggressive timelines, or regulators ask for multiple competing risk frameworks and…
Another write up:
https://www.computerweekly.com/news/252474170/TSB-programme-...