Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK bank fined £49M over IT system meltdown

bbc.com

Post reply on HN
Print view View original

41–50 of 84 posts

Re: UK bank fined £49M over IT system meltdown

#41

Post by PeterisP »

jbkkd wrote: post #35
azalemeth wrote: post #16

I have a bank account with TSB and got compensation as a result of this mix-up. Some rather personal experiences of the fiasco: – Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid…

Honest question - why do you still have an account with them?

In general, banks compete on other attributes - a small difference in a mortgage interest rate is a lot of money and makes up for a HUGE difference in the quality of internet services; and whenever loan market becomes tight (and thus it's not attractive to refinance to another bank) people are pretty much locked in.

Re: UK bank fined £49M over IT system meltdown

#42

Post by gghhzzgghhzz »

VincentEvans wrote: post #13

I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.

It's not really the quantity of data that is important in migrations like this.

It's what is and isn't in the data - often a lot of junk in my experience if the source system is a legacy system that has evolved

what meaning that data has within a completely different system

what the demands are on the completeness of that data is in the new system

how to deal with exceptions

and whether that data can ever be frozen, or whether it is still online (as in the case of banking transactions)

This is unlikely to be simply a technical problem of ETLing tables, changing date ranges from inclusive to exclusive and mapping some address fields.

Of course the size of the data after a certain point does make a big difference to risk planning and business continuity planning. It's not possible to rollback and try again within the migration window should a catastrophic issue occur, and it's not possible to simply run some bulk updates to fix issues during the go-live validation.

It is noted though in this project that the data migration itself was not found to contribute to the failure.

Re: UK bank fined £49M over IT system meltdown

#43

Post by insomniacity »

petepete wrote: post #8

Hopefully Virgin Money will get one too. They broke their Android app earlier this year and since they make you verify web logins using the app I was unable to access any of my business accounts for ~3 weeks. If something really urgent had come up I could have done what I needed via telephone banking or in a branch, but it was a huge pain in the arse because of a single point of failure. Just let me use a Yubikey as…

+1 on the Yubikey. I'm pretty good at moving my savings around and getting the best interest rate possible - the side effect is a ton of accounts, which means I'm drowning in 'secure memorable passcode key PINs' and my SMS inbox is full of SMS 2FA codes, and I'm wondering what it would take to get a bank to offer Webauthn/FIDO.

How about a website where we pledged to open an account and deposit £X into savings, or switch current account, if they offered Webauthn/FIDO?

Re: UK bank fined £49M over IT system meltdown

#44

Post by ryandrake »

usr1106 wrote: post #21
azalemeth wrote: post #16

I have a bank account with TSB and got compensation as a result of this mix-up. Some rather personal experiences of the fiasco: – Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid…

As a privacy-aware user, when making a contract with a bank (or buying a flight ticket or whatever) you should get assertions that their web site meets certain quality standards so you can use your browser to access the account or actually check in. Paper did not have those incompatibility problems... However, from the BBC article I conclude that even customers with a default browser could not necessarily use their a…

Businesses can change, too. My credit union[1] recently made a web site change causing me to no longer be able to log in. The new shiny red login button they probably paid $millions for an incompetent developer to provide does nothing when you click it (desktop Safari). I vetted the old site which worked perfectly, but now it doesn’t. I’m working on moving my business elsewhere.

1: https://www.techcu.com/

Re: UK bank fined £49M over IT system meltdown

#45

Post by graderjs »

zeristor wrote: post #17
arpinum wrote: post #10

The 250+ page analysis of the incident was an excellent insight into how large IT projects fail: https://www.tsb.co.uk/news-releases/slaughter-and-may/slaugh... money quote: > This situation has all the hallmarks of business management strong-arming the IT organization into an unrealistic timeline. When business leaders push for overly-aggressive timelines, or regulators ask for multiple competing risk frameworks and…

The report is by Slaughter & May, one of the more delightful company names in the City of London. My understanding was that they’re a law firm, perhaps they’ve also branched into IT consultancy?

No this kind of thing is called an audit and law firms are typically involved.

Re: UK bank fined £49M over IT system meltdown

#46

Post by joshuaissac »

zeristor wrote: post #17
arpinum wrote: post #10

The 250+ page analysis of the incident was an excellent insight into how large IT projects fail: https://www.tsb.co.uk/news-releases/slaughter-and-may/slaugh... money quote: > This situation has all the hallmarks of business management strong-arming the IT organization into an unrealistic timeline. When business leaders push for overly-aggressive timelines, or regulators ask for multiple competing risk frameworks and…

The report is by Slaughter & May, one of the more delightful company names in the City of London. My understanding was that they’re a law firm, perhaps they’ve also branched into IT consultancy?

Law firms are often hired to conduct independent reviews when things go wrong or when allegations of wrongdoing are made, e.g. RBS in 2013,[1] RICS in 2018,[2] Baker McKenzie in 2018[3], and UNICEF in 2020.[4][5]

1. https://www.natwestgroup.com/news-and-insights/feature-conte...

2. https://www.rics.org/uk/about-rics/corporate-governance/inde...

3. https://www.legalbusiness.co.uk/blogs/metoo-latest-bakers-ap...

4. https://www.unicef.org.uk/press-releases/unicef-uk-confirms-...

5. https://www.civilsociety.co.uk/news/unicef-appoints-differen...

Re: UK bank fined £49M over IT system meltdown

#47

Post by JCM9 »

I remember when this all happened. Would be interesting if it was the result of some really interesting technical bug that nobody could have foresaw followed by a fascinating effort to save the migration.

In fact it was all quite boring and simply the result of the sheer incompetence of the bank’s leadership in running bank IT.

Re: UK bank fined £49M over IT system meltdown

#48

Post by jw1224 »

jayceedenton wrote: post #22

I will always remember this incident as the time when the UK general public were exposed, en masse, to Spring error messages. The confusion caused by ordering a member of the general public not to request a bean from a bean factory in a destroy method implementation still makes me laugh, even now.

“Just want to see my balance and these guys @tsb think I'm robbing a bean factory with a bomb, jesus”

Brilliant. Reminisce with a screenshot here:

https://twitter.com/thejackthomson_/status/98856435451268710...

Re: UK bank fined £49M over IT system meltdown

#49

Post by PeterisP »

sofixa wrote: post #18
lmm wrote: post #7

Earlier quoted context omitted.

True as far as it goes, but "everything broke" as a predictable result of poor decisions they made, such as moving everyone over in one go.

> such as moving everyone over in one go. It's the core banking system of a big bank. Handling consistent state between the old and new systems while progressively migrating customers would probably have been extremely complex. They also get to have a maintenance window (few people will complain if they get warned their bank and all transactions won't work for 2 hours in the middle of the night on Monday). A "big ban…

I've been a part of multiple "big bang" migrations in banking (generally scheduled to the coincidences of local banking holidays next to a weekend, so you can afford multiple days of semi-downtime) and all of them had explicit multiple testing gates for potential rollback during the migration, where after pretty much all the stuff is on the new system, the board convenes and after looking at the difficulties (there inevitably are some unexpected difficulties) make the decision whether they "accept" the switch to the new system or postpone the switch.

Part of your preparation and testing is the rollback of a partial migration - if you're irreversibly committed to the "big bang" before you know its outcome, then your preparation and testing has failed.

Re: UK bank fined £49M over IT system meltdown

#50

Post by FpUser »

teleforce wrote: post #19
VincentEvans wrote: post #13

I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.

Just wondering if the migration disaster at this scale can be avoided using modern cluster and orchestration technology like Kubernetes?

Kubernetes are to help you scale. They do not fix one's incompetence. They increase complexity of the stack and if anything would make it even worse for the incompetents.
Post reply on HN