Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK bank fined £49M over IT system meltdown

bbc.com

Post reply on HN
Print view View original

21–30 of 84 posts

Re: UK bank fined £49M over IT system meltdown

#21

Post by usr1106 »

azalemeth wrote: post #16

I have a bank account with TSB and got compensation as a result of this mix-up. Some rather personal experiences of the fiasco: – Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid…

As a privacy-aware user, when making a contract with a bank (or buying a flight ticket or whatever) you should get assertions that their web site meets certain quality standards so you can use your browser to access the account or actually check in.

Paper did not have those incompatibility problems...

However, from the BBC article I conclude that even customers with a default browser could not necessarily use their account

Edit: Forgotten not added.

Re: UK bank fined £49M over IT system meltdown

#22

Post by jayceedenton »

I will always remember this incident as the time when the UK general public were exposed, en masse, to Spring error messages.

The confusion caused by ordering a member of the general public not to request a bean from a bean factory in a destroy method implementation still makes me laugh, even now.

Re: UK bank fined £49M over IT system meltdown

#23

Post by jhugo »

teleforce wrote: post #19
VincentEvans wrote: post #13

I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.

Just wondering if the migration disaster at this scale can be avoided using modern cluster and orchestration technology like Kubernetes?

Without even having got around to reading the whole report yet, I can promise you that a f*ckup on this scale cannot be avoided solely through technology decisions. The problem was (is always) with the people and the structures they were working in.

Re: UK bank fined £49M over IT system meltdown

#24

Post by usr1106 »

teleforce wrote: post #19
VincentEvans wrote: post #13

I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.

Just wondering if the migration disaster at this scale can be avoided using modern cluster and orchestration technology like Kubernetes?

No technology can compensate poor planning and technical incompetence. From all I read that was the root cause of the problem. So the same people and processes using Kubernetes: No.

(Of course this is just speculation. I have no insider knowledge.)

Re: UK bank fined £49M over IT system meltdown

#25

Post by kitd »

lmm wrote: post #7
makomk wrote: post #6

Earlier quoted context omitted.

That probably has very little to do with it. The immediate cause of all the problems was that Lloyds TSB was forcibly split up in order to try and increase competition and the Lloyds half kept the IT department, and when the TSB half tried to move over to the existing IT platform of their new parent company everything broke.

True as far as it goes, but "everything broke" as a predictable result of poor decisions they made, such as moving everyone over in one go.

Note that "they" in this case is the new parent company. IIRC, they were a fairly new bank, heavily reliant on technology. They had the tech but not the customers. TSB were the opposite. The parent thought their tech, which up to that point was only dealing with 100,000s of records, could deal with billions with little change. They were spectacularly wrong and it showed in all testing. But their management pushed ahead to go live anyway.

Their actions are not far from criminal negligence IMHO.

Re: UK bank fined £49M over IT system meltdown

#26

Post by ilyt »

arpinum wrote: post #10

The 250+ page analysis of the incident was an excellent insight into how large IT projects fail: https://www.tsb.co.uk/news-releases/slaughter-and-may/slaugh... money quote: > This situation has all the hallmarks of business management strong-arming the IT organization into an unrealistic timeline. When business leaders push for overly-aggressive timelines, or regulators ask for multiple competing risk frameworks and…

Sadly far too often managers and decisionmakers think timeline and deadlines is a topic for haggle, not discussion. They treat it same as haggling on a bazaar.

Re: UK bank fined £49M over IT system meltdown

#27

Post by Nextgrid »

zeristor wrote: post #17
arpinum wrote: post #10

The 250+ page analysis of the incident was an excellent insight into how large IT projects fail: https://www.tsb.co.uk/news-releases/slaughter-and-may/slaugh... money quote: > This situation has all the hallmarks of business management strong-arming the IT organization into an unrealistic timeline. When business leaders push for overly-aggressive timelines, or regulators ask for multiple competing risk frameworks and…

The report is by Slaughter & May, one of the more delightful company names in the City of London. My understanding was that they’re a law firm, perhaps they’ve also branched into IT consultancy?

> they’re a law firm, perhaps they’ve also branched into IT consultancy?

And that's exactly how such IT disasters begin.

Re: UK bank fined £49M over IT system meltdown

#28

Post by hennell »

VincentEvans wrote: post #13

I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.

When you say "emphasis on" you mean the single mention with no distinction between that stat and the any other of the facts of the case? I don't think they really put any weight on if that's a lot or not.

Personally I would say it is a lot. No other number mentioned in the article even approaches a billion. Billions are big. It might be not be true big data big, but it is still a lot of customer records for a migration project (depending on what exactly they were trying to do within the migration) and it does illustrate why they had so many issues, because there was a lot to deal with.

Re: UK bank fined £49M over IT system meltdown

#29

Post by ilyt »

jayceedenton wrote: post #22

I will always remember this incident as the time when the UK general public were exposed, en masse, to Spring error messages. The confusion caused by ordering a member of the general public not to request a bean from a bean factory in a destroy method implementation still makes me laugh, even now.

I remember when we were telling devs to stop returning java 503's with stacktraces to the user.

Devs fixed it by returning 200's with stack traces.

And as page was ESI stitched together on Varnish, when they fucked up there wasn't just a stacktrace, but a bunch of different ones in various parts of the page.

Re: UK bank fined £49M over IT system meltdown

#30

Post by [deleted] »

kitd wrote: post #25
lmm wrote: post #7

Earlier quoted context omitted.

True as far as it goes, but "everything broke" as a predictable result of poor decisions they made, such as moving everyone over in one go.

Note that "they" in this case is the new parent company. IIRC, they were a fairly new bank, heavily reliant on technology. They had the tech but not the customers. TSB were the opposite. The parent thought their tech, which up to that point was only dealing with 100,000s of records, could deal with billions with little change. They were spectacularly wrong and it showed in all testing. But their management pushed ahe…

[deleted]
Post reply on HN