Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UK bank fined £49M over IT system meltdown

bbc.com

Post reply on HN
Print view View original

51–60 of 84 posts

Re: UK bank fined £49M over IT system meltdown

#51

Post by VincentEvans »

teleforce wrote: post #19
VincentEvans wrote: post #13

I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.

Just wondering if the migration disaster at this scale can be avoided using modern cluster and orchestration technology like Kubernetes?

I think downvotes are unnecessary and this is a finely crafted joke.

Re: UK bank fined £49M over IT system meltdown

#52

Post by ywei3410 »

ilyt wrote: post #29
jayceedenton wrote: post #22

I will always remember this incident as the time when the UK general public were exposed, en masse, to Spring error messages. The confusion caused by ordering a member of the general public not to request a bean from a bean factory in a destroy method implementation still makes me laugh, even now.

I remember when we were telling devs to stop returning java 503's with stacktraces to the user. Devs fixed it by returning 200's with stack traces. And as page was ESI stitched together on Varnish, when they fucked up there wasn't just a stacktrace, but a bunch of different ones in various parts of the page.

I really feel this is an infra problem rather than a dev problem though; the reverse-proxy should strip 5xx response bodies before the egress no?

Re: UK bank fined £49M over IT system meltdown

#53

Post by pixl97 »

VincentEvans wrote: post #13

I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.

What were your latency requirements on pulling a record out and how complex were the joins to pull said records?

If you have a simple db structure with a few tables and very clear data/index rules then billions and billions of records is pretty easy. Your indexes cut out 99% of the work and everything runs smooth and efficient.

But then you can have eldritch horrors where your stored procedures look like seedy detective novels where you chase join after join and have scary high memory requirements on execution.

Re: UK bank fined £49M over IT system meltdown

#54

Post by varispeed »

azalemeth wrote: post #16

I have a bank account with TSB and got compensation as a result of this mix-up. Some rather personal experiences of the fiasco: – Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid…

> – The online browser-based system was telemetry and JS heavy, replacing a far leaner page

I remember one of those banks using the "leaner" page also had heavy telemetry turned on at some point. I type very fast, so I noticed that when I was entering my user id, it was lagging heavily. Then I turned on developer tools only to see that they were logging all keystrokes to analytics. Including username and password. At first I thought I got a virus or something, but these appeared to be legit scripts from the bank. So I decided to not use that bank account for a while. I wonder why would they turn something like that on.

Re: UK bank fined £49M over IT system meltdown

#55

Post by varispeed »

heurisko wrote: post #2

2008: "The UK-based IT department of the fifth largest bank continues to dwindle as more jobs go overseas... This round of cuts, starting in June and lasting 12 months, involves up to 250 permanent IT roles and 200 contractors from the bank's technical delivery division, responsible for software development and design." [1] 2018: "Timeline of trouble: how the TSB IT meltdown unfolded". [2] It's probably more complica…

There was also a heavy enforcement of IR35 in the banking sector, so that substantially reduced the access to talent pool.

There was also a tightening of posted worker regulations, so that banks couldn't ship workers from overseas as a source of cheap talented workforce.

Re: UK bank fined £49M over IT system meltdown

#56

Post by ilyt »

ywei3410 wrote: post #52
ilyt wrote: post #29

Earlier quoted context omitted.

I remember when we were telling devs to stop returning java 503's with stacktraces to the user. Devs fixed it by returning 200's with stack traces. And as page was ESI stitched together on Varnish, when they fucked up there wasn't just a stacktrace, but a bunch of different ones in various parts of the page.

I really feel this is an infra problem rather than a dev problem though; the reverse-proxy should strip 5xx response bodies before the egress no?

Eh, depends, for some apps 503 is legit response that should be returned to customer/app, in other cases it's app being badly designed. We did that few times when it made sense but in most it does not. There is no error code for "down for maintenance" so 503 was also used sometimes for that purpose (although we recommended devs to just fail healthchecks so loadbalancer displays its own error page in those cases)

The other problem (let's just say devs were not... that great with architecture) was that they were getting exceptions in the logs without attached URL and other metadata so they kinda wanted to get that exception directly on the webpage.

I'll paint you a picture of how shoddy some stuff was: they were using a templating language to generate JSONs (and had many bugs in it too) for years, instead of just natively encoding some JSON in Java, coz of party shoddy architecture, partly years-long war between frontend and backed dept. They migrated to Git in... 2018 I think ? No CI/CD of any sort till recently.

End result (of them insisting that they will send the exception to frontend to be seen) was actually exceptions being signed and encrypted blob wrapped with a bunch of JS that gathers all the errors (incl. errors that JS on site might've done), adds all the metadata it sees from "browser perspective" and sends it to monitoring endpoint where that is shoved into ES cluster.

Sort of retarded version of distributed tracing that is now in vogue... done somewhere in 2013. But it did catch a bunch of bugs that were "only" showing to users in browser

Re: UK bank fined £49M over IT system meltdown

#57

Post by justinclift »

ryandrake wrote: post #44
usr1106 wrote: post #21

Earlier quoted context omitted.

As a privacy-aware user, when making a contract with a bank (or buying a flight ticket or whatever) you should get assertions that their web site meets certain quality standards so you can use your browser to access the account or actually check in. Paper did not have those incompatibility problems... However, from the BBC article I conclude that even customers with a default browser could not necessarily use their a…

Businesses can change, too. My credit union[1] recently made a web site change causing me to no longer be able to log in. The new shiny red login button they probably paid $millions for an incompetent developer to provide does nothing when you click it (desktop Safari). I vetted the old site which worked perfectly, but now it doesn’t. I’m working on moving my business elsewhere. 1: https://www.techcu.com/

Found similar kinds of things happening with the occasional website too, mainly due to my use of Firefox. eg buttons that used to work, suddenly "do nothing"

Pretty sure its caused by "Chrome-only" developers, as going through the hassle of installing a Chrome/webkit based browser gets things working. But really, fuck that. ;)

Ubereats made this change recently. Naturally, there's no way on their website to contact them about it. :/

Re: UK bank fined £49M over IT system meltdown

#58

Post by iechoz6H »

azalemeth wrote: post #16

I have a bank account with TSB and got compensation as a result of this mix-up. Some rather personal experiences of the fiasco: – Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid…

"I could see someone else's account in detail but was too honest to do anything with that knowledge"

Are you patting yourself on the back for not commiting fraud?

Re: UK bank fined £49M over IT system meltdown

#59

Post by orangepurple »

varispeed wrote: post #54
azalemeth wrote: post #16

I have a bank account with TSB and got compensation as a result of this mix-up. Some rather personal experiences of the fiasco: – Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid…

> – The online browser-based system was telemetry and JS heavy, replacing a far leaner page I remember one of those banks using the "leaner" page also had heavy telemetry turned on at some point. I type very fast, so I noticed that when I was entering my user id, it was lagging heavily. Then I turned on developer tools only to see that they were logging all keystrokes to analytics. Including username and password. At…

Report that to the regulators.

If you're in the US I know for a fact the regulators listen to and review complaints.

https://www.federalreserve.gov/faqs/credit_12666.htm

Re: UK bank fined £49M over IT system meltdown

#60

Post by azalemeth »

iechoz6H wrote: post #58
azalemeth wrote: post #16

I have a bank account with TSB and got compensation as a result of this mix-up. Some rather personal experiences of the fiasco: – Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid…

"I could see someone else's account in detail but was too honest to do anything with that knowledge" Are you patting yourself on the back for not commiting fraud?

I meant more that I didn't answer the question "if I make a bank transaction, like I want to, will it come out of my account or theirs"?
Post reply on HN