Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RBS IT cockup: This sort of thing can destroy a bank, normally

theregister.co.uk

Post reply on HN
Print view View original

11–20 of 71 posts

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#11

Post by objclxt »

davedx wrote: post #5
andyking wrote: post #2

I have been affected by this over the past week or so. I've been on holiday, and came back to a house with no fresh food, and no working bank account to pay for any, so I've been eating processed shite out of the freezer like a student for four days, and it's making me feel a bit poorly, not to mention the freezer is starting to look a little bare. I haven't had breakfast for days, and I have about two meals left if…

I've been with the Nationwide for years now, and never had a problem with them. Being a building society they're a bit less profit driven, though I don't know if anything can protect against offshore cost-cutting exercises in this day and age :/

One of the problems with a large-scale outage for a large financial institution is it will cascade down to other banks.

So if a bank the size of RBS/NatWest (comparable to Bank of America, or Chase) falls over and you don't have an account it might not affect you - unless your employer uses that bank to carry out their pay runs, in which case you're not going to get paid. Several newspapers were reporting people whose house purchases fell through because their solicitors were using NatWest as an intermediary.

(I actually bank with Nationwide as well, although their online service is a bit odd. Then again, when is that not true for a bank?)

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#12

Post by andyking »

pja wrote: post #7
andyking wrote: post #2

I have been affected by this over the past week or so. I've been on holiday, and came back to a house with no fresh food, and no working bank account to pay for any, so I've been eating processed shite out of the freezer like a student for four days, and it's making me feel a bit poorly, not to mention the freezer is starting to look a little bare. I haven't had breakfast for days, and I have about two meals left if…

The Co-Op? (Although rumour has it that they're going to completely replace their computer systems either this year or early next year so that might be a case of jumping from one basket case to another!) RBS appears to have fouled up the scheduling of their overnight batch run in a way that meant large tranches of transactions simply weren't going through the system. Fixing the problem has taken days (which may be do…

I can't use the Co-op. They won't let me have an account because I apparently have a bad credit rating. (The real reason, I expect, is because I have no credit rating as I've never applied for a credit card or loan, or been late on a bill.) They're probably overdue a computer system overhaul as they've recently taken over several other banks and building societies, notably the Britannia where I had a few friends working.

My pay now seems to have appeared, and my second account doesn't have £0.00 in it any more, so I've got my fingers crossed that I'll be able to go to Tesco tonight. I might go to the branch and get cash though, just in case I get the dreaded 'card declined'.

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#13

Post by Zenst »

timrobinson wrote: post #6
tomgallard wrote: post #3

Some more information about the tech behind the failings here: http://www.theregister.co.uk/2012/06/25/rbs_natwest_what_wen... TLDR: Looks like a failure in their CA-7 batch processing systems, which compounded so they weren't able to back out. Lots of the experts in RBS's implementation of CA-7 have been made redundant, and been replaced with offshore-expertise I'm not going to question the proficiency of the offsho…

Specifically, it looks like a manual error during a deployment: http://www.guardian.co.uk/technology/2012/jun/25/how-natwest... "It seems whoever made the update to CA-7 managed to delete or corrupt the files which hold the schedule for the overnight jobs, so they did not run, or ran incorrectly"

The aspect that they didn't export the job queue prior to update is something that cross's the border of neglegence with both feet.

Still can't understand why a bank that had people who made bad deals and lost money causing the bank to partialy fail are kept inplace and paid bonus's and the IT people who did there job well are replaced by cheap labour external to the country and this is at a time when there going on about TAX evasion, this too me makes no sence and is why I don't run banks :|.

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#14

Post by rjknight »

stevejalim wrote: post #9
andyking wrote: post #2

I have been affected by this over the past week or so. I've been on holiday, and came back to a house with no fresh food, and no working bank account to pay for any, so I've been eating processed shite out of the freezer like a student for four days, and it's making me feel a bit poorly, not to mention the freezer is starting to look a little bare. I haven't had breakfast for days, and I have about two meals left if…

+1 for First Direct, but don't take my word for it: http://www.moneysavingexpert.com/banking/compare-best-bank-a...

Ditto.

I had actually decided to move away from Natwest before this outage - as part of RBS acquiring some bank branches from one of the other bankrupt banks, the EU competition authorities forced them to sell some of their accounts to Santander, and they thought it would be a good idea to sell half of my accounts (business and personal credit card) whilst retaining the other half (personal, savings and joint account). Given that I would crawl over broken glass rather than use Santander, this prompted me to think about alternatives and duly switch my personal banking to First Direct (which took a matter of minutes and has worked perfectly), and my business banking will go to either HSBC or Barclays (under consideration because Barclays offer data feeds which I can pipe directly into FreeAgent).

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#15

Post by Isofarro »

Zenst wrote: post #13
timrobinson wrote: post #6

Earlier quoted context omitted.

Specifically, it looks like a manual error during a deployment: http://www.guardian.co.uk/technology/2012/jun/25/how-natwest... "It seems whoever made the update to CA-7 managed to delete or corrupt the files which hold the schedule for the overnight jobs, so they did not run, or ran incorrectly"

The aspect that they didn't export the job queue prior to update is something that cross's the border of neglegence with both feet. Still can't understand why a bank that had people who made bad deals and lost money causing the bank to partialy fail are kept inplace and paid bonus's and the IT people who did there job well are replaced by cheap labour external to the country and this is at a time when there going on…

IT is seen as an overhead, and as something that doesn't contribute directly to profitability. When compared to dealers, dealers bring in the profit, IT slurps it away.

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#16

Post by arethuza »

Zenst wrote: post #13
timrobinson wrote: post #6

Earlier quoted context omitted.

Specifically, it looks like a manual error during a deployment: http://www.guardian.co.uk/technology/2012/jun/25/how-natwest... "It seems whoever made the update to CA-7 managed to delete or corrupt the files which hold the schedule for the overnight jobs, so they did not run, or ran incorrectly"

The aspect that they didn't export the job queue prior to update is something that cross's the border of neglegence with both feet. Still can't understand why a bank that had people who made bad deals and lost money causing the bank to partialy fail are kept inplace and paid bonus's and the IT people who did there job well are replaced by cheap labour external to the country and this is at a time when there going on…

"who made bad deals and lost money"

Quite an understatement that, the €71bn RBS acquisition of part of ABN Amro often makes it into the lists of "worst business deals ever":

http://www.independent.co.uk/news/business/analysis-and-feat...

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#17

Post by jacques_chester »

Bankers do sometimes forget that these days, their entire business is an elaborate front end for a general ledger program.

I predict that banks which see IT as a key source of competitive advantage will prosper in the coming decades. Banks which see it as a cost centre will eventually be slugged with debacles like these.

Of course, such self-serving aggrandisement as this is not without its own problems. Banking software is enormously complex and is meant to be utterly reliable. Two things that rarely mix well.

Add in the storied difficulties of running enormous software projects and you begin to understand why most banks just don't want to hear about how the transactions occur, just that they occur.

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#18

Post by SideburnsOfDoom »

JonWood wrote: post #10
andyking wrote: post #2

I have been affected by this over the past week or so. I've been on holiday, and came back to a house with no fresh food, and no working bank account to pay for any, so I've been eating processed shite out of the freezer like a student for four days, and it's making me feel a bit poorly, not to mention the freezer is starting to look a little bare. I haven't had breakfast for days, and I have about two meals left if…

I moved from Natwest to First Direct a couple of months ago, and have had a great experience so far. Their online banking isn't quite so nice as Natwest's, but that's more than made up for by the amazing phone service. I've had to call a couple of times (once to replace a lost card), and almost instantly been put through to someone who can solve my problem.

I have been with First Direct for a few years and I really like it, wouldn't move. It's good for people like me who want to interact with their bank firstly online, and secondly on the phone, and in person at a branch at a distant third.

First Direct is entirely owned by HSBC - it's essentially a branch of HSBC on the internet not the high street. If you need to deposit a paper cheque or the like you will be directed to a HSBC branch.

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#19

Post by AlexMuir »

pja wrote: post #7
andyking wrote: post #2

I have been affected by this over the past week or so. I've been on holiday, and came back to a house with no fresh food, and no working bank account to pay for any, so I've been eating processed shite out of the freezer like a student for four days, and it's making me feel a bit poorly, not to mention the freezer is starting to look a little bare. I haven't had breakfast for days, and I have about two meals left if…

The Co-Op? (Although rumour has it that they're going to completely replace their computer systems either this year or early next year so that might be a case of jumping from one basket case to another!) RBS appears to have fouled up the scheduling of their overnight batch run in a way that meant large tranches of transactions simply weren't going through the system. Fixing the problem has taken days (which may be do…

The Co-Op run Smile, the internet bank. They've got a great premium current account.

However, I've applied six times online (the only way you can). I'd wade through all my details, previous addresses, employers details, passwords, pin numbers - the works. Took at least 20mins a time - then I'd go to the final page and get a generic "There has been an error with your request, please return to your previous page." And that's it - my entire application lost. So I have been unable to open an account for months. I've given up.

I phoned them, but with no reference number, the woman said they weren't aware of any IT issues and that was that.

Re: RBS IT cockup: This sort of thing can destroy a bank, normally

#20

Post by timrobinson »

Zenst wrote: post #13
timrobinson wrote: post #6

Earlier quoted context omitted.

Specifically, it looks like a manual error during a deployment: http://www.guardian.co.uk/technology/2012/jun/25/how-natwest... "It seems whoever made the update to CA-7 managed to delete or corrupt the files which hold the schedule for the overnight jobs, so they did not run, or ran incorrectly"

The aspect that they didn't export the job queue prior to update is something that cross's the border of neglegence with both feet. Still can't understand why a bank that had people who made bad deals and lost money causing the bank to partialy fail are kept inplace and paid bonus's and the IT people who did there job well are replaced by cheap labour external to the country and this is at a time when there going on…

If it's true that this is a manual error, then I have two questions:

1) Why did it take until the early hours of Wednesday to find out? Why not run a manual check after the change went in?

2) If they found out on Wednesday morning, why did it take until Friday - after three failed batch runs - to fix it?

Post reply on HN