Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B • Hacker News Board

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#211

Post by dctoedt » Thu, Mar 01, 2018, 11:45 PM UTC

dpweb wrote: post #210

dctoedt wrote: post #38

Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128…

Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company. But after selling the deal, their workers have to solve a difficult engineering and organizational management problem…

> Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company.

An EDS executive's alleged overpromising, in landing a big contract to build a CRM system for British Sky Broadcasting, ended up costing EDS big bucks: USD $460 million, or more than four times the value of the original contract [0].

[0] http://www.oncontracts.com/eds-british-sky-overpromising/ (self-cite)

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#212

Post by pixl97 » Thu, Mar 01, 2018, 11:50 PM UTC

threeseed wrote: post #37

jandrese wrote: post #8

Earlier quoted context omitted.

These large government software projects end up costing way more than they should pretty much every time. The process is just broken. There's way too much design by committee and tying the hands of the people who are doing the actual work, preventing them from implementing the correct solution and instead making them implement the one the disconnected-from-reality specification writers decided on. Then they have to g…

This is completely wrong. The CA government engaged the vendor, IBM, after they had decided on a list of requirements from what they wanted from a solution. In your argument you say that the hands of the people doing the actual work were tied. Well that's the vendor, and so no, their hands weren't tied. And not sure if you've ever worked at a project like this but the committee meetings aren't expensive. They are bas…

> but the committee meetings aren't expensive. They are basically free. And there often should be far more of them. The real cost comes from major changes in the design too late in the project plan.

https://en.wiktionary.org/wiki/bikeshedding

> The term was coined as a metaphor to illuminate Parkinson’s Law of Triviality. Parkinson observed that a committee whose job is to approve plans for a nuclear power plant may spend the majority of its time on relatively unimportant but easy-to-grasp issues, such as what materials to use for the staff bikeshed, while neglecting the design of the power plant itself, which is far more important but also far more difficult to criticize constructively. It was popularized in the Berkeley Software Distribution community by Poul-Henning Kamp[1] and has spread from there to the software industry at large.

http://bikeshed.com/

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#213

Post by dragonwriter » Thu, Mar 01, 2018, 11:52 PM UTC

da02 wrote: post #132

dragonwriter wrote: post #68

Earlier quoted context omitted.

> Have you actually tried what you are saying ? Because it's a nice idea just completely unrealistic. I've been in places where it's done. In fact, I've been involved for a long time in a public sector environment with three key systems, one of which is a central claims adjudication system, and two of which are accounting and tracking systems that interface with it, performing similar functions to each other but for…

Was this at the city, state, or federal level? Or in another country?

Large state.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#214

Post by Nelson69 » Thu, Mar 01, 2018, 11:54 PM UTC

j45 wrote: post #175

Nelson69 wrote: post #162

Earlier quoted context omitted.

Didn't IBM effectively give them that guidance? It seems kind of simple in a way, if IBM was hired to build X and failed to do so, then there will be a law suit or some form of arbitration and some money will be recovered or something. It's pretty standard breach of contract type stuff. You can audit everything too, maybe IBM was billing for stuff they shouldn't have; I kind of doubt it but maybe there was something…

Interesting idea. What if X wasn't defined to begin, besides a high level of saying "one way or another we'll get you a working result". Any reasonable firm is hired to map out X to Y as best as possible, or at least be sure they did everything to uncover the possibilities, as much as they may, or may not occur. I have a hard time believing IBM has only experienced this once. I'm not saying IBM would do this, but may…

I guarantee that "X" wasn't defined when the contract was signed. Not defined as something that could be implemented. It was a problem solving mission.

I also want to clarify, I'm not saying IBM plotted X to Y in the most efficient way or that they did it when others could not in my last sentence. I was saying that now that Canada is a $billion deep. They have 3 choices: 1) kill it all and be $billion poorer and have no solution. 2) Somehow beat IBM in to a better deal and get a solution for like $1.2billion, I think news like this is part of the beating. or 3) Probably spend more than like $1.2b to get someone else to get it across the finish line or start over from scratch. Presumably they need something and at this point IBM is probably the closest to the finish line; change horses midstream and you usually get wet. Hey and let's throw out the other condition, IBM might own the product so switching vendors might be a defacto start from scratch.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#215

Post by pixl97 » Thu, Mar 01, 2018, 11:54 PM UTC

dade_ wrote: post #121

tboyd47 wrote: post #102

> The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs. > The government will spend C$431 million to keep the program running in the near term, on top of C$460 million already spent to put Phoenix in place and fix the prob…

There is some missing math. They likely have 30 or 40 payroll systems all up for renewal/replacement. The old systems are probably loaded with features and forgotten requirements that would have been discovered during deployment.

Govt.office "We need to open these X year old databases/files for tax purposes once a year"

Programmer.ibm "What the hell format is this?!"

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#216

Post by lovich » Thu, Mar 01, 2018, 11:56 PM UTC

dpweb wrote: post #210

dctoedt wrote: post #38

Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128…

Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company. But after selling the deal, their workers have to solve a difficult engineering and organizational management problem…

I agree with some of your points but I don't think that deconstructioning the pieces of the project is a panacea. What frequently occurs when that happens is that you get distributed work along with distributed responsibility. Then whenever anything goes wrong or needs to change, there is no one who can make it happen.

You get team A who needs team B to make a new api, but they won't do it till they get a new device because their kpis don't improve for any work they do for team A, so HR needs to be involved the do hiring but they need to talk to accounting to approve the budget and on and on and on.

If you've ever seen Rick and Morty, the episode where aliens have accidentally pulled another character in and the leader is trying to find out why, when every department blames another and he says "oh, so it's nobodies fault" is a perfect example of most large projects

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#217

Post by tempestn » Fri, Mar 02, 2018, 12:00 AM UTC

loufe wrote: post #164

bcholmes wrote: post #156

Earlier quoted context omitted.

And why isn't "The Harper Government"(TM) not named?

I'm not a fan of the convservatives but it's still unfair to place the blame of a bad choice of client on a political party. A lot of non-partisans were involved in getting the project running.

Sure, but it would make more sense to refer to them by name than as "Justin Trudeau's Conservative predecessors" or whatever.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#218

Post by i_feel_great » Fri, Mar 02, 2018, 12:00 AM UTC

Narkov wrote: post #154

tootie wrote: post #149

Earlier quoted context omitted.

Or Oregon v Oracle over their state healthcare exchange. http://www.oregonlive.com/politics/index.ssf/2016/09/post_18...

Or in Queensland with the $1.5b health payroll failure. https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-...

I have known two people who worked in the Queensland Health payroll roll-out. Both actually blamed the government for constantly changing specs, and politicians making claims in parliament about deliverables and deadlines that they then had to meet. They also said IBM (or whatever actual contractor they had on the ground) should have managed things better.

I now work with someone who was at Main Roads Queensland who claims the whole of the Queensland government has an implicit ban on IBM for any new work, and are trying to replace all current IBM products and services.

Edit: My previous job used DB2 on an IBM server. During end of financial year time we had to ask them to turn on more server processors on the server. Jeeesus they charge like a wounded bull!

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#219

Post by simonh » Fri, Mar 02, 2018, 12:03 AM UTC

giobox wrote: post #181

goodroot wrote: post #77

As a Canadian who has worked at IBM, I'm not surprised that this didn't find success. In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale. With that being said, IBM isn't a successful technology company with a proven record building good software products. They were an unwise choice from the get-go. IBM is a successful fin…

> In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale. How in any way is this "In IBMs defense"? We should think it OK that they pursued and committed to a large government contract that in reality they were not qualified to complete? The sentence reads more as a condemnation.

“Our” in this case clearly refers to Canadians, not IBM. Do you know what public sector means?

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#220

Post by tempestn » Fri, Mar 02, 2018, 12:06 AM UTC

CivBase wrote: post #80

Does anyone know what the real problem was here? > Some people were paid too much, others not at all. Those issues snowballed with the deluge of requests to fix incorrect paychecks. > IBM said it’s “fulfilling its obligations on the Phoenix contract, and the software is functioning as intended.” Okay, so the cost is coming from incorrect paycheck values, but the software is working correctly? Does that mean it's huma…

From the anecdotes I've heard, it sounds like the support staff is just wildly insufficient to handle all the issues that crop up with a major change like this. Initially the government switched over a trial group of (iirc) something like 40k employees. There were a bunch of problems with people not being paid or being paid the wrong amounts, but for some incomprehensible reason (perhaps they wanted to ditch the old system to focus on fixing the new one or something crazy like that?) they switched over everyone before fixing those.

Now, I personally know people who've been getting incorrect pay for over a year, who submit a complaint to their HR, who pass it up the chain, where... nothing seems to happen. Presumably because the 500 people or whatever it is tasked to deal with this are literally dealing with tens of thousands of issues, many of which aren't trivial changes (and so block the many that actually are trivial changes, but aren't even being seen).

Basically it was released before it was ready, initial signs of issues were ignored, and now the human side of the system is overwhelmed. It's kind of like when a web server gets overwhelmed with traffic, and it isn't set to reject at a reasonable threshold, so things start getting queued and timing out, people start refreshing pages, and it just snowballs until the whole thing goes down. Which apparently is what has now happened.