Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B • Hacker News Board

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#161

Post by wahern » Thu, Mar 01, 2018, 10:06 PM UTC

dctoedt wrote: post #38

Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128…

Apropos HN, he then says,

  officials set out to fix Indiana's poorly-performing
  welfare system by inserting an untested, theoretical
  experiment, and substitute personal caseworkers with
  computers and phone calls ("remote eligibility"). This
  is now admitted to be an error, and there is nothing in
  this case, or the Court's power, that can be done to
  correct it, or remedy the lost taxpayer money or
  personal suffering of needy Hoosiers.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#162

Post by Nelson69 » Thu, Mar 01, 2018, 10:09 PM UTC

j45 wrote: post #84

goodroot wrote: post #77

As a Canadian who has worked at IBM, I'm not surprised that this didn't find success. In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale. With that being said, IBM isn't a successful technology company with a proven record building good software products. They were an unwise choice from the get-go. IBM is a successful fin…

I know folks who work at IBM as well - don't mean to put you on the spot - but IBM should be far better at leading and guiding customers to success. Except at $250-350/hour, below average performance is more profitable than successful and a lot of good people suffer on both sides.

Didn't IBM effectively give them that guidance?

It seems kind of simple in a way, if IBM was hired to build X and failed to do so, then there will be a law suit or some form of arbitration and some money will be recovered or something. It's pretty standard breach of contract type stuff. You can audit everything too, maybe IBM was billing for stuff they shouldn't have; I kind of doubt it but maybe there was something really amiss.

On the other hand if IBM was hired to build X, then it changed in to X' and then X'' and 10,000 changes later it turns our, Canada really wanted Y from the start, isn't IBM guiding them along their process to figure that out? It's an insanely expensive way to build and buy stuff.

Is there supposed to be some sort of internal customer's advocate that looks at the project and says "hmm, this is a payroll system, it shouldn't cost a $billion typically, or this was a $100m contract that is already up to $300m, something must be wrong..." Don't get me wrong, in most sales driven organizations, they're high-fiving whoever inked that contract and trying to get invited to the parties he throws on his yacht. The catastrophic failure sucks for everybody, it's hard to see someone else finishing the job more inexpensively that getting IBM to do it though.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#163

Post by mkstowegnv » Thu, Mar 01, 2018, 10:11 PM UTC

kabdib wrote: post #94

blantonl wrote: post #85

The software platform itself, a heavily modified version of Oracle’s PeopleSoft program ... What implementation of Peoplesoft isn't heavily modified? Adapting business processes to COTS software packages is extremely challenging. Anyone remember the absolute bank that Peoplesoft consultants made back in the day when ERP and Business Process Re-engineering was all the rage?

When I last "dealt with" PeopleSoft, the standard operating procedure was: 1) Pay PeopleSoft/Oracle a metric f--kton of money. 2) A CD arrives in the mail. (This little envelope cost $800,000?) 3) Your PeopleSoft consultants show up, laugh at that CD (tossing it into the microwave) and pull out their own distros with a zillion customizations that they proceed to spray over your servers. You have no idea what they are…

I was privvy to discussions surrounding a search for a university president. One of the candidates dropped out when he heard that the university was replacing a perfectly functional home brewed accounting system with PS - it had brought his previous university to its knees and he didn't want to ever deal with it again.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#164

Post by loufe » Thu, Mar 01, 2018, 10:18 PM UTC

bcholmes wrote: post #156

mattnewton wrote: post #153

Earlier quoted context omitted.

Yeah why is his name there at all?

And why isn't "The Harper Government"(TM) not named?

I'm not a fan of the convservatives but it's still unfair to place the blame of a bad choice of client on a political party. A lot of non-partisans were involved in getting the project running.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#165

Post by lopmotr » Thu, Mar 01, 2018, 10:21 PM UTC

"there is reputational risk for IBM in not helping us fix this,”" - not true. IBM has a long history of exactly this kind of problem. Canada didn't care about their existing poor reputation and the next government to hire them won't either.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#166

Post by cgio » Thu, Mar 01, 2018, 10:22 PM UTC

chime wrote: post #56

zaroth wrote: post #23

Not having ever worked on anything like this, the death knell for these projects must be the inability to ever reduce functionality into a reasonable core set. Basically the 80/20 rule but where there are practically an infinite number of edge cases that stretch the project on indefinitely? You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% co…

Having worked on similar things on a smaller scale but higher up the chain, I can tell you the list of possible issues is a mile long. Everything from lack of leadership and conflicting goals to ever-changing scopes and insufficient resources is a candidate for the big blame. A lot of people, organizations, and systems have to work well together to succeed. As techies, we think the technical solution is the most supe…

I would have gone for the 8 hrs option. Polluting the ui with redundant information and flow on impact to reporting potentially. The impact of saving some hrs might end up amounting to even more. Also, testing and lifecycle of this approach could be worse. And a sign of how we sometimes get carried away by tge tech definition. Why call it 1 or more tickets rather than any tickets?

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#167

Post by flukus » Thu, Mar 01, 2018, 10:24 PM UTC

ugh123 wrote: post #147

goodroot wrote: post #141

Earlier quoted context omitted.

You're right. That was too sweeping, in fairness. I've edited in 'good software products'; that record is categorically abysmal. IBM makes amazing mainframes and physical technical systems, and conducts some fascinating cutting-edge research, and even built really effective punch-card machines for use within Nazi concentration camps. ( https://en.wikipedia.org/wiki/IBM_and_the_Holocaust )

You're mistakenly conflating IBM's consulting business with the rest of their current technology, both hardware and software.

No, IBM has conflated their consulting service with the rest of their technology by running under the same brand name.

If they want the benefits of sharing that brand between operations then they deserve the negatives that come along with it.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#168

Post by cletus » Thu, Mar 01, 2018, 10:28 PM UTC

Narkov wrote: post #154

tootie wrote: post #149

Earlier quoted context omitted.

Or Oregon v Oracle over their state healthcare exchange. http://www.oregonlive.com/politics/index.ssf/2016/09/post_18...

Or in Queensland with the $1.5b health payroll failure. https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-...

Ha, funny story about that.

My brother-in-law's uncle is a doctor (specialist) in Queensland. A couple of years ago he told us a story about this. Because of the massive screwups a bunch of private practitioners like himself weren't getting paid by the state government for work they did for the public health system.

It reached the point where a lot of doctors were simply not going to provide nonessential services if they continued to not get paid (this went on for a year or more). So some workers in the state health department intervened to help resolve the situation. I don't remember the specifics but they essentially approved an advancement that sort of looked like a loan as far as the government system was concerned.

So this guy did get paid. Almost a year later he started getting threatening letters from the government for defaulting on his loan and I think they were threatening court action.

The health department who organized this were sympathetic but said because of the system their hands were tied.

This actually reached the point of the government initiating an investigation and potentially having ramifications for his medical license with the health department seemingly powerless to stop these wheels that were put in motion and looked to be headed to court.

So this guy ended up sending a letter to the health minister threatening to not provide any medical services to the state government at all if this wasn't resolved. Now this would actually have left the state in a pickle as for some things he was the only provider in the state, basically.

Sure enough it was resolved within 2 weeks and the health minister apologized.

Anyway, from this one anecdote it certainly seemed like Queensland health was a disaster.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#169

Post by greenshackle3 » Thu, Mar 01, 2018, 10:32 PM UTC

mseebach wrote: post #139

zaroth wrote: post #23

Not having ever worked on anything like this, the death knell for these projects must be the inability to ever reduce functionality into a reasonable core set. Basically the 80/20 rule but where there are practically an infinite number of edge cases that stretch the project on indefinitely? You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% co…

> You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞ I read something in a trade journal probably probably about 15 years ago, that really stuck with me: When businesses buy big software solutions (I think the article was about ERP systems), they should prepare to adjust and change their processes (within reason, of course) and not ju…

Having worked on ERP software for medium businesses, that sounds spot-on to me; in my experience the biggest challenge was not so much technical as getting clients to change their processes so they can be systematized in a sane way.

There is a lot of resistance from employees not willing to change the way they work. Even seemingly simple things like standardizing language / glossary between departments so we can have a single representation in the system can be an uphill battle.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#170

Post by lopmotr » Thu, Mar 01, 2018, 10:34 PM UTC

diggernet wrote: post #129

x0x0 wrote: post #60

Earlier quoted context omitted.

It seems like lots of the payroll was being carried out by humans in local units. Humans are very good at holding lots of exceptions in our heads and, in the absence of clear rules, doing reasonable things. Computers are shit at both of the above. So things like Bob negotiated with his boss to leave early on Thursday to cover childcare and, in exchange, is coming in early on Tuesdays... but got asked to come in Satur…

> Humans are very good at holding lots of exceptions in our heads and, in the absence of clear rules, doing reasonable things. Computers are shit at both of the above. Which is a remarkably concise summary of my thoughts regarding self driving cars.

When did you form those thoughts? The last 10 years has had pretty monotonic progress in the opposite direction. It seems unlikely for the technology to suddenly stop and certainly not go backwards.