Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

itprotoday.com

Post reply on HN
Print view View original

251–260 of 333 posts

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#251

Post by mikekchar »

vidanay wrote: post #22

So, what makes a payroll system at this scale so difficult? On the face of it, there are salary employees and there are hourly employees. How many taxing bodies does Canada have? Federal, provincial, and municipal? Is that a few dozen or a few hundred? Is Canadian tax code crazy complicated that doesn't allow for a rules based system? I'm honestly curious as to how a "simple" payroll system can go so far off the rail…

The headline is misleading. According to the content of the article, the project has been going for 10 years and has spend $460 million. The government has pledged another $431 million in the future to deal with problems going forward.

$460 million over 10 years is $46 million a year, which is still a lot of money, but if you break it down it's not nearly so mystifying. If all the money is being spent by contractors, we could assume that the loaded labour rate of the contractors is about $200K per year. It's been more than a decade since I was in Canada, but I'm guessing this is not an outrageous rate. That breaks down to 230 people working on the project. In truth, the payroll system seems to have been operational for the last 2 years and the system employs 500 employees (it was supposed to replace 1200 employees, thus saving money, but there is no indication if those 1200 were, indeed, replaced). I'm guessing that the $460 million includes those 500 employees for at least 2 years and may also include capital costs to house them (the reporter may not be interested in depreciating capital assets in their report).

So it's hard to say exactly how many contractors worked on this, but it's probably safe to say that there were more than 100. A government project with more than 100 contractors == not possible to succeed in my experience. Just the communication issues alone would be insurmountable. As time wore on and pressure to put something into use grew, the mistakes would have piled up causing absolute mayhem.

Unfortunately this is an all too common occurrence. If you go to any meetup in Ottawa and talk to government contractors I'm sure there will be a couple wiping their brow and saying, "This is a stroke of luck because it will take some of the heat off our our project that is burning even more money".

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#252

Post by kevinSuttle »

dpweb wrote: post #210
dctoedt wrote: post #38

Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128…

Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company. But after selling the deal, their workers have to solve a difficult engineering and organizational management problem…

If only people knew how often employees of companies like this scrambled to build something in a mad panic because some exec made a promise to a client about software that wasn’t even designed yet.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#253

Post by wolco »

theossuary wrote: post #196
samstave wrote: post #186

Earlier quoted context omitted.

The ONLY person who would think all that is a good idea are the engineers who were able to update their linkedin with "implemented ML based solution to replace human resources with computer resources to streamline the welfare system for the state of X"

I don't think any engineer would consider that a good idea. But if somebody tells you that's what you're working on for the next few years, make lemonade I guess.

Still sounds like fun to build.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#254

Post by tempestn »

grafporno wrote: post #242
tempestn wrote: post #220

Earlier quoted context omitted.

From the anecdotes I've heard, it sounds like the support staff is just wildly insufficient to handle all the issues that crop up with a major change like this. Initially the government switched over a trial group of (iirc) something like 40k employees. There were a bunch of problems with people not being paid or being paid the wrong amounts, but for some incomprehensible reason (perhaps they wanted to ditch the old…

> a trial group of (iirc) something like 40k employees Oh boy.

Don't quote me on that number; it's from memory. Point is, it was a fraction of the total, and then they released it to everyone without solving the problems in the trial group.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#255

Post by bsder »

bloomingfractal wrote: post #5

and that's why governments should hire software engineers that can be in the loop and understand not only the technical but the policy side. Hiring an army of contractors earning 300k/year to deliver "something" in the waterfall model is a recipe for disaster.

"Understand" != "Empowered"

One of the things people grossly underestimate when dealing with government is that nobody ever really has enough power to make unilateral decisions.

I'm not even putting the bar at a correct decision, simply A decision.

Just try terminating a person or a contract with the government. Good luck getting the 12 signatures you need.

Even IF IBM screwed up this badly, who with sufficient power in the government is going to sign off on the order to fire IBM? It's a small world, and if you piss off somebody more powerful than you, your position is toast.

The cardinal rule of bureaucracy is to never sign off on anything unless you absolutely have to.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#256

Post by jlarocco »

dpweb wrote: post #210
dctoedt wrote: post #38

Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128…

Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company. But after selling the deal, their workers have to solve a difficult engineering and organizational management problem…

> Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company.

Companies as large as IBM don't have a single unified culture. The sales groups are sales oriented, and the engineering groups are engineering/ tech oriented. Most of the time the two aren't in the same building, and often not even the same city.

> Draw horizontal lines on the org chart. The sales guys are at the top. They are generally competent. Proof = they sold a massive deal. The devs are at the bottom. They are often competent. They have to be or they wouldn't get hired or find work. You can PROVE someone doesn't know dev work. But, they don't have enough POWER to change things or fix things.

> Reasons for failure:

> 1. The middle. This is the breakdown. Many PMs often have no real skills. ORGANIZING for success, given a complex technical and organizational problem. 2. The projects are too big. Any huge project is from the get-go at an unacceptable level of risk. Decentralization, Deconstruction is powerful. The projects must be broken into smaller pieces to be managed.

It seems like you're stretching to blame PMs, and I don't think it's deserved.

The sales guy's job isn't just to sell the biggest deal - it's to sell the biggest deal that the company can actually execute and deliver.

And it's no secret that there at least as many incompetent devs as there are smart, amazing devs. There's a reason "fizzbuzz" is a weed out question.

Likewise, there are good and bad PMs.

Without all of the information it's impossible to place blame on a single group of employees.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#258

Post by watertom »

I was the CTO for a large, Fortune 100 company.

When I first took over the position I would regularly get requirements documents for internal projects that were 3 or 400 hundred pages. The project team would dutifully carry out the requirements gathering process, everything was meticulously documented, with data flows, and process maps, etc. Everything was a requirement, everything was mandatory, and everything had top priority. IT wasn’t allowed to say no, wasn’t allowed to criticize the business, and wasn’t allowed to analyze if what was specified made sense, or would work. Luckily I had a CIO that was willing to back me up when I started rejecting these projects. In six months I killed 10 or so very large multi-year software projects, in each case the business was forced to use existing software and change their processes. We saved a ton of money and implemented the solutions in weeks not years. These would have projects that had upwards of 100 people for 2, 3 or 4 years, all because the business wanted a “perfect” process.

I consulted to the DoD and this pattern was repeated, over and over again.

Most assuredly this is what happened with the Canadian Payroll project. Every person had their pet requirement(s) and they made sure they “got them in”, in the end the system that was specified couldn’t be built.

It’s like SAP, if you buy SAP take it out of the box and adapt your processes to SAP, if you do you’ll have great experience. If you try to customize SAP, it going to get very expensive, you’ll have a ton of problems, you won’t be able to take upgrades and hou’ll Need a ton of staff just go try to keep it running day in and day out.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#259

Post by tyingq »

watertom wrote: post #258

I was the CTO for a large, Fortune 100 company. When I first took over the position I would regularly get requirements documents for internal projects that were 3 or 400 hundred pages. The project team would dutifully carry out the requirements gathering process, everything was meticulously documented, with data flows, and process maps, etc. Everything was a requirement, everything was mandatory, and everything had t…

Been there, and truly understand.

Re: Canada to Scrap IBM Payroll Plan Gone Awry Costing $1B

#260

Post by wallace_f »

dpweb wrote: post #210
dctoedt wrote: post #38

Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128…

Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company. But after selling the deal, their workers have to solve a difficult engineering and organizational management problem…

Why are the sales guys necessarily competent in seeing that a project adds value? They only need to sell it.
Post reply on HN