IBM contract cost for failure-plagued Phoenix payroll system jumped to $185M

The mess is probably on both sides. The issue is that management doesn't care about IT until everything breaks apart. This means IT is seen as cost and not worth the trouble to do well and keep in house. Then of course everything starts falling apart and it's easier to put the fault on the vendors. Of course IBM has its faults, but I'd guess they are more on the overselling side (we can do everything and cheaper and better!) than on the pure technical side.

Ive been on another IBM run project that started off very similarly for a large energy company. A few thoughts about this.

1) IBM pulled the same types of stunts on this project and also failed spectacularly. Its not in the news though, as private companies are reluctant to admit failure.

2) IBM and other big consulting companies pour large amounts of revenue and time into relationship management and networking. This keeps the "nobody ever got fired for hiring consulting shop x" fallacy alive and kicking.

3) There is never a large implementation that is "out of the box and turn key" If a consulting shop tells you this is possible without great sacrifice and compromise to business process, you should run the other direction.

4) IBM and other big consulting shops regularly oversell their specialty domain knowledge, and often end up farming it out to independent vendor/domain experts after they realize they're in over their head.

5) While I think IBM is a shell of its former self, their behavior here is big consulting industry standard.

6) Most private companies know that a project will largely come in way over budget, and often don't care as long as they can afford it. The overages are usually written off and as long as the operational risk is shifted away from the business then people are usually happy.

As an independent consultant, Im obviously biased, but my opinion comes from experience working along side with Deloitte, IBM, PWC, and some other big consulting shops. I am usually one of a few niche independent contractors that are brought in to handle specialty items.

So why does this keep happening? Most enterprises/governments do not want to place their faith in boutique consulting shops or independent consultants as this seems too risky even if they would likely be able to implement for half the money in half the time. The arbitrage for big consulting shops comes from this. Most big consulting companies are going to put 1 or 2 senior and expensive(to their own payroll) people on the project and leave the grunt work to cheaper junior onsite or offshore resources. That work is still billed at 150+ per hour and the senior folks 300+ per hour.

Payroll isnt "hard" per se, its just hard to model into a single comprehensive process. Complicating this is the fact that laws and taxes vary by employee type and location. Internal business rules vary everywhere, and even though best practices exist, there is always differences from one place to another. Finally, people get really pissed when their money aint right...

Ive noticed, anecdotally, that IBM in particular has had issues with big projects and variances. I chalk that up to being the relative newbie in the big consulting world, and they're likely underbidding to get a foothold.

This is what happens when non-technical bureaucrats want to design a system with full managerial control within the structure of a multi-level bureaucracy (meaning there isn't just one group of non-technical bureaucrats designing the system, there are 10 levels, and all want input, and all must approve of each others' inputs).

And this is why contractors love doing business with the Government of Canada. Bill, baby, bill.

So basically the problem isn't the inherent complexity of payroll systems, it is project management and process.

You're right about the accretion of ad-hoc fixes over time. Some of these might be unnecessary when analysed with hindsight and need not be faithfully reproduced in a new component or system.

It may not be simple, but it's been done before. Many, many times. That's the mind-boggling part IMHO. They're not trying to find a way to travel near light speed. They're building a payroll system.