We've invented waterfall

Here's what I would like to see from a consulting company for once. Don't interview the stake holders of the company, they don't use the software like their employees do. They have a high level overview based on feedback from their employees. When you interview them, you're getting second hand information that will likely be missing key points that they forgot to mention or just simply think is obvious and doesn't require mentioning.

So here's what you should do. Over the shoulder review of a representative employee of each department. HR, AP, AR etc. Actively watch and understand the processes they go through. FOR AN ENTIRE 8 HOUR SHIFT.

No, seriously.

Now, you've got a good idea of how its done current.

Then do the stake holder interview to find out what they want to do differently.

Then plan, develop and deploy your software.

I seriously don't get why despite there being a recognized need for understanding the problem scope which everyone one will invest hours in meetings and conference calls talking about, no one actually invests any (significant) time doing, or reviewing the actual process. It's the fastest most efficient way to understand the problem. It would totally diminish months of back and forth "here's what you asked for" "no, that's not what I asked for" development review process that almost _always_ occurs with consultant projects.

/rant

I wondered who the man with the clipboard was, micro-managing every step of the process.

Say what you will about large corporations but at the very least they can get this right. I'm working for a 4,000+ company now and we take these steps very seriously during our Engineering design. Also part of the job is working with vendors (Oracle, HP, others) who send people on sight from around the world to work with us (the client) and properly interview for the right knowledge.

Call me bias, but a lot of smaller companies overlook tried and tested standards. Proper requirement elicitation was documented in IEEE Computer Society SWEBOK prior to 2004 and even that is a summary of important knowledge to Software Engineering profession. Why is it that when a new company comes along with a truly innovative concept they feel they can innovate the rest of the process as well?

> this was well known 10 years ago.

This was well known 40 years ago:

The most deadly thing in software is the concept, which almost universally seems to be followed, that you are going to specify what you are going to do, and then do it. And that is where most of our troubles come from. The projects that are called successful, have met their specifications. But those specifications were based upon the designers’ ignorance before they started the job.

--Douglas Ross, 1968 (http://homepages.cs.ncl.ac.uk/brian.randell/NATO/nato1968.PD..., p. 32)

Let me tell you why you don't see this happening more often.

I did this on a project a few years back. I replaced a paper workflow process that was taking up two people each in three departments with a web-based workflow that increased visibility, dropped turn-around time from days to minutes, increased accountability and accuracy and trimmed those 16 person hours of processing down to 1-2 per department.

Everyone who directly interacted with the new system loved it. Numerous edge cases that would have been lost in high-level review were caught and integrated from day 1 due to my actually watching people do the job for a day or two per department. The solution has been rock solid (minor maintenance only) for five years.

And I almost lost the job.

The people who sign the checks were furious. The balance of political power between departments were thrown for a loop. One head in particular treated the thing as a near-existential threat. His entire concept of his job revolved around being the authoritative interface for retrieving and maintaining pieces of data that were no longer exclusively under his control. Another flipped out because middle management saw the results as cause to reduce his headcount and budget, and thus importance.

These two departments fought for months, refusing to contribute their shares of budget that were pledged toward modernizing this system.

On a technical and practical level, it was the single best experience I've ever had as a consultant. On a personal and economic level, is was one of the worst. It was some of the hardest money I've ever tried to collect. It was some of the most time and energy I've put into the political and 'sales' side of a job (the part I treat as a necessary evil, but very much evil). The corporation has made out like a bandit in the long run. But I paid the price.

It's simply too easy and financially rewarding to allow a client's political nonsense to screw up every stage of a project. I have less stress, the people who pay me are happier and I bill far more hours.

As with most software, internally developed software included, you don't see better projects more often because the incentives are horribly perverted and stacked against it.