Earlier quoted context omitted.
After having read a bit, I'm a bit disappointed. While the intent goes in the direction I would like to see, the actual execution does not follow scientific principles. "Consultant collects some data, derives conclusions, and writes a book" is actually again only anecdotal. You need to think about your sample before collecting the data, otherwise your outcome is biased by who gave you a consulting gig. Consequently,…
> the actual execution does not follow scientific principles. It can't really. The practitioners are generally brought into a project under heavy non-disclosure and generally allowed to keep stats on the project as long as it is completely anonymized. This makes it hard to publish data in a scientific research mode, because it's hard to reproduce or validate independently. Most of the time the practitioners are calle…
I totally agree that consultants working from quantitative data about past projects are much better than the usual cargo cults.
However, there are a lot of dangers in the process:
- If you are a consultant in automotive embedded software, you will be contracted for these projects, so your data will be quite useless to guide startup Web application development (for example, LOC can make sense for embedded C, but not for languages like Scala or Haskell, where you can implement the same feature elegantly in 100 lines or clumsily in 1000).
- As consultant, you want to generate revenue. So there is a strong incentive to oversell how solid your insights are. There is no counter-force in place to balance that bias out.
- While scientific publication is somewhat broken, it is (in CS) a quite good quality control with respect to scientific method. The book did not undergo any quality control by other experts. So now it becomes a matter of personal trust towards the author.
Summarized, I think that practitioner's data collections and their personal reports about them are useful in some cases, but cannot replace scientific empiric research.