11–20 of 20 posts
Minor adjustment. When following this kind of approach I typically define the criteria before the solutions. This reduces the chance of adding bias to your criteria.
A handy mnemonic: Meaning before metric, measure before method (2MBM)
A strength of DEA is that criteria are not weighted but that alternatives are compared against the efficient frontier (similar to risk/return in a Markowitz portfolio). DEA is very useful for benchmarking many alternatives.
AHP uses pair-wise comparison which is less prone to manipulation than scoring is but it needs a group of people that do the comparison.
[0] https://en.wikipedia.org/wiki/Data_envelopment_analysis [1] https://en.wikipedia.org/wiki/Analytic_hierarchy_process
The promise of this quantifying is that results are more objective and precise, so everyone can come to rational agreement. In practice, it doesn't do that. What attributes are chosen to quantify, how they're scored, how they're weighted—these are all subject to a great deal of fiddling and constant debate. "You over-weighted X!" or "You didn't consider Y!" Et cetera. Truly never-ending, and unless participants already well-aligned, doesn't secure genuine consensus.
Results are also highly perturbable. Tweak the weights and/or scores but a little and they tell an entirely different story. New winners emerge, clear victories become dead heats, and the former Red Lantern Award winner is suddenly in the middle of the pack.
The major problem with this approach is that you go into the analysis with an opinion as to what is the better solution, then you tweak the weights and scores to match that opinion. This is not data-driven or even rational, it's just a way to express "numerically" what your guts tell you. Clearly someone who disagrees with you will just tell you you got the weights and scores wrong. The main value of this exercise is…
Aka, the I.T. House of Quality!!!!
https://japan.irca.org/dcms_media/other/House%20of%20Quality...
This is an ancient technique. I used essentially the same process ~30 years ago (1987–95) to publish N-way product comparisons. The approach was old even then (see e.g. https://en.wikipedia.org/wiki/Quality_function_deployment ). The promise of this quantifying is that results are more objective and precise, so everyone can come to rational agreement. In practice, it doesn't do that. What attributes are chosen to qua…
This looks like a good way to pick something everyone will feel good about, but not necessarily the best strategy. Pretty much quantitate way to design by committee
That's a great point. I suspect though, the critique applies to decision making by consensus more broadly. Decisions made by consensus are not necessarily the optimal strategy (as defined by efficacy to achieve targets). This method simply reflects that. However, the benefit of this method is that all participants are forced to be more rigorous about why they hold certain beliefs, and then making an attempt to aggreg…
I’m curious whether it can be modified to find global maximum, but avoid McNamara fallacy
I guess the main issue will be the quality of input data
This is an ancient technique. I used essentially the same process ~30 years ago (1987–95) to publish N-way product comparisons. The approach was old even then (see e.g. https://en.wikipedia.org/wiki/Quality_function_deployment ). The promise of this quantifying is that results are more objective and precise, so everyone can come to rational agreement. In practice, it doesn't do that. What attributes are chosen to qua…
Then, I created two conversations. One was about our confidence on those estimated ratings together, and the second was the proposed weights and whose they were, which I revealed after so we could add up what the model provided and have a much faster discussion about whether it yielded what we ought to do.
Nobody wants to be subject to process, but almost everyone wants to appeal to it to get their way, so I wouldn't use this tool to make decisions themselves, but instead to make higher quality ones on teams faster.
This is an ancient technique. I used essentially the same process ~30 years ago (1987–95) to publish N-way product comparisons. The approach was old even then (see e.g. https://en.wikipedia.org/wiki/Quality_function_deployment ). The promise of this quantifying is that results are more objective and precise, so everyone can come to rational agreement. In practice, it doesn't do that. What attributes are chosen to qua…
Something else to consider is that weights scaling linearly may not make sense.
More importantly, criteria may overlap. In the example in the article, I suspect technical ease and scaling ease are highly correlated, which effectively means you're double counting.
Basically the technique only works when you have fully independent criteria which cover the full spectrum of what matters and which can be weighted objectively using a scale that represents true relative importance.
This is an ancient technique. I used essentially the same process ~30 years ago (1987–95) to publish N-way product comparisons. The approach was old even then (see e.g. https://en.wikipedia.org/wiki/Quality_function_deployment ). The promise of this quantifying is that results are more objective and precise, so everyone can come to rational agreement. In practice, it doesn't do that. What attributes are chosen to qua…
what's a better technique that has worked for you?