There's a problem with language productivity comparisons that Armstrong mentions: it's impossible to write the same program in two different languages. If you have the same team write it, their second try will benefit from everything they learned the first time, and that is so huge a part of programming that it is sure to distort the outcome and may even dwarf any language effects. But if you use different teams instead, you've traded one confounding variable for another—the effect of switching teams—which is also hugely influential. Thus it's impossible to do an apples-to-apples comparison, and most such experiments deserve high skepticism. It's too easy to consciously or unconsicously engineer the outcome you expect, which is presumably why we nearly always hear that the experimenter's pet language won the day. Has the experimenter's pet language ever
not won the day?
That makes me think of a more modest way to do these experiments that might return more reliable results: use the same team twice, but have them solve the problem in their favorite language first. That is, if A is the pet language and you want to compare A to B, write the program first in A and then in B. This biases the test in B's favour, because A will get penalized for all the time it took to learn about the problem while B will get all that benefit for free. Since there's already a major bias in favor of A, this levels the playing field some.
Here's why I think this might be more reliable. If you run the experiment this way and A comes out much better, you now have an answer to the charge that the second time was easier: all that benefit went to B and B still lost. Conversely, if A doesn't come out much better, you now have evidence that the language effect isn't so great once you account for the learning effect.