I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

Firstly, I think we should put aside the fact that it is labeled as a test of "logical reasoning": it is certainly not a test of formal logical reasoning, and an ambiguous or erroneous label does not necessarily make it a bad question (it is not necessary that it be characterized at all.)

Secondly, we are not logically obliged to accept that it has only one answer among the options presented, though if it has more or less than one while the people who posed it thought it had exactly one, that is a problem (I once was nearly expelled from a class for making this point at greater length than the instructor liked!) On the other hand, the question asks which of the candidate answers is most strongly suggested by the passage, which is not a statement that the others are false.

Here, however, it certainly has no more than one answer among the candidates: there is nothing in the passage that has any bearing on options A, B, D or E - this is perhaps most obvious in the case of B, but the others are like it. In particular, with respect to D, that specific issue is not raised, and furthermore, if it was the government's opinion now that D was the case to the extent of having a bearing on the decision, there would be no need to explain its position in terms of a potential future determination that the tests are inconclusive.

C, on the other hand, is suggested by the government's explanation: if the tests were conclusive, their future refutation would not be a worry.

As I said, this is not a test of formal logic, where the government's response would not imply the possibility of future refutation. Nevertheless, to explain something on the basis of a premise that is only formally possible would be almost as much an informal fallacy as begging the question, IMHO, and one might suspect it is being offered deceitfully (a concept that has no place at all in logic.)

The sort of analysis of natural language called for here (to see what are and are not the issues being considered) is useful and important, for lawyers and the rest of us, and it is, as I have set out above, more objective than "makes sense" or "sounds plausible." If people were more practiced in analytical reading, then corporations, governments and other organizations would less easily get away with blatant non-sequiturs in their explanations of their positions and actions ("there is no evidence the attackers took any personal or confidential information"...)

It makes sense as a robotically developed null hypothesis, but it doesn't make sense in the real world.

The paper reports on the results of a 'logic' test administered to undergrads and uses this to define competence. It's a key part of their evidence their effect is real.

> It's about people's self-perception in relation to a task at which they are, or are not, competent. That task could be juggling watermelons or strangling geese for all it matters.

The specific tasks matter a great deal.

The whole paper relies very heavily on the following assumption: DK can accurately and precisely tell the difference between competence and lack of competence. In other words, that they know the right answers to the questions they're asking their undergrads.

In theory this isn't a difficult bar to meet. They work at a school and schools do standardized testing on a routine basis. There are lots of difficult tasks for which there are objectively correct and incorrect answers, like a maths test.

But when we read their paper, the first two tasks they chose aren't replicable, meaning we can't verify DK actually knew the right answers. Plus the first task is literally a joke. There isn't even a right answer to the question to begin with, so their definition of "competence" is meaningless. The other tasks might or might not have right answers that DK correctly selected, but we can't verify that for ourselves (OK, I didn't check their grammar test but given the other two are unverifiable why bother).

That's a problem because the DK effect could appear in another situation they didn't consider: what if DK don't actually know the right answers to their questions but their students do. If this occurs then what you'd see is this: some students would answer with the "wrong" (right) answers and rate their own confidence highly, because they know their answer is correct and don't realize the professors disagree. Other students might realize that the professors are expecting a different answer and put down the "right" (wrong) answer, but they'd know they were playing a dangerous game and so rate their confidence as lower. That's all it would take to create the DK effect without the underlying effect actually existing. To exclude this possibility we have to be able to check that DK's answers to their own test questions are correct, but we can't verify that. Nor should we take it on faith given their dubious approach to question design.