Earlier quoted context omitted.
That's a good analogy. It's really hard to "measure knowledge," and in my humble opinion, as someone who once was a teacher, tests are one of the hardest ways of doing so.
> tests are one of the hardest ways of doing so I've never understood how someone could master a subject and yet be unable to answer any questions about it. In my experience, people who did well on tests tended to understand the topic, and the people who didn't do well made excuses.
For one, the latter are often bullshit, as TFA points out. For two, they measure all sorts of stuff aside from actual proficiency. If nothing else, unless they took all the same classes - literally the same classes - throughout their school careers to date, no two kids got the same education. As anyone who's got the vaguest training in science can tell you, that kind of uncontrolled variability in your population will destroy any validity your measure might have. And lastly, so often these standardized tests really do have hacky questions put together with hack procedures. For my part, I distinctly remember completely stumping an IQ test proctor when I was a kid (yeah, I had helicopter parents). I had 5 cards, each showing a house, with the sun and shadows in different positions. I was supposed to put them in the correct order. So, naturally, my first move was to ask if the pictures showed the north or south side of the house.
There's no standardized test that measures the kinds of reasoning skills that really matter in life, such as the kind you'd use to make an educated guess that a question is being asked by the kind of people who would assume, unwittingly, that east is always on the right hand side.