This is seriously f'd up. When I was hired by IBM years ago, they also had a test called the IPAT (Initial Programming Aptitude Test) but, guess what?, there was no programming in it! Instead, the test was designed to see if one had the basic reasoning and logic skills which would be necessary to solve a wide variety of programming-related problems -- not one stupid, specific problem in a particular language. That's…
So basically, an IQ test. I believe these run into legal grey areas because they have to be obviously relevant to the job in question, which likely leads to the tests getting progressively more domain-specific as time goes on. Programming does seem to be a field in which general intelligence is a big predictor of success. Whereas a field like management probably personality and general intelligence are more equally w…
Basically a scientific experiment, have someone else ask and grade the questions, but those results were not considered for hire/nohire. Then 6 months after a bunch of people were hired we pull the results of the test and compare to their performance reviews. If the questions were a good predictor or future success we could use them on future interviews.
The overhead of this was more than we were willing to pay so we went back to HR's allowed questions.