Do test takers differ in ability?

Psychometrics deals with individual differences. Can we test whether test takers differ in ability to begin with? The open-source software package dexter, that we developed with two friends at Cito, has a function called 𝐢𝐧𝐝𝐢𝐯𝐢𝐝𝐮𝐚𝐥_𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞𝐬 that allows you to do precisely this. If no differences in ability are found, observed differences in test scores are random. An IRT model will fit perfectly, yet we measure nothing.

Under the hood, the function works by estimating a single ability for all persons. Dexter’s documentation does not mention that this adds an important possibility to 𝐬𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐬𝐞𝐭𝐭𝐢𝐧𝐠 that is not available elsewhere. To wit, we can test whether judges have actually been able to generate passing scores for one, and only one, (border-line) candidate. If so, observed discrepancies between judges are as expected. Otherwise, the judges either agree too much, or too little.

Wish to know more? Contact us at info@metior.consulting