Hierarchical Bayesian Multi-Dimensional IRT Applied to 200k Concept Tests

We report the use of our group’s hierarchical Bayesian implementation of the Multi-dimensional Nominal Categories Model followed by standard factor rotations of the principal dimensions to obtain 29 curated sparse dimensions from a set of 203,564 (104,998 pre and 98,566 post) administrations of a multiple-choice concept test in mechanics. We emphasize our careful attention to issues common to fitting such multi-parameter models to large data sets: a novel set of filters to remove administrations from non-conscientious testees, use of Bayesian methods to avoid overfitting, selecting the best transformations to find easily identifiable sparse dimensions, and verification and pruning of these using bootstrap samples. We demonstrate that most dimensions are invariant across different demographically different samples of students as well as between pre-instruction vs post-instruction samples. Most sparse dimensions correspond to well-known misconceptions in mechanics.