Adaptive Comparative Judgement

A Method in Search of a Theory in Need of an Instrument

Abstract

Adaptive comparative judgement (ACJ) is increasingly used in technology and design education to assess open-ended student work through holistic pairwise comparisons. Despite strong evidence of reliability, much ACJ research continues to treat the method “as it is”, focusing on algorithmic efficiency and technical feasibility while leaving the cognitive basis of judgement largely implicit. This paper proposes a Cognitive Theory of Comparative Judgment to reframe ACJ as a problem of human decision-making under constraint. The paper (i) deconstructs ACJ as a judgement environment, (ii) advances four initial postulates concerning computational constraint, heuristic strategy, algorithmic tension, and the normative outcome of disagreement, and (iii) introduces OpenACJ, an open-access, adaptable platform designed to operationalise and test these postulates through configurable choice architecture and process data capture. The implication is a shift from asking whether ACJ “works” to explaining how it works, enabling theoretically grounded improvements to validity in the assessment of complex design capability.