关键词: 计算机化分类测验, 终止规则, 似然比, 随机缩减, 贝叶斯决策理论

Abstract:

Computerized classification testing (CCT) can adaptively classify test-takers into two or more different categories, and it has been widely used in qualifying tests and clinical psychology or medical diagnosis. As an essential part of CCT, the termination rule determines when the test is to be stopped and to which category the test-taker is ultimately classified into, directly affecting the test efficiency and classification accuracy. According to the theoretical basis of the termination rules, existing rules can be roughly divided into the likelihood ratio, Bayesian decision theory, and confidence interval rules. And their core ideas are constructing hypothesis tests, designing loss functions, and comparing the relative positions of confidence intervals, respectively. At the same time, when constructing specific termination rules, the requirement of different test scenarios (e.g., the number of categories and the number of tests’ dimensions) should also be considered.
There are advantages and disadvantages to each of the three types of termination rules. Specifically, the likelihood ratio rule is based on the likelihood ratio test, with better theoretical properties. However, the method requires prior determination of the indifference interval and the type I and II error rates, introducing the impact of subjective factors. Also, it is more challenging to extend the method in complex test situations, such as multidimensional and multicategory CCT. Bayesian decision theory rules make classification decisions based on the loss function. It can dynamically optimize the decision from a more global perspective since it works backward from the final stage of the test. In addition, the variety of loss functions makes the method very flexible in form and makes it easy to be applied to different test situations. However, in practice, the flexibility will inevitably result in the uncertainty of the choice of loss function, and the inappropriate loss function may be biased. The confidence interval method is the most straightforward because of its relatively simple principle and low computational effort. However, this method is less robust and has a relatively low test efficiency.
Currently, CCT is mainly applied in eligibility tests and clinical medicine questionnaires. In eligibility tests, all three types of termination rules have the potential to be widely applied. However, in practice, the principles of the likelihood ratio rule and the Bayesian decision theory rule are not easily understood by the general public, and these methods are also accompanied by the problem of over-exposure of items for their preference of cut-point based item selection methods. Therefore, the confidence interval rule, which is relatively simple in principle and has alleviated item exposure, has been widely used in existing qualifying tests. Bayesian decision theory rules are more applicable in clinical questionnaires because of their finer control over various classification losses.
The following can be considered for future research on CCT termination rules. First, Bayesian decision theory rules can be improved by considering non-statistical constraints with the help of the flexibility of its loss function. Second, termination rules can be developed for multidimensional and multicategory CCT to meet more practical needs. Third, termination rules that integrate response time can be developed to improve test efficiency and classification accuracy. Fourth, it is possible to construct termination rules under the framework of machine learning.

Key words: computerized classification testing, termination rule, likelihood radio, stochastic curtailment, Bayesian decision theory