Abstract:

Test security and item pool utilization rate are very important in computerized adaptive testing (CAT), especially in the high-stakes tests. Most existing methods only focus on the item exposure rate, but rarely control the test overlap rate. Way (1998) suggested that the item exposure and the test overlap rate be two indices of test security. Following this reasoning, Chen (2010) proposed an on–line version of the Sympson-Hetter procedure with general test overlap control (SHGT) that didn’t need iterative simulations. Although the SHGT method could control item exposure and general test overlap simultaneously without iterative simulations, the item pool utilization rate was not very ideal when the item exposure or test overlap rate was slightly high or the number of examinees who shared the information with another examinee was small. Thus, the test security was threatened. To address the limitation of the SHGT method, we combined the a-stratified method with the SHGT method, and proposed three new methods: SHGT_a method, SHGT_b method, and SHGT_c method. Simulation results indicated that: (1) Compared to the SHGT method, these three new methods could not only improve the item pool utilization rate, but also maintain a very high precision of ability estimate in the same experiment condition; (2) With the increase of maximum item exposure rate (rmax) and maximum general test overlap rate ( ), and the decrease of a, the precision of ability estimation increased. Compared to the SHGT method, these three new methods could maintain a higher item pool utilization rate; (3) SHGT_b and SHGT_c outperformed SHGT_a in the aspect of the precision of ability estimation when rab was high; (4) The three new methods had a good performance in respect of the precision of ability estimation; (5) Compared to the SHGT method, these three new methods could also effectively solve the problem of item exposure over-control. Some future directions of study were suggested at the end of this paper.

Key words: item exposure rate, general test overlap rate, computerized adaptive testing, a-stratified method, selection strategy