ISSN 0439-755X
CN 11-1911/B

Acta Psychologica Sinica ›› 2015, Vol. 47 ›› Issue (9): 1188-1198.doi: 10.3724/SP.J.1041.2015.01188

Previous Articles    

The Block Item Pocket Method to Allow Item Review in CAT

LIN Zhe1; CHEN Pin2; XIN Tao1,2   

  1. (1 Institute of Developmental Psychology, Beijing Normal University, Beijing 100875, China)
    (2 National Innovation Center for Assessment of Basic Education Quality, Beijing 100875, China)
  • Received:2014-11-06 Published:2015-09-25 Online:2015-09-25
  • Contact: XIN Tao, E-mail: xintao@bnu.edu.cn

Abstract:

Most computerized adaptive testing (CAT) do not allow examinees to review items because it will drastically decrease measurement precision and bring about extra cheating strategies (Wainer, 1993; Wise, 1996). Allowing item review is essential to make CAT comparable with traditional tests. It also matters in application. Item review enables examinees to correct mistakes due to carelessness, which can further improve the precision of ability estimation. No such option may cause some negative consequences for their overall performance especially in high-stake examinations, such as tension or anxiety (Vispoel, Henderickson, & Bleiler, 2000). Therefore, it is worth trying if allowing item review could alleviate problems mentioned at the beginning (Wise, 1996; Vispoel, 2000, 2005).

Several methods have been proposed, including the successive block method (Stocking, 1997) and the item pocket (IP) method (Han, 2013). However, both methods are limited in some ways. Stocking’s method does not allow examinees to skip items and requires a large number of blocks which may bring about some extra adverse effects because of frequent decision to go to next block. Han’s method can avoid limitations of Stocking’s. But it requires an appropriate IP size and may result in high bias in large IP size situation. The present study proposed the block item pocket (BIP) method which sets fewer but larger blocks with a proper total IP size. This method keeps advantages of Stocking’s and Han’s and overcomes their disadvantages.
Two simulation studies of two response strategies were conducted to evaluate validity of the BIP method. Item parameters were randomly drawn from uniform distribution (b ~ U (-3, 3)) and (α ~ U (0, 2)). Each examinee was administered a fixed-length CAT with 30 items. The initial item for each examinee was randomly drawn from θ ~ U (-0.5, 0.5). For the CAT administration, the Maximum Fisher Information method was adopted to select items. The interim and final scores were estimated using MLE method in most conditions. When responses were less than 5 or when all answers were correct or wrong, EAP method was adopted. Each study contained five conditions: non-review, 1 blocks IP method, 2 blocks, 3 blocks and 6 blocks BIP method. Statistics like BIAS, MAE, and RMSE were used as evaluation criteria.

Results indicated that: (1) BIP method had better estimate precision than IP method at low ability level under normal strategy; (2) When dealing with Wainer-like strategy, BIP method was far more precise than item pocket method at all ability levels; (3) As the number of blocks increased, estimate precision got closer to non-review condition. Advantages of this new method and future directions were discussed.

Key words: computerized adaptive testing, item review, item pocket method, answer change, block item pocket method