Abstract:

In cognitive diagnostic models (CDMs), differential item functioning (DIF) refers to the probabilities of success of an item being different for examinees with the same attribute mastery pattern in the groups. The detection of DIF is an important step to ensure the fairness and validity of results from CDMs for all groups. Hou et al. (2014) proposed that the Wald statistic can be used to detect DIF in CDMs. Unfortunately, their results revealed that the Wald statistic based on the information matrix estimation method developed by de la Torre (2009, 2011) yielded inflated Type I error rates. However, Li and Wang (2015) found that the Type I error rates of the Wald statistic in which MCMC algorithms were implemented were slightly inflated in their study under the same conditions. In this study, we proposed an improved Wald statistic based on the observed information matrix for DIF assessment. As a general demonstration, we took the log-linear cognitive diagnosis model (LCDM; Henson et al., 2009) as an example. In this simulation study, in order to compare the results with previous studies (e.g., Hou et al.,2014; Li & Wang, 2015), we followed the simulation design used by Hou et al. (2014), except that we implemented the observed or cross-product (XPD) information matrix in the Wald statistic computation. Parameters set in the studies were: the test length at 30, the number of attributes at 5, and the maximum number of required attributes for an item at 3. Binary item response data were generated from the DINA model. Three sets of true item parameter values were considered for the reference group. Two DIF sizes: .05 and .10, and two types of DIF: uniform and nonuniform, were manipulated. Two sample sizes were considered, 500 and 1,000. Each condition was replicated 1000 times, and the estimation code was written in R (R Core Team, 2015). The simulation results showed that: (1) for the relatively discriminating items, Wald statistic had accurate Type I error control when the observed information matrix was used in its computation. However, when the slip and guessing parameters were large , the Type I error control was slightly conservative. (2) When the XPD information matrix was used for the computation of the Wald statistic, the Type I error control was conservative; that is, the performance of the observed information matrix was better than the XPD information matrix. (3) The number of attributes required for success on the item did not have a notable impact on the Type I error control of Wald statistic, irrespective of whether the observed or the XPD information matrix was used for the statistic. (4) The power rates of Wald statistic for detecting DIF increased as the sample size increased. We conclude that our improved Wald statistic provided follows asymptotically a chi-square distribution with degrees of freedom equal to 2, for DINA model. The improved Wald statistic is a useful and powerful tool for DIF detection in CDMs.

Key words: Wald statistic, differential item functioning, cognitive diagnosis model, observed information matrix, cross-product information matrix