Joint cognitive diagnostic modeling for probabilistic attributes incorporating item responses and response times

doi:10.3724/SP.J.1041.2023.01573

Abstract

Abstract:

Compared with the conventional CDM with deterministic or binary attributes, the CDM with probabilistic attributes (probabilistic-CDM) can achieve a more refined diagnosis of attribute mastery status, which helps distinguish individual differences between students and provides more reference information for teacher feedback. However, existing probabilistic CDMs can only analyze a single modal of data—item response accuracy (RA), ignoring other modals of data such as item response times (RTs). RTs reflect the cognitive processing speed of the participant. With the popularity of computerized testing, recording RT data has become routine. However, how to use RTs in probabilistic CDM to further improve parameter estimation accuracy and enrich the diagnostic feedback information is still an unsolved methodological problem. To this end, the current study proposes three joint probabilistic CDMs based on the joint-hierarchical and joint-cross-loading cognitive diagnostic modeling approaches (see Figure 1).

First, based on joint-hierarchical modeling, the joint-hierarchical probabilistic CDM (JRT-PINC) was proposed in Study 1, see Figure 1(a), which achieved the purpose of using RT to improve diagnostic accuracy. A simulation study was conducted to investigate the psychometric performance of the JRT-PINC under various simulated testing conditions, in which three independent variables, including sample size, test length, and the correlation between person parameters, were manipulated. Second, two joint-cross-loading probabilistic CDMs (CJRT-PINC-θ and CJRT-PINC-m) were proposed based on the joint-cross-loading modeling, see Figure 1(b) and 1(c). In contrast to the JRT-PINC model, two CJRT-PINC models directly used RTs to provide information for latent abilities or attributes by introducing item-level cross-loading parameters. Two CJRT-PINC models released some conditional independence assumptions in JRT-PINC, increasing their application scope. Two simulation studies were conducted to explore their performance under different simulated conditions with different degrees of cross-loading. Third, Study 3 aims to explore the relative merits of the JRT-PINC and two CJRT-PINC models, that is, the necessity of considering cross-loading in the joint analysis of RA and RT. Finally, an empirical example was conducted to illustrate the practical applicability of the proposed models and to compare them with existing CDMs (e.g., CDMs with deterministic attributes).

The simulation results mainly indicated that: (1) all three proposed models can be well recovered under different simulated conditions (see Tables 1-6); (2) CJRT-PINC-θ makes fuller use of the information contained in RTs and thus improves the accuracy of the parameter estimation of the core constructs (e.g., latent ability and attributes) than CJRT-PINC-m; and (3) the adverse effects of ignoring the possible cross-loadings are more severe than redundantly considering them (see Table 7). The results of the empirical example indicated that: (1) probabilistic attributes provide more refined feedback on participants' mastery of attributes than deterministic attributes; and (2) two CJRT-PINC models fit this data better than the JRT-PINC model (see Table 8).

Overall, this paper introduced RTs in probabilistic CDM for the first time and proposed three joint probabilistic CDMs based on two joint cognitive diagnostic modeling approaches. This study enriched the scope of application of probabilistic CDMS and provided methodological guidance for further refined and comprehensive diagnosis by jointly analyzing multi-modal data in technology-enhanced assessment systems.

TIAN Yashu, ZHAN Peida, WANG Lijun. (2023). Joint cognitive diagnostic modeling for probabilistic attributes incorporating item responses and response times. Acta Psychologica Sinica, 55(9), 1573-1586.

N	I	ρ_θτ	JRT-PINC						HO-PINC
			θ			τ			θ
						Bias	RMSE	Cor	Bias	RMSE	Cor	Bias	RMSE	Cor
200	15	−0.5	−0.001	0.458	0.887	0.000	0.121	0.896	−0.002	0.479	0.875
		−0.3	−0.003	0.468	0.881	0.001	0.122	0.896	−0.002	0.480	0.875
		0.0	−0.003	0.473	0.879	−0.000	0.122	0.949	−0.004	0.480	0.875
		0.3	−0.001	0.469	0.881	−0.001	0.121	0.899	−0.003	0.480	0.874
		0.5	−0.000	0.458	0.887	−0.000	0.120	0.902	−0.001	0.483	0.874
	30	−0.5	−0.001	0.397	0.915	−0.000	0.085	0.953	−0.004	0.412	0.908
		−0.3	0.000	0.403	0.913	−0.001	0.086	0.953	−0.004	0.411	0.909
		0.0	−0.002	0.405	0.911	−0.000	0.086	0.975	−0.004	0.413	0.908
		0.3	−0.002	0.397	0.915	−0.002	0.086	0.954	−0.004	0.410	0.909
		0.5	−0.003	0.389	0.919	−0.001	0.085	0.953	−0.005	0.411	0.909
500	15	−0.5	−0.000	0.456	0.888	−0.000	0.121	0.902	0.000	0.476	0.876
		−0.3	0.000	0.467	0.881	−0.001	0.122	0.901	0.000	0.477	0.876
		0.0	−0.000	0.473	0.878	−0.000	0.122	0.948	−0.001	0.476	0.875
		0.3	−0.001	0.470	0.880	−0.001	0.122	0.901	−0.001	0.478	0.875
		0.5	−0.000	0.460	0.885	−0.001	0.121	0.903	−0.001	0.478	0.876
	30	−0.5	0.000	0.391	0.918	−0.000	0.088	0.948	0.000	0.407	0.911
		−0.3	0.002	0.400	0.914	−0.001	0.088	0.948	0.000	0.407	0.911
		0.0	−0.001	0.402	0.913	−0.000	0.089	0.973	0.000	0.405	0.912
		0.3	0.000	0.397	0.915	−0.001	0.088	0.948	0.000	0.405	0.912
		0.5	0.001	0.387	0.920	0.000	0.088	0.948	0.000	0.405	0.912

N	I	ρ_θτ	JRT-PINC						HO-PINC
			θ			τ			θ
						Bias	RMSE	Cor	Bias	RMSE	Cor	Bias	RMSE	Cor
200	15	−0.5	−0.001	0.458	0.887	0.000	0.121	0.896	−0.002	0.479	0.875
		−0.3	−0.003	0.468	0.881	0.001	0.122	0.896	−0.002	0.480	0.875
		0.0	−0.003	0.473	0.879	−0.000	0.122	0.949	−0.004	0.480	0.875
		0.3	−0.001	0.469	0.881	−0.001	0.121	0.899	−0.003	0.480	0.874
		0.5	−0.000	0.458	0.887	−0.000	0.120	0.902	−0.001	0.483	0.874
	30	−0.5	−0.001	0.397	0.915	−0.000	0.085	0.953	−0.004	0.412	0.908
		−0.3	0.000	0.403	0.913	−0.001	0.086	0.953	−0.004	0.411	0.909
		0.0	−0.002	0.405	0.911	−0.000	0.086	0.975	−0.004	0.413	0.908
		0.3	−0.002	0.397	0.915	−0.002	0.086	0.954	−0.004	0.410	0.909
		0.5	−0.003	0.389	0.919	−0.001	0.085	0.953	−0.005	0.411	0.909
500	15	−0.5	−0.000	0.456	0.888	−0.000	0.121	0.902	0.000	0.476	0.876
		−0.3	0.000	0.467	0.881	−0.001	0.122	0.901	0.000	0.477	0.876
		0.0	−0.000	0.473	0.878	−0.000	0.122	0.948	−0.001	0.476	0.875
		0.3	−0.001	0.470	0.880	−0.001	0.122	0.901	−0.001	0.478	0.875
		0.5	−0.000	0.460	0.885	−0.001	0.121	0.903	−0.001	0.478	0.876
	30	−0.5	0.000	0.391	0.918	−0.000	0.088	0.948	0.000	0.407	0.911
		−0.3	0.002	0.400	0.914	−0.001	0.088	0.948	0.000	0.407	0.911
		0.0	−0.001	0.402	0.913	−0.000	0.089	0.973	0.000	0.405	0.912
		0.3	0.000	0.397	0.915	−0.001	0.088	0.948	0.000	0.405	0.912
		0.5	0.001	0.387	0.920	0.000	0.088	0.948	0.000	0.405	0.912

N	I	ρ_θτ	RMSE										Cor
			JRT-PINC					HO-PINC					JRT-PINC					HO-PINC
						m₁	m₂	m₃	m₄	m₅	m₁	m₂	m₃	m₄	m₅	m₁	m₂	m₃	m₄	m₅	m₁	m₂	m₃	m₄	m₅
200	15	−0.5	0.129	0.137	0.139	0.145	0.139	0.155	0.152	0.152	0.155	0.158	0.899	0.901	0.896	0.888	0.875	0.888	0.890	0.886	0.877	0.861
		−0.3	0.130	0.136	0.144	0.150	0.141	0.158	0.148	0.153	0.158	0.160	0.895	0.897	0.892	0.883	0.869	0.887	0.891	0.885	0.876	0.860
		0	0.132	0.137	0.144	0.148	0.141	0.161	0.147	0.155	0.157	0.157	0.894	0.896	0.890	0.881	0.867	0.886	0.891	0.885	0.876	0.861
		0.3	0.127	0.138	0.145	0.151	0.143	0.154	0.148	0.159	0.158	0.157	0.896	0.896	0.890	0.882	0.869	0.887	0.891	0.884	0.876	0.862
		0.5	0.128	0.139	0.144	0.147	0.142	0.159	0.151	0.158	0.157	0.161	0.898	0.899	0.894	0.886	0.874	0.884	0.889	0.883	0.874	0.860
	30	−0.5	0.097	0.108	0.124	0.127	0.133	0.117	0.125	0.142	0.144	0.151	0.946	0.940	0.928	0.912	0.890	0.940	0.933	0.920	0.904	0.881
		−0.3	0.100	0.110	0.126	0.128	0.132	0.120	0.124	0.142	0.143	0.148	0.945	0.939	0.927	0.911	0.887	0.940	0.933	0.921	0.905	0.882
		0	0.103	0.110	0.128	0.130	0.134	0.122	0.126	0.142	0.148	0.153	0.944	0.938	0.925	0.909	0.885	0.939	0.933	0.920	0.903	0.879
		0.3	0.097	0.110	0.123	0.131	0.129	0.121	0.126	0.140	0.148	0.150	0.946	0.939	0.928	0.912	0.890	0.940	0.933	0.921	0.905	0.881
		0.5	0.096	0.110	0.124	0.127	0.126	0.120	0.127	0.143	0.144	0.148	0.947	0.940	0.929	0.915	0.894	0.940	0.933	0.921	0.905	0.881
500	15	−0.5	0.111	0.131	0.130	0.139	0.139	0.132	0.140	0.141	0.145	0.160	0.906	0.905	0.901	0.890	0.874	0.896	0.896	0.892	0.879	0.861
		−0.3	0.113	0.132	0.134	0.140	0.142	0.132	0.140	0.141	0.143	0.163	0.902	0.901	0.896	0.885	0.867	0.896	0.895	0.892	0.879	0.861
		0	0.113	0.135	0.135	0.142	0.144	0.133	0.140	0.140	0.145	0.165	0.900	0.898	0.894	0.883	0.864	0.895	0.895	0.892	0.880	0.861
		0.3	0.114	0.136	0.134	0.142	0.146	0.132	0.143	0.140	0.145	0.165	0.900	0.899	0.895	0.884	0.866	0.894	0.894	0.891	0.879	0.860
		0.5	0.111	0.134	0.132	0.139	0.140	0.131	0.140	0.141	0.144	0.163	0.904	0.902	0.900	0.889	0.872	0.894	0.895	0.891	0.879	0.860
	30	−0.5	0.088	0.098	0.107	0.119	0.125	0.101	0.107	0.121	0.126	0.145	0.949	0.943	0.932	0.916	0.895	0.945	0.938	0.926	0.908	0.884
		−0.3	0.087	0.101	0.109	0.121	0.128	0.100	0.109	0.122	0.126	0.145	0.947	0.941	0.929	0.913	0.890	0.944	0.938	0.925	0.908	0.884
		0	0.090	0.100	0.109	0.121	0.128	0.101	0.108	0.121	0.125	0.144	0.947	0.941	0.929	0.912	0.889	0.945	0.939	0.926	0.910	0.886
		0.3	0.089	0.098	0.109	0.120	0.126	0.100	0.105	0.123	0.125	0.145	0.948	0.942	0.930	0.915	0.892	0.945	0.939	0.926	0.910	0.886
		0.5	0.089	0.095	0.107	0.116	0.123	0.100	0.105	0.122	0.125	0.141	0.949	0.944	0.933	0.919	0.898	0.945	0.939	0.927	0.910	0.886

N	I	ρ_θτ	RMSE										Cor
			JRT-PINC					HO-PINC					JRT-PINC					HO-PINC
						m₁	m₂	m₃	m₄	m₅	m₁	m₂	m₃	m₄	m₅	m₁	m₂	m₃	m₄	m₅	m₁	m₂	m₃	m₄	m₅
200	15	−0.5	0.129	0.137	0.139	0.145	0.139	0.155	0.152	0.152	0.155	0.158	0.899	0.901	0.896	0.888	0.875	0.888	0.890	0.886	0.877	0.861
		−0.3	0.130	0.136	0.144	0.150	0.141	0.158	0.148	0.153	0.158	0.160	0.895	0.897	0.892	0.883	0.869	0.887	0.891	0.885	0.876	0.860
		0	0.132	0.137	0.144	0.148	0.141	0.161	0.147	0.155	0.157	0.157	0.894	0.896	0.890	0.881	0.867	0.886	0.891	0.885	0.876	0.861
		0.3	0.127	0.138	0.145	0.151	0.143	0.154	0.148	0.159	0.158	0.157	0.896	0.896	0.890	0.882	0.869	0.887	0.891	0.884	0.876	0.862
		0.5	0.128	0.139	0.144	0.147	0.142	0.159	0.151	0.158	0.157	0.161	0.898	0.899	0.894	0.886	0.874	0.884	0.889	0.883	0.874	0.860
	30	−0.5	0.097	0.108	0.124	0.127	0.133	0.117	0.125	0.142	0.144	0.151	0.946	0.940	0.928	0.912	0.890	0.940	0.933	0.920	0.904	0.881
		−0.3	0.100	0.110	0.126	0.128	0.132	0.120	0.124	0.142	0.143	0.148	0.945	0.939	0.927	0.911	0.887	0.940	0.933	0.921	0.905	0.882
		0	0.103	0.110	0.128	0.130	0.134	0.122	0.126	0.142	0.148	0.153	0.944	0.938	0.925	0.909	0.885	0.939	0.933	0.920	0.903	0.879
		0.3	0.097	0.110	0.123	0.131	0.129	0.121	0.126	0.140	0.148	0.150	0.946	0.939	0.928	0.912	0.890	0.940	0.933	0.921	0.905	0.881
		0.5	0.096	0.110	0.124	0.127	0.126	0.120	0.127	0.143	0.144	0.148	0.947	0.940	0.929	0.915	0.894	0.940	0.933	0.921	0.905	0.881
500	15	−0.5	0.111	0.131	0.130	0.139	0.139	0.132	0.140	0.141	0.145	0.160	0.906	0.905	0.901	0.890	0.874	0.896	0.896	0.892	0.879	0.861
		−0.3	0.113	0.132	0.134	0.140	0.142	0.132	0.140	0.141	0.143	0.163	0.902	0.901	0.896	0.885	0.867	0.896	0.895	0.892	0.879	0.861
		0	0.113	0.135	0.135	0.142	0.144	0.133	0.140	0.140	0.145	0.165	0.900	0.898	0.894	0.883	0.864	0.895	0.895	0.892	0.880	0.861
		0.3	0.114	0.136	0.134	0.142	0.146	0.132	0.143	0.140	0.145	0.165	0.900	0.899	0.895	0.884	0.866	0.894	0.894	0.891	0.879	0.860
		0.5	0.111	0.134	0.132	0.139	0.140	0.131	0.140	0.141	0.144	0.163	0.904	0.902	0.900	0.889	0.872	0.894	0.895	0.891	0.879	0.860
	30	−0.5	0.088	0.098	0.107	0.119	0.125	0.101	0.107	0.121	0.126	0.145	0.949	0.943	0.932	0.916	0.895	0.945	0.938	0.926	0.908	0.884
		−0.3	0.087	0.101	0.109	0.121	0.128	0.100	0.109	0.122	0.126	0.145	0.947	0.941	0.929	0.913	0.890	0.944	0.938	0.925	0.908	0.884
		0	0.090	0.100	0.109	0.121	0.128	0.101	0.108	0.121	0.125	0.144	0.947	0.941	0.929	0.912	0.889	0.945	0.939	0.926	0.910	0.886
		0.3	0.089	0.098	0.109	0.120	0.126	0.100	0.105	0.123	0.125	0.145	0.948	0.942	0.930	0.915	0.892	0.945	0.939	0.926	0.910	0.886
		0.5	0.089	0.095	0.107	0.116	0.123	0.100	0.105	0.122	0.125	0.141	0.949	0.944	0.933	0.919	0.898	0.945	0.939	0.927	0.910	0.886

Analysis model	μ_φ	θ			τ
Analysis model	μ_φ	Bias	RMSE	Cor	Bias	RMSE	Cor
CJRT-PINC-θ	0.1	−0.006	0.295	0.958	0.000	0.138	0.969
CJRT-PINC-θ	0.5	−0.007	0.287	0.958	−0.001	0.194	0.971
HO-PINC	0.1	−0.007	0.481	0.871	-	-	-
HO-PINC	0.5	−0.008	0.503	0.864	-	-	-

Joint cognitive diagnostic modeling for probabilistic attributes incorporating item responses and response times

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References

Related Articles 0

Recommended Articles

Metrics

Comments

Analysis model	μ_φ	Index	m₁	m₂	m₃	m₄	m₅
CJRT-PINC-m	0.1	Bias	0.006	−0.014	−0.021	0.012	−0.026
		RMSE	0.129	0.131	0.140	0.154	0.144
		Cor	0.904	0.905	0.901	0.888	0.873
	0.5	Bias	0.012	−0.010	0.002	−0.009	−0.005
		RMSE	0.127	0.124	0.132	0.135	0.135
		Cor	0.907	0.910	0.906	0.897	0.882
HO-PINC	0.1	Bias	0.040	0.016	−0.006	−0.009	−0.054
		RMSE	0.153	0.145	0.149	0.154	0.159
		Cor	0.891	0.894	0.888	0.877	0.861
	0.5	Bias	0.038	0.012	−0.006	−0.021	−0.065
		RMSE	0.154	0.148	0.154	0.156	0.159
		Cor	0.889	0.892	0.886	0.876	0.859

Data generation model	Analysis model	DIC	ppp.RA	ppp.RT
JRT-PINC	JRT-PINC	7935.189	0.479	0.790
	CJRT-PINC-θ	7935.783	0.499	0.792
	CJRT-PINC-m	7947.637	0.502	0.792
CJRT-PINC-θ	JRT-PINC	8979.986	0.505	0.706
	CJRT-PINC-θ	7978.639	0.517	0.790
	CJRT-PINC-m	8087.316	0.433	0.784
CJRT-PINC-m	JRT-PINC	8046.958	0.485	0.781
	CJRT-PINC-θ	7952.253	0.503	0.782
	CJRT-PINC-m	7952.049	0.506	0.791

Index	Analysis model
Index	HO-PINC	JRT-DINA	JRT-PINC	CJRT-PINC-θ	CJRT-PINC-m
DIC	17410	50090	41772	41186	40746
ppp.RA	0.728	0.611	0.571	0.638	0.404
ppp.RT	—	0.594	0.596	0.604	0.603