ISSN 0439-755X
CN 11-1911/B
主办：中国心理学会
　　　中国科学院心理研究所
出版：科学出版社

心理学报 ›› 2020, Vol. 52 ›› Issue (4): 528-540.doi: 10.3724/SP.J.1041.2020.00528

• 研究报告 • 上一篇    

计算机动态测验中问题解决过程策略的分析：多水平混合IRT模型的拓展与应用

李美娟1,2, 刘玥3, 刘红云3,4()   

  1. 1 北京教育科学研究院北京教育督导与教育评价研究中心, 北京 100036
    2 北京师范大学中国基础教育质量监测协同创新中心
    3 北京师范大学心理学部
    4 北京师范大学心理学部应用实验心理北京市重点实验室, 北京 100875
  • 发布日期:2020-02-25 出版日期:2020-04-25
  • 通讯作者: 刘红云 E-mail:hyliu@bnu.edu.cn

Analysis of the Problem-solving strategies in computer-based dynamic assessment: The extension and application of multilevel mixture IRT model

LI Meijuan1,2, LIU Yue3, LIU Hongyun3,4()   

  1. 1 Educational Supervision and Quality Assessment Research Center, Beijing Academy of Educational Sciences, Beijing 100036, China
    2 Collaborative Innovation Center of Assessment toward Basic Education Quality, Beijing Normal University, Beijing 100875, China
    3 Faculty of Psychology, Beijing Normal University, Beijing 100875, China
    4 Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing 100875, China
  • Online:2020-02-25 Published:2020-04-25
  • Contact: LIU Hongyun E-mail:hyliu@bnu.edu.cn

RichHTML

62

PDF (PC)

2184

摘要：

学生在完成计算机动态测验过程中, 会产生大量带有时间标记的过程性数据。本研究基于5个国家(地区) 3196名学生在PISA2012一道交通问题解决任务上的139990条数据, 将多水平混合IRT (MMixIRT)模型进行拓展, 用于探索问题解决过程策略的类别特点。结果表明, 该模型不仅可以基于行为序列对不同国家(地区)学生在解决问题时策略使用情况的典型特征进行分析, 还可以提供个体水平的能力估计值。拓展的MMixIRT模型可用于分析过程性数据的特征。

关键词: 计算机动态测验, 问题解决过程策略, 过程性数据, 拓展的多水平混合IRT模型

Abstract:

Problem-solving competence is defined as the capacity to engage in cognitive processing to understand and resolve problem scenarios where a solution is not obvious. Computer-based assessments usually provide an interactive environment in which students can solve a problem by choosing among a set of available actions and taking one or more steps to complete a task. All students’ actions are automatically recorded in system logs as coded and time-stamped strings. These strings are called process data. The process data have multi-level structures in which the actions are nested within a single individual and therefore they are logically interconnected. Recently, researches have focused on characterizing process data and analyzing the response strategies to solve the problem.

This study proposed an extended MMixIRT model which incorporated the multilevel structure into a mixture IRT model. It can classify latent groups at process level that have different problem solving strategies, and estimate the students’ abilities at the student level simultaneously. This model takes the accumulated response information as the specific steps at the process level and defines a more free matrix to determine the weight information used for ability estimation at the student level. Specifically, in the standard MMixIRT model, the student-level latent variables are generally obtained from the measurement results made by the process-level response variables, while students’ final responses are used to estimate their problem-solving abilities in the extended MMixIRT model.

This research applied process data recorded in one of the items (Traffic CP007Q02) of problem solving in PISA 2012. The samples were 3196 students from Canada, Hongkong-China, Shanghai-China, Singapore, and America. Based on the log file of the process record, there were 139,990 records in the final data file. It was found that (1) The model can capture different problem-solving strategies used by students at the process level, as well as provide ability estimates at the student level. (2) The model can also analyze the typical characteristics of students’ strategy in problem-solving across different countries for targeted instructional interventions.

It is concluded that the extended MMixIRT model can analyze response data at process and student levels. These analyses not only play an important role in the scoring, but also provide valuable information to psychometricians and test developers, help them to better understand what distinguishes well performing students from the ones that are not, and eventually lead to better test design.

Key words: computer-based dynamic assessment, problem-solving strategy, process data, the extended MMixIRT model

中图分类号: