ISSN 1671-3710
CN 11-4766/R

心理科学进展 ›› 2022, Vol. 30 ›› Issue (3): 522-535.doi: 10.3724/SP.J.1042.2022.00522

• 研究方法 • 上一篇    下一篇


刘耀辉1, 徐慧颖1, 陈琦鹏1, 詹沛达1,2()   

  1. 1浙江师范大学教师教育学院心理学系
    2浙江省智能教育技术与应用重点实验室, 金华 321004
  • 收稿日期:2021-07-08 出版日期:2022-03-15 发布日期:2022-01-25
  • 通讯作者: 詹沛达
  • 基金资助:

The measurement of problem-solving competence using process data

LIU Yaohui1, XU Huiying1, CHEN Qipeng1, ZHAN Peida1,2()   

  1. 1Department of Psychology, College of Teacher Education, Zhejiang Normal University, Jinhua 321004, China
    2Key Laboratory of Intelligent Education Technology and Application of Zhejiang Province, Zhejiang Normal University, Jinhua 321004, China
  • Received:2021-07-08 Online:2022-03-15 Published:2022-01-25
  • Contact: ZHAN Peida


问题解决能力是指在没有明显解决方法的情况下个体从事认知加工以理解和解决问题情境的能力。对问题解决能力的测量需要借助相对更复杂、更真实、具有可交互性的问题情境来诱导问题解决行为的呈现。使用虚拟测评抓取问题解决的过程数据并分析其中所蕴含的潜在信息是当前心理计量学中测量问题解决能力的新趋势。首先, 回顾问题解决能力测量方式的发展:从纸笔测验到虚拟测评。然后, 总结对比两类过程数据的分析方法:统计建模法和数据挖掘法。最后, 从非认知因素的影响、多模态数据的利用、问题解决能力发展的测量、其他高阶思维能力的测量和问题解决能力概念及结构的界定五个方面展望未来可能的研究方向。

关键词: 问题解决能力, 过程数据, 虚拟测评, 计算机化测验, 高阶思维能力


Problem-solving competence is an individual’s capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. The measurement of problem-solving competence requires the use of relatively more complex and real problem situations to induce the presentation of problem-solving behaviors. This brings challenges to both the measurement methods of problem-solving competence and the corresponding data analysis methods. Using virtual assessments to capture the process data in problem-solving and mining the potential information contained therein is a new trend in measuring problem-solving competence in psychometrics.

Firstly, this paper reviews the development of measurement methods from pen-and-paper tests to virtual assessments. Compared with the traditional paper-and-pencil test, modern virtual assessments are not only conducive to simulating real problem situations, improving the ecological validity of the test, but also can record the process data generated by individuals in the process of problem-solving. Process data refers to man-machine or man-human interaction data with timestamps that can reflect the process of individual problem-solving. It records the detailed steps of individual problem solving and reflects the strategy and cognitive process of individual problem-solving. However, it is not easy to adopt effective methods to analyze process data.

Secondly, two methods of analyzing process data are summarized and compared: data mining methods and statistical modeling methods. Data mining is the process of using algorithms to uncover new relationships, trends, and patterns from big data. It is a bottom-up, data-driven research method that focuses on describing and summarizing data. Its advantage is that it can use existing algorithms to analyze a variety of process data at the same time, screen out variables related to individual problem-solving competence, and realize the classification of individual problem-solving competence. But sometimes, different algorithms could get different conclusions based on the same data, which leads to part of the results can not be explained. This method can not construct variables that can reflect the individual's latent trait, either. Statistical modeling method mainly refers to the method of analyzing data by using the idea of artificial modeling. It is a top-down, theory-driven approach. In statistical modeling, function models are generally constructed based on theoretical assumptions, and the observed variables are assumed to be randomly generated by the probability law expressed by the model. For the data recorded by virtual assessments, the existing modeling methods can be divided into three categories: psychometric joint modeling, hidden Markov modeling, and multi-level modeling. The main advantage of statistical modeling is that its results are easy to interpret and conform to the general process of psychological and educational research. Its limitation lies in that the modeling logic has not been unified yet because different types of process data need to be modeled separately. However, by giving full play to the advantages of the two data analysis methods, different problems in psychological and educational assessments can be dealt with. The interpretability of the results is very important in psychological and educational measurements, which determines the dominant role of statistical modeling in process data analysis.

Finally, the possible future research directions are proposed from five aspects: the influence of non-cognitive factors, the use of multimodal data, the measurements of the development of problem-solving competence, the measurements of other higher-order thinking competence, and the definition of the concept and structure of problem-solving competence.

Key words: problem-solving competence, processing data, virtual assessment, computer-based assessment, higher-order thinking competence