基于过程数据的问题解决能力测量及数据分析方法

doi:10.3724/SP.J.1042.2022.00522

摘要/Abstract

摘要：

问题解决能力是指在没有明显解决方法的情况下个体从事认知加工以理解和解决问题情境的能力。对问题解决能力的测量需要借助相对更复杂、更真实、具有可交互性的问题情境来诱导问题解决行为的呈现。使用虚拟测评抓取问题解决的过程数据并分析其中所蕴含的潜在信息是当前心理计量学中测量问题解决能力的新趋势。首先, 回顾问题解决能力测量方式的发展：从纸笔测验到虚拟测评。然后, 总结对比两类过程数据的分析方法：统计建模法和数据挖掘法。最后, 从非认知因素的影响、多模态数据的利用、问题解决能力发展的测量、其他高阶思维能力的测量和问题解决能力概念及结构的界定五个方面展望未来可能的研究方向。

关键词: 问题解决能力, 过程数据, 虚拟测评, 计算机化测验, 高阶思维能力

Abstract:

Problem-solving competence is an individual’s capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. The measurement of problem-solving competence requires the use of relatively more complex and real problem situations to induce the presentation of problem-solving behaviors. This brings challenges to both the measurement methods of problem-solving competence and the corresponding data analysis methods. Using virtual assessments to capture the process data in problem-solving and mining the potential information contained therein is a new trend in measuring problem-solving competence in psychometrics.

Firstly, this paper reviews the development of measurement methods from pen-and-paper tests to virtual assessments. Compared with the traditional paper-and-pencil test, modern virtual assessments are not only conducive to simulating real problem situations, improving the ecological validity of the test, but also can record the process data generated by individuals in the process of problem-solving. Process data refers to man-machine or man-human interaction data with timestamps that can reflect the process of individual problem-solving. It records the detailed steps of individual problem solving and reflects the strategy and cognitive process of individual problem-solving. However, it is not easy to adopt effective methods to analyze process data.

Secondly, two methods of analyzing process data are summarized and compared: data mining methods and statistical modeling methods. Data mining is the process of using algorithms to uncover new relationships, trends, and patterns from big data. It is a bottom-up, data-driven research method that focuses on describing and summarizing data. Its advantage is that it can use existing algorithms to analyze a variety of process data at the same time, screen out variables related to individual problem-solving competence, and realize the classification of individual problem-solving competence. But sometimes, different algorithms could get different conclusions based on the same data, which leads to part of the results can not be explained. This method can not construct variables that can reflect the individual's latent trait, either. Statistical modeling method mainly refers to the method of analyzing data by using the idea of artificial modeling. It is a top-down, theory-driven approach. In statistical modeling, function models are generally constructed based on theoretical assumptions, and the observed variables are assumed to be randomly generated by the probability law expressed by the model. For the data recorded by virtual assessments, the existing modeling methods can be divided into three categories: psychometric joint modeling, hidden Markov modeling, and multi-level modeling. The main advantage of statistical modeling is that its results are easy to interpret and conform to the general process of psychological and educational research. Its limitation lies in that the modeling logic has not been unified yet because different types of process data need to be modeled separately. However, by giving full play to the advantages of the two data analysis methods, different problems in psychological and educational assessments can be dealt with. The interpretability of the results is very important in psychological and educational measurements, which determines the dominant role of statistical modeling in process data analysis.

Finally, the possible future research directions are proposed from five aspects: the influence of non-cognitive factors, the use of multimodal data, the measurements of the development of problem-solving competence, the measurements of other higher-order thinking competence, and the definition of the concept and structure of problem-solving competence.

Key words: problem-solving competence, processing data, virtual assessment, computer-based assessment, higher-order thinking competence

中图分类号:

B841

刘耀辉, 徐慧颖, 陈琦鹏, 詹沛达. (2022). 基于过程数据的问题解决能力测量及数据分析方法. 心理科学进展 , 30(3), 522-535.

LIU Yaohui, XU Huiying, CHEN Qipeng, ZHAN Peida. (2022). The measurement of problem-solving competence using process data. Advances in Psychological Science, 30(3), 522-535.

图/表 6

图1 基于虚拟测评中过程数据的问题解决能力测量

图2 Novak (1961)提出的分部作答模式(红色为正确答案)

图3 OECD (2003)问题解决框架.

图4 证据中心设计框架中的能力模型、证据模型和任务模型 (Zhao et al., 2015; Mislevy et al., 2003)

图5 循环研究过程 (Johnson & Christensen, 2014)

图6 心理与教育研究的一般过程

参考文献 84

[1]	成素梅, 荣小雪. (2003). 波普尔的证伪方法与非充分决定性论题. 自然辩证法研究, 19(1), 15-19+29.
[2]	何珺子, 王小军. (2017). 认知能力和非认知能力的教育回报率--基于国际成人能力测评项目的实证研究. 经济与管理研究, 38(5), 66-74.
[3]	洪永淼, 汪寿阳. (2021). 大数据、机器学习与统计学: 挑战与机遇. 计量经济学报, 1(1), 17-35. doi: 10.12012/T03-19
[4]	李一茗, 黎坚. (2020). 复杂问题解决能力的概念、影响因素及培养策略. 北京师范大学学报(社会科学版), (5), 36-48.
[5]	刘红云, 骆方. (2008). 多水平项目反应理论模型在测验发展中的应用. 心理学报, 40(1), 92-100.
[6]	孙鑫, 黎坚, 符植煜. (2018). 利用游戏log-file预测学生推理能力和数学成绩--机器学习的应用. 心理学报, 50(7), 761-770.
[7]	王光宏, 蒋平. (2004). 数据挖掘综述. 同济大学学报(自然科学版), 32(2), 246-252.
[8]	吴忭, 胡艺龄, 赵玥颖. (2019). 如何使用数据: 回归基于理解的深度学习和测评--访国际知名学习科学专家戴维·谢弗. 开放教育研究, 25(1), 4-12.
[9]	徐俊怡, 李中权. (2021). 基于游戏的心理测评. 心理科学进展, 29(3), 394-403.
[10]	袁建林, 刘红云. (2017). 核心素养测量: 理论依据与实践指向. 教育研究, 38(7), 21-36.
[11]	袁建林, 刘红云. (2020). 过程性测量:教育测量的新范式. 中国考试, (12), 1-9.
[12]	詹沛达. (2019). 计算机化多维测验中作答时间和作答精度数据的联合分析. 心理科学, 42(1), 170-178.
[13]	张博, 黎坚, 徐楚, 李一茗. (2014). 11-14岁超常儿童与普通儿童问题解决能力的发展比较. 心理学报, 46(12), 1823-1834.
[14]	张生, 任岩, 骆方. (2019). 学生高阶思维能力的评价: 复杂问题解决的测量述评. 中国特殊教育, 10, 90-96.
[15]	钟志贤. (2004). 促进学习者高阶思维发展的教学设计假设. 电化教育研究, (12), 21-28.
[16]	祖霁云, Patrick Kyllonen. (2019). 非认知能力的重要性及其测量. 中国考试, (9), 22-31.
[17]	Agard, C., & von Davier, A. (2018). The virtual world and reality of testing: Building virtual assessments. In H. Jiao & R. Lissitz (Eds.), Technology enhanced innovative assessment: Development, modeling, and scoring from an interdisciplinary perspective (pp.1-30). Charlotte, NC: Information Age Publishing.
[18]	Autor, D., & Dorn, D. (2009). This job is "getting old": Measuring changes in job opportunities using occupational age structure. American Economic Review, 99(2), 45-51. doi: 10.1257/aer.99.2.45 URL
[19]	Baker, C., Saxe, R., & Tenenbaum, J. (2011). Bayesian theory of mind: Modeling joint belief-desire attribution. Proceedings of the annual meeting of the cognitive science society, 33.
[20]	Banfield, J., & Wilkerson, B. (2014). Increasing student intrinsic motivation and self-efficacy through gamification pedagogy. Contemporary Issues in Education Research, 7(4), 291-298.
[21]	Bergner, Y., Shu, Z., & von Davier, A. A. (2014). Visualization and confirmatory clustering of sequence data from a simulation- based assessment task. Proceedings of the 7th International Conference on Educational Data Mining (pp.177-184), London, UK.
[22]	Bergner, Y., & von Davier, A. (2018). Process data in NAEP: Past, present, and future. Journal of Educational and Behavioral Statistics, 44(6), 706-732. doi: 10.3102/1076998618784700 doi: 10.3102/1076998618784700 URL
[23]	Bezirhan, U., Davier, M. V., & Grabovsky, I. (2021). Modeling item revisit behavior: The hierarchical speed-accuracy- revisits model. Educational and Psychological Measurement, 81(2), 363-387. doi: 10.1177/0013164420950556 doi: 10.1177/0013164420950556 URL
[24]	Brookhart, S. M. (2010). How to assess higher-order thinking skills in your classroom. Alexandria, VA: ASCD.
[25]	Carroll, K. A. & Harris, C. M. (2020). Using a repetitive instructional intervention to improve students' higher-order thinking skills. College Teaching, 69(2), 82-90. doi: 10.1080/87567555.2020.1823310 URL
[26]	Chen, Y. (2020). A continuous-time dynamic choice measurement model for problem-solving process data. Psychometrika, 85(4), 1052-1075. doi: 10.1007/s11336-020-09734-1 URL
[27]	de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333-353. doi: 10.1007/BF02295640 URL
[28]	Dicerbo, K. E. & Kidwai, K. (2013). Detecting player goals from game log files. Vox Sanguinis, 11(3), 350-376. doi: 10.1111/j.1423-0410.1966.tb04613.x URL
[29]	Diehl, M., Marsiske, M., Horgas, A., Rosenberg, A., Saczynski, J., & Willi, S. (2005). The revised observed tasks of daily living: A performance-based assessment of everyday problem solving in older adults. Journal of Applied Gerontology, 24(3), 211-230. doi: 10.1177/0733464804273772 URL
[30]	Diserens, D., Schwartz, M. W., Guenin, M., & Taylor, L. A. (1986). Measuring the problem-solving ability of students and residents by microcomputer. Journal of Medical Education, 61(6), 461-466. pmid: 3712410
[31]	Doerner, D. (1980). On the difficulties people have in dealing with complexity. Simulation & Gaming, 11(1), 87-106.
[32]	Fayyad, U., Piatetsky-shapiro, G., & Smyth, P. (1996). Knowledge discovery and data mining: Towards a unifying framework. Shapiro, 96, 82-88.
[33]	Fossey, W. A. (2017). An evaluation of clustering algorithms for modeling game-based assessment work processes. Unpublished doctoral dissertation, University of Maryland, College Park. URL https://drum.lib.umd.edu/bitstream/handle/1903/20363/Fossey_umd_0117E_18587.pdf?sequence=1
[34]	Fox, J. P., & Marianti, S. (2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51(4), 540-553. doi: 10.1080/00273171.2016.1171128 URL
[35]	Frensch, P. A., & Funke, J. (2002). Thinking and problem solving. In N. Cowan (Ed.). Experimental psychology and its implications for human development: Encyclopedia of life support systems (EOLSS), developed under the auspices of the UNESCO. Oxford, UK: Eolss Publishers.
[36]	Funke, J. (1983). Einige bemerkungen zu problemen der problemlöseforschung oder: Ist testintelligenz doch ein prädiktor? [Some comments to problems of problem solving research, or: An intelligence test is a predictor, isn’t it?]. Diagnostica, 29, 283-302.
[37]	Greiff, S., Wüstenberg, S., & Avvisati, F. (2015). Computer- generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers & Education, 91, 92-105. doi: 10.1016/j.compedu.2015.10.018 URL
[38]	Greiff, S., Wüstenberg, S., & Funke, J. (2012). Dynamic problem solving: A new assessment perspective. Applied Psychological Measurement, 36(3), 189-213. doi: 10.1177/0146621612439620 URL
[39]	Greiff, S., Wüstenberg, S., Holt, D. V., Goldhammer, F., & Funke, J. (2013). Computer-based assessment of complex problem solving: Concept, implementation, and application. Educational Technology Research and Development, 61(3), 407-421. doi: 10.1007/s11423-013-9301-x URL
[40]	Han, Z., He, Q., & von Davier, M. (2019). Predictive feature generation and selection using process data from PISA interactive problem-solving items: An application of random forests. Front Psychology, 10, 2461. doi: 10.3389/fpsyg.2019.02461 URL
[41]	Hao, J., Shu, Z., & von Davier, A. (2015). Analyzing process data from game/scenario-based tasks: An edit distance approach. Journal of Educational Data Mining, 7(1), 33-50.
[42]	Hao, J., Smith, L., Mislevy, R., von Davier, A., & Bauer, M. (2016). Taming log files from game/simulation-based assessments: Data models and data analysis tools (Research Report No. RR-16-10) Princeton, NJ: Educational Testing Service.
[43]	He, Q., Borgonovi, F., & Paccagnella, M. (2021). Leveraging process data to assess adults’ problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers & Education, 166(17), 104170. doi: 10.1016/j.compedu.2021.104170 URL
[44]	He, Q., & von Davier, M. (2016). Analyzing process data from problem-solving items with N-Grams: Insights from a computer-based large-scale assessment. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp.750-777). IGI Global. http://doi:10.4018/978-1-4666-9441-5.ch029
[45]	Jeon, M., Boeck, P. D., Luo, J., Li, X., & Lu, Z. L. (2021). Modeling within-item dependencies in parallel data on test responses and brain activation. Psychometrika, 86(1), 239-271. doi: 10.1007/s11336-020-09741-2 URL
[46]	Jiao, H., Liao, D., & Zhan, P. (2019). Utilizing process data for cognitive diagnosis. In M. von Davier & Y. S. Lee (Eds.), Handbook of diagnostic classification models: Models and model extensions, applications, software packages (pp.421-436). Cham: Springer International Publishing.
[47]	Jiao, H., & Lissitz, R. (2018). Technology enhanced innovative assessment: Development, modeling, and scoring from an interdisciplinary perspective. Charlotte, NC: Information Age Publishing.
[48]	Johnson, R. B., & Christensen, L. (2014). Educational research: Quantitative, qualitative and mixed methods approaches (5^th edition, pp.59-65). Thousand Oaks, CA: SAGE Publications.
[49]	Li, J., Zhang, B., Du, H., Zhu, Z., & Li, Y. (2015). Metacognitive planning: Development and validation of an online measure. Psychological Assessment, 27(1), 260-271. doi: 10.1037/pas0000019 URL
[50]	Liu, C., & Cheng, Y. (2018). An application of the support vector machine for attribute-by-attribute classification in cognitive diagnosis. Applied Psychological Measurement, 42(1), 58-72. doi: 10.1177/0146621617712246 pmid: 29881112
[51]	Liu, H. Y., Liu, Y., & Li, M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372. doi: 10.3389/fpsyg.2018.01372 URL
[52]	Man, K. W., & Harring, J. R. (2020). Assessing preknowledge cheating via innovative measures: A multiple-group analysis of jointly modeling item responses, response times, and visual fixation counts. Educational and Psychological Measurement, 81(3), 441-465. doi: 10.1177/0013164420968630 URL
[53]	Man, K. W., Harring, J. R., Jiao, H., & Zhan, P. (2019). Joint modeling of compensatory multidimensional item responses and response times. Applied Psychological Measurement, 43(8), 639-654. doi: 10.1177/0146621618824853 URL
[54]	Marshall, J. (1977). Assessment of problem-solving ability. Medical Education, 11(5), 329-334. pmid: 904495
[55]	Mayer, R.E. (1990). Problem solving. In M. W. Eysenck (Ed.), The Blackwell dictionary of cognitive psychology (pp.284-288). Basil Blackwell, Oxford.
[56]	Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). Focus article: On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1(1), 3-62. doi: 10.1207/S15366359MEA0101_02 URL
[57]	Molenaar, D., Bolsinova, M., & Vermunt, J. (2018). A semi‐parametric within‐subject mixture approach to the analyses of responses and response times. British Journal of Mathematical and Statistical Psychology, 71(2), 205-228. doi: 10.1111/bmsp.2018.71.issue-2 URL
[58]	Molenaar, D., Oberski, D., Vermunt, J., & de Boeck, P. (2016). Hidden Markov item response theory models for responses and response times. Multivariate Behavioral Research, 51(5), 606-626. doi: 10.1080/00273171.2016.1192983 pmid: 27712114
[59]	NCES. (2014). NAEP TEL Wells sample item. National Center for Education Statistics. Retrieved February 24, 2019, from http://nces.ed.gov/nationsreportcard/tel/wells_item.aspx
[60]	Novak, J. D. (1961). An approach to the interpretation and measurement of problem solving ability. Science Education, 45(2), 122-131. doi: 10.1002/(ISSN)1098-237X URL
[61]	OECD. (2003). The PISA 2003 assessment framework: Mathematics, reading, science and problem solving knowledge and skills. Paris: OECD Publishing.
[62]	OECD.(2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. Paris: OECD Publishing.
[63]	OECD.(2014). PISA 2012 results: Creative problem solving: Students' skills in tackling real-life problems (Volume V). Paris: OECD Publishing.
[64]	OECD.(2016). PISA 2015 assessment and analytical framework: Science, reading, mathematic and financial literacy. Paris, PISA, OECD Publishing.
[65]	OECD.(2019). PISA 2021 creative thinking framework: Third Draft[R]. Paris: OECD Publishing.
[66]	Omodei, M. M., & Wearing, A. J. (1995). The fire chief microworld generating program: An illustration of computer-simulated microworlds as an experimental paradigm for studying complex decision-making behavior. Behavior Research Methods Instruments & Computers, 27(3), 303-316. doi: 10.3758/BF03200423 URL
[67]	Qiao, X., & Jiao, H. (2018). Data mining techniques in analyzing process data: A didactic. Frontiers in Psychology, 9, 2231. doi: 10.3389/fpsyg.2018.02231 pmid: 30532716
[68]	Shu, Z., Bergner, Y., Zhu, M., Hao, J., von Davier, A. (2017). An item response theory analysis of problem-solving processes in scenario-based tasks. Psychological Test and Assessment Modeling, 59(1), 109-131.
[69]	Shute, V., Ke, F., & Wang, L. (2017). Assessment and adaptation in games. In P. Wouters & H. van Oostendorp (Eds.), Instructional techniques to facilitate learning and motivation of serious games (pp.59-78). New York, NY: Springer.
[70]	Shute, V., & Moore, G. (2018). Consistency and validity in game-based stealth assessment. In H. Jiao & R. Lissitz (Eds.), Technology enhanced innovative assessment: Development, modeling, and scoring from an interdisciplinary perspective (pp.31-51). Charlotte, NC: Information Age Publishing.
[71]	Shute, V. J., & Rahimi, S. (2020). Stealth assessment of creativity in a physics video game. Computers in Human Behavior, 116, 1-13.
[72]	Soller, A., & Stevens, R. (2007). Applications of stochastic analyses for collaborative learning and cognitive assessment. In G. R. Hancock & K. M. Samuelsen (Eds.) Advances in latent variable mixture models (pp.217-253). Information Age Publishing.
[73]	Stanek, S. & Sabat, A. (2019). The use of IT tools in the assessment and development of leadership abilities. Problemy Zarządzania - Management Issues, 5(85), 89-110.
[74]	Ulitzsch, E., He, Q., Ulitzsch, V., Molter, H., Nichterlein, A., Niedermeier, R., & Pohl, S. (2021). Combining clickstream analyses and graph-modeled data clustering for identifying common response processes. Psychometrika, 86(1), 190-214. doi: 10.1007/s11336-020-09743-0 pmid: 33544300
[75]	Unal, E., & Cakir, H. (2021). The effect of technology- supported collaborative problem solving method on students’ achievement and engagement. Education and Information Technologies, 26, 4127-4150. doi: 10.1007/s10639-021-10463-w URL
[76]	van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181-204.
[77]	van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287-308. doi: 10.1007/s11336-006-1478-z URL
[78]	Wang, S. Y., Zhang, S. S., Douglas, J., & Culpepper, S. (2018). Using response times to assess learning progress: A joint model for responses and response times. Measurement: Interdisciplinary Research and Perspectives, 16(1), 45-58. doi: 10.1080/15366367.2018.1435105 URL
[79]	Weir, K. (2018). Designing smarter tech tools: New technology in educational gaming, health-care communication, robotics and more is benefiting from psychologists’ input. URL https://www.apa.org/monitor/2018/11/cover-tech-tools.aspx
[80]	Zhan, P., & He, K. (2021). A longitudinal diagnostic model with hierarchical learning trajectories. Educational Measurement: Issues and Practice, 40(3), 18-30. https://doi.org/10.1111/emip.12422
[81]	Zhan, P., Jiao, H., & Liao, D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262-286. doi: 10.1111/bmsp.2018.71.issue-2 URL
[82]	Zhan, P., & Qiao, X. (2020, July 13). A diagnostic classification analysis of problem-solving competence using process data: An item expansion method. https://doi.org/10.31234/osf.io/wtyae
[83]	Zhao, W., Shute, V., & Wang, L. (2015). Stealth assessment of problem-solving skills from gameplay. Interservice/ Industry Training, Simulation, and Education Conference (I/ITSEC), (15212), 1-11.
[84]	Zoanetti, N. (2010). Interactive computer based assessment tasks: How problem-solving process data can inform instruction. Australasian Journal of Educational Technology, 26(5), 585-606.