Please wait a minute...
Acta Psychologica Sinica    2018, Vol. 50 Issue (7) : 761-770     DOI: 10.3724/SP.J.1041.2018.00761
Reports of Empirical Studies |
Using game log-file to predict students' reasoning ability and mathematical achievement: An application of machine learning
Xin SUN1,Jian LI1,2(),Zhiyu FU1
1 Faculty of Psychology, Beijing Normal University
2 Beijing Key Lab of Applied Experimental Psychology, Beijing 100875, China
Download: PDF(536 KB)   HTML Review File (1 KB) 
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks     Supporting Info
Guide   
Abstract  

With the development of the progress of information technology, the deficiency of traditional psychological testing is becoming more obvious, such as test anxiety and test exposure. Some researchers have begun to test individuals using game-based assessment, which has many advantages, such as increasing the motivation and input level of the participants, and providing the possibility for the implementation of log-file technology. However, the current data analysis and scoring logic ignore substantial information of process, and thus cannot accurately assess individual characteristics and abilities. The advantages of machine learning in data analysis provide a new direction. The machine learning algorithm can analyze the log-file data by building a complex model.

The present study attempted to use game-based assessment combining game log-file and machine learning techniques to predict participants’ ability: reasoning ability and mathematical achievement. Participants were 360 first and second grade students from a middle school in Beijing; predictive variables were a series of features extracted from the game log-file, outcome variables were dichotomous variables calculated from Raven test and mathematics achievement, which took 25th and 75th percentile as the cutoff line. In the model training, the random forest algorithm was selected, 70% samples were randomly selected for cross validation and hyper parametric search, and then the prediction was carried out on the other 30% of samples.

Results showed that the logarithm of the ratio of the first step time to the average execution time was the highest features of average importance ratio, and the number of steps that are different from the optimal solution, thinking time ratio, execution between fluctuation, proportion of repeat steps all contributed to the mathematical achievement prediction model; reasoning ability prediction model was similar. With these important features, it could be found that the reasoning ability prediction model had 76.11% precision, 65.72% accuracy, 63.10% recall and 65.01% F1 scores; the mathematical achievement prediction model had 83.07% precision, 73.70% accuracy, 73.33% recall and 75.57% F1 score.

The finding of the present study showed that the random forest model had acceptable predictive effect when predicting reasoning ability and mathematics achievement classification based on the game log-file, with 75% precision of reasoning and 80% precision of math. In conclusion, the research provides a new method to predict the cognitive ability and academic achievement of the students; the game log-file combined with machine learning can establish an effective discrimination model. This result can provide some reference and direction for the development of educational psychological assessment.

Keywords video game      Sokoban      machine learning      reasoning ability      mathematical achievement.     
ZTFLH:  B849: G44  
Issue Date: 29 May 2018
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
Xin SUN
Jian LI
Zhiyu FU
Cite this article:   
Xin SUN,Jian LI,Zhiyu FU. Using game log-file to predict students' reasoning ability and mathematical achievement: An application of machine learning[J]. Acta Psychologica Sinica, 2018, 50(7): 761-770.
URL:  
http://journal.psych.ac.cn/xlxb/EN/10.3724/SP.J.1041.2018.00761     OR     http://journal.psych.ac.cn/xlxb/EN/Y2018/V50/I7/761
  
  
特征 平均值 标准差 最小值 最大值
失败组
第一步用时/平均执行时间 22.71 24.26 2.52 198.34
ln (第一步用时/平均执行时间) 2.31 0.82 0.81 4.97
完成箱子的比例 0.33 0.08 0.00 0.57
第一步用时/总时间 0.22 0.12 0.04 0.76
ln (第一步用时/总时间) -1.92 0.60 -3.31 -0.29
思考步数占比 -2.39 0.23 -3.04 -1.69
平均执行时间 0.64 0.15 0.37 1.33
执行间波动 2.15 1.20 0.35 10.52
重复步数占比 0.07 0.03 0.00 0.20
与最优步数相差 -5.75 9.45 -23.36 65.78
与最优路径重合步数占比 0.17 0.04 0.04 0.32
成功组
第一步用时/平均执行时间 24.36 23.81 2.65 168.97
ln (第一步用时/平均执行时间) 2.49 0.78 0.92 4.95
第一步用时/总时间 0.25 0.14 0.04 0.77
ln (第一步用时/总时间) -1.77 0.61 -3.18 -0.27
思考步数占比 -2.61 0.27 -3.53 -1.64
平均执行时间 0.48 0.11 0.33 1.18
执行间波动 1.17 0.76 0.20 5.43
重复步数占比 0.03 0.02 0.00 0.16
与最优步数相差 7.65 5.45 0.00 52.67
与最优路径重合步数占比 0.71 0.14 0.17 1.06
  
表现类型 预测为阳性 预测为阴性
实际为阳性 TP FN
实际为阴性 FP TN
  
  
最优化目标 F1 查准率 查全率 精确率
推理能力
F1优先 68.83% 74.40% 61.19% 63.46%
查准率优先 63.72% 75.51% 59.17% 65.03%
查全率优先 65.01% 74.91% 63.10% 64.21%
精确率优先 64.22% 76.11% 59.05% 65.72%
数学成绩
F1优先 71.14% 79.35% 71.11% 68.02%
查准率优先 75.57% 83.07% 73.33% 73.70%
查全率优先 73.09% 81.06% 71.78% 70.62%
精确率优先 71.65% 80.19% 69.67% 69.44%
  
[1] Baumert A., Schlösser T., & Schmitt M . ( 2014). Economic games: A performance-based assessment of fairness and altruism. European Journal of Psychological Assessment, 30( 3), 178-192.
url: http://econtent.hogrefe.com/doi/abs/10.1027/1015-5759/a000183
[2] Berg, W.K., &Byrd D.L . ( 2002). The Tower of London spatial problem-solving task: Enhancing clinical and research implementation. Journal of Clinical and Experimental Neuropsychology, 24( 5), 586-604.
pmid: 12187443 url: http://www.tandfonline.com/doi/abs/10.1076/jcen.24.5.586.1006
[3] Bors, D.A., &Vigneau F. , ( 2003). The effect of practice on Raven's Advanced Progressive Matrices. Learning and Individual Differences, 13( 4), 291-312.
url: http://linkinghub.elsevier.com/retrieve/pii/S1041608003000153
[4] Breiman, L. ( 2001). Random forests. Machine Learning, 45( 1), 5-32.
url: http://link.springer.com/10.1023/A:1010933404324
[5] Cassady, J.C., &Johnson R.E . ( 2002). Cognitive test anxiety and academic performance. Contemporary Educational Psychology, 27( 2), 270-295.
url: http://linkinghub.elsevier.com/retrieve/pii/S0361476X0191094X
[6] Csapó B., Ainley J., Bennett R. E., Latour T., & Law N . ( 2012). Technological issues for computer-based assessment. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching of 21st century skills( pp. 143-230). Dordrecht: Springer.
url: http://www.springerlink.com/content/fulltext.pdf?id=doi:10.1007/978-94-007-2324-5_4
[7] DiCerbo, K.E ., & Behrens, J. T .( 2012). Implications of the digital ocean on current and future assessment. In R. W. Lissitz & H. Jiao (Eds.), Computers and their impact on state assessments: Recent history and predictions for the future (pp. 143-306). Charlotte, NC: Information Age Publishing.
[8] Di Giunta L., Alessandri G., Gerbino M., Kanacri P. L., Zuffiano A., & Caprara G. V . ( 2013). The determinants of scholastic achievement: The contribution of personality traits, self-esteem, and academic self-efficacy. Learning and Individual Differences, 27, 102-108.
url: http://linkinghub.elsevier.com/retrieve/pii/S1041608013000976
[9] Duncan G. J., Dowsett C. J., Claessens A., Magnuson K., Huston A. C., Klebanov P., .. Japel C . ( 2007). School readiness and later achievement. Developmental Psychology, 43( 6), 1428-1446.
pmid: 18020822 url: http://doi.apa.org/getdoi.cfm?doi=10.1037/0012-1649.43.6.1428
[10] Greiff S., Wüstenberg S., & Avvisati F . ( 2015). Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers & Education, 91, 92-105.
url: http://dl.acm.org/citation.cfm?id=2850402
[11] Harrington, P . ( 2013). Machine learning in action (R. Li, P. Li, Y. D. Qu, & B. Wang, Trans.). Beijing, China: Posts & Telecom Press.
[11] [ Harrington,P. ( 2013). 机器学习实战 (李锐, 李鹏, 曲亚东, 王斌译). 北京: 人民邮电出版社.]
[12] Hausknecht J. P., Halpert J. A., Di Paolo N. T., & Moriarty Gerrard, M. O. ( 2007). Retesting in selection: A meta- analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92( 2), 373-385.
pmid: 17371085 url: http://doi.apa.org/getdoi.cfm?doi=10.1037/0021-9010.92.2.373
[13] Heinzen T. E., Landrum R. E., Gurung R. A.R., & Dunn, D. S. ( 2015). Game-based assessment:The mash-up we've been waiting for. In T. Reiners & L. C. Wood (Eds.), Gamification in education and business (pp. 201-217). Switzerland: Springer International Publishing.
url: http://link.springer.com/chapter/10.1007/978-3-319-10208-5_11
[14] Hembree, R ( 1988). Correlates, causes, effects, and treatment of test anxiety. Review of Educational Research, 58( 1), 47-77.
url: http://journals.sagepub.com/doi/10.3102/00346543058001047
[15] Ikeda M., Iwanaga M., & Seiwa H . ( 1996). Test anxiety and working memory system. Perceptual and Motor Skills, 82( 3), 1223-1231.
pmid: 8823887 url: http://journals.sagepub.com/doi/10.2466/pms.1996.82.3c.1223
[16] Judd L. L., Schettler P. J., & Rush A. J . ( 2016). A brief clinical tool to estimate individual patients’ risk of depressive relapse following remission: Proof of concept. American Journal of Psychiatry, 173( 11), 1140-1146.
pmid: 27418380 url: http://ajp.psychiatryonline.org/doi/10.1176/appi.ajp.2016.15111462
[17] Keogh, E ., &French C.C . ( 2001). Test anxiety, evaluative stress, and susceptibility to distraction from threat. European Journal of Personality, 15( 2), 123-141.
url: http://doi.wiley.com/10.1002/%28ISSN%291099-0984
[18] Kinnunen, R., &Vauras M. , ( 1995). Comprehension monitoring and the level of comprehension in high-and low-achieving primary school children's reading. Learning and Instruction, 5( 2), 143-165.
url: http://linkinghub.elsevier.com/retrieve/pii/095947529500009R
[19] Köstering L., Schmidt C. S. M., Egger K., Amtage F., Peter J., Klöppel S., ..Kaller C. P . ( 2015). Assessment of planning performance in clinical samples: Reliability and validity of the Tower of London task (TOL-F). Neuropsychologia, 75, 646-655.
pmid: 26197091 url: https://linkinghub.elsevier.com/retrieve/pii/S0028393215301020
[20] Li J., Zhang B., Du H., Zhu Z., & Li Y. M . ( 2015). Metacognitive planning: Development and validation of an online measure. Psychological Assessment, 27( 1), 260-271.
pmid: 25222433 url: http://doi.apa.org/getdoi.cfm?doi=10.1037/pas0000019
[21] Moharil B., Gokhale C., Ghadge V., Tambvekar P., Pundlik S., & Rai G . ( 2014). Real time generalized log file management and analysis using pattern matching and dynamic clustering. International Journal of Computer Applications, 91( 16), 1-6.
url: http://adsabs.harvard.edu/abs/2014IJCA...91p...1M
[22] Neisser, U. ( 1997). Rising scores on intelligence tests: Test scores are certainly going up all over the world, but whether intelligence itself has risen remains controversial. American Scientist, 85( 5), 440-447.
url: http://www.jstor.org/stable/27856851
[23] Pedregosa F., Varoquaux G., Gramfort A., Michel V., Thirion B., Grisel O., .. Duchesnay é . ( 2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12, 2825-2830.
[24] Pressley, M., &Afflerbach P. , ( 1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, N.J.: Erlbaum.
[25] Raven, J. ( 1989). The raven progressive matrices: A review of national norming studies and ethnic and socioeconomic variation within the united-states. Journal of Educational Measurement, 26( 1), 1-16.
url: http://www.blackwell-synergy.com/toc/jedm/26/1
[26] Schmidt, F.L . ( 2002). The role of general cognitive ability and job performance: Why there cannot be a debate. Human Performance, 15( 1-2), 187-210.
url: http://www.informaworld.com/openurl?genre=article&doi=10.1207/S15327043HUP1501&02_12&magic=crossref||D404A21C5BB053405B1A640AFFD44AE3
[27] Sonnleitner P., Brunner M., Greiff S., Funke J., Keller U., Martin R., .. Latour T . ( 2012). The Genetics Lab: Acceptance and psychometric characteristics of a computer- based microworld assessing complex problem solving. Psychological Test and Assessment Modeling, 54( 1), 54-72.
[28] Tan P. N., Steinbach M., & Kumar V . ( 2006). Introduction to data mining . India:Pearson Education.
[29] Tenorio Delgado M., Arango Uribe P., Aparicio Alonso A., & Rosas Díaz R . ( 2016). TENI: A comprehensive battery for cognitive assessment based on games and technology. Child Neuropsychology, 22( 3), 276-291.
pmid: 25396766 url: http://www.tandfonline.com/doi/full/10.1080/09297049.2014.977241
[30] Veenman M. V. J., Wilhelm P., & Beishuizen J. J . ( 2004). The relation between intellectual and metacognitive skills from a developmental perspective. Learning and Instruction, 14( 1), 89-109.
url: http://linkinghub.elsevier.com/retrieve/pii/S095947520300135X
[31] Veenman M. V. J., Bavelaar L., De Wolf L., &van Haaren, M. G. P. ( 2014). The on-line assessment of metacognitive skills in a computerized learning environment. Learning and Individual Differences, 29, 123-130.
url: http://linkinghub.elsevier.com/retrieve/pii/S1041608013000058
[32] Ventura, M., &Shute V ., ( 2013). The validity of a game-based assessment of persistence. Computers in Human Behavior, 29( 6), 2568-2572.
url: http://linkinghub.elsevier.com/retrieve/pii/S0747563213002252
[33] Wu Y. Y., Kosinski M., & Stillwell D . ( 2015). Computer- based personality judgments are more accurate than those made by humans. Proceedings of the National Academy of Sciences of the United States of America, 112( 4), 1036-1040.
pmid: 25583507 url: http://www.pnas.org/lookup/doi/10.1073/pnas.1418680112
[34] Zhang B., Li J., Xu C., & Li Y. M . ( 2014). The developmental differences of problem solving ability between intellectually- gifted and intellectually-average children aged from 11-14 years old. Acta Psychologica Sinica, 46, 1823-1834.
url: http://d.wanfangdata.com.cn/Periodical/xlxb201412004
[34] [ 张博, 黎坚, 徐楚, 李一茗 . ( 2014). 11~14岁超常儿童与普通儿童问题解决能力的发展比较. 心理学报, 46, 1823-1834.]
url: http://d.wanfangdata.com.cn/Periodical/xlxb201412004
[35] Zhang Z., Song Y. F., Cui L. Q., Liu X. Q., & Zhu T. S . ( 2016). Emotion recognition based on customized smart bracelet with built-in accelerometer. PeerJ, 4, e2258.
pmid: 27547564 url: https://peerj.com/articles/2258
[1] ZHANG Bo; LI Jian; XU Chu; LI Yiming. The Developmental Differences of Problem Solving Ability between Intellectually-gifted and Intellectually-average Children Aged from 11-14 Years Old[J]. Acta Psychologica Sinica, 2014, 46(12): 1823-1834.
[2] GUO Xiao-Li1,JIANG Guang-Rong,ZHU Xu. Short-Term Desensitizing Effects of Violent Video Games: Comparison Between Two Exposure Ways[J]. , 2009, 41(03): 259-266.
[3] Sun Changhua, Wu Zhenyun, Wu Zhiping,Xu Shulian (Institute of Psychology,Chinese Academy of Sciences). AGE DIFFERENCES IN RAVEN TEST AND THE RELATION BETWEEN THE DIFFERENCES AND MEMORY TRAINING OF “METHOD OF LOCI”[J]. , 1994, 26(01): 59-63.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Acta Psychologica Sinica
Support by Beijing Magtech