Evaluating the Impact of Instructional Support Using Data Mining and Process Mining: A Micro-Level Analysis of the Effectiveness of Metacognitive Prompts

Christoph Sonnenberg, Maria Bannert


In computer-supported learning environments, the deployment of self-regulatory skills represents an essential prerequisite for successful learning. Metacognitive prompts are a promising type of instructional support to activate students’ strategic learning activities. However, despite positive effects in previous studies, there are still a large number of students who do not benefit from provided support. Therefore, it may be necessary to consider explicitly the conditions under which a prompt is beneficial for a student, i.e., so-called adaptive scaffolding. The current study aims to (i) classify the effectiveness of prompts on regulatory behavior, (ii) investigate the correspondence of the classification with learning outcome, and (iii) discover the conditions under which prompts induce regulatory activities (i.e., the proper temporal positioning of prompts). The think-aloud data of an experiment in which metacognitive prompts supported the experimental group (n = 35) was used to distinguish between effective and non-effective prompts. Students’ activities preceding the prompt presentation were analyzed using data mining and process mining techniques. The results indicate that approximately half of the presented prompts induced metacognitive learning activities as expected. Moreover, the number of induced monitoring activities correlates positively with transfer performance. Finally, the occurrence of orientation and monitoring activities, which are not well-embedded in the course of learning, increases the effectiveness of a presented prompt. In general, our findings demonstrate the benefits of investigating metacognitive support using process data, which can provide implications for the design of effective instructional support.

Full Text:



Aleven, V. (2013). Help seeking and intelligent tutoring systems: Theoretical perspectives and a step towards theoretical integration. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (pp. 311--335). New York: Springer. doi:10.1007/978-1-4419-5546-3 21

Aleven, V., Roll, I., Mclaren, B. M., & Koedinger, K. R. (2016). Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26(1), 205--223. doi:10.1007/s40593-015-0089-1

Azevedo, R. (2005). Using hypermedia as a metacognitive tool for enhancing student learning? The Role of self-regulated learning. Educational Psychologist, 40(4), 199--209. doi:10.1207/s15326985ep4004 2

Azevedo, R. (2009). Theoretical, conceptual, methodological, and instructional issues in research on metacognition and self-regulated learning: A discussion. Metacognition and Learning, 4, 87--95. doi:10.1007/s11409-009-9035-7

Azevedo, R. (2014). Issues in dealing with sequential and temporal characteristics of self- and socially-regulated learning. Metacognition and Learning, 9, 217--228. doi:10.1007/s11409-014-9123-1

Azevedo, R., Cromley, J. G., Moos, D. C., Greene, J. A., & Winters, F. I. (2011). Adaptive content and process scaffolding: A key to facilitating students' self-regulated learning with hypermedia. Psychological Testing and Assessment Modeling, 53(1), 106--140.

Azevedo, R., Cromley, J. G., & Seibert, D. (2004). Does adaptive scaffolding facilitate students' ability to regulate their learning with hypermedia? Contemporary Educational Psychology, 29, 344--370. doi:10.1016/j.cedpsych.2003.09.002

Azevedo, R., Cromley, J. G., Winters, F. I., Moos, D. C., & Greene, J. A. (2005). Adaptive human scaffolding facilitates adolescents' self-regulated learning with hypermedia. Instructional Science, 33, 381--412. doi:10.1007/s11251-005-1273-8

Azevedo, R., & Hadwin, A. F. (2005). Scaffolding self-regulated learning and metacognition - Implications for the design of computer-based scaffolds. Instructional Science, 33, 367--379. doi:10.1007/s11251-005-1272-9

Azevedo, R., Landis, R. S., Feyzi-Behnagh, R., Duffy, M., Trevors, G., Harley, J. M., ldots Hossain, G. (2012). The effectiveness of pedagogical agents' prompting and feedback in facilitating co-adapted learning with MetaTutor. Lecture Notes in Computer Science, 212--221. doi:10.1007/978-3-642-30950-2 27

Bannert, M. (2006). Effects of reflection prompts when learning with hypermedia. Journal of Educational Computing Research, 35(4), 359--375. doi:10.2190/94V6-R58H-3367-G388

Bannert, M. (2007). Metakognition beim Lernen mit Hypermedia. Erfassung, Beschreibung und Vermittlung wirksamer metakognitiver Lernstrategien und Regulationsaktivitäten [Metacognition and hypermedia learning]. Münster: Waxmann.

Bannert, M. (2009). Promoting self-regulated learning through prompts: A discussion. Zeitschrift für Pädagogische Psychologie, 23(2), 139--145. doi:10.1024/1010-0652.23.2.139

Bannert, M., & Mengelkamp, C. (2013). Scaffolding hypermedia learning through metacognitive prompts. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (pp. 171--186). New York: Springer. doi:10.1007/978-1-4419-5546-3 12

Bannert, M., & Reimann, P. (2012). Supporting self-regulated hypermedia learning through prompts. Instructional Science, 40(1), 193--211. doi:10.1007/s11251-011-9167-4

Bannert, M., Reimann, P., & Sonnenberg, C. (2014). Process mining techniques for analysing patterns and strategies in students' self-regulated learning. Metacognition and Learning, 9(2), 161--185. doi:10.1007/s11409-013-9107-6

Bannert, M., Sonnenberg, C., Mengelkamp, C., & Pieger, E. (2015). Short- and long-term effects of students' self-directed metacognitive prompts on navigation behavior and learning performance. Computers in Human Behavior, 52, 293--306. doi:10.1016/j.chb.2015.05.038

Ben-Eliyahu, A., & Bernacki, M. L. (2015). Addressing complexities in self-regulated learning: A focus on contextual factors, contingencies, and dynamic relations. Metacognition and Learning, 10, 1--13. doi:10.1007/s11409-015-9134-6

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289-300.

Bloom, B. S. (1956). Taxonomy of educational objectives: The classification of educational goals. New York: D. McKay.

Bouchet, F., Harley, J. M., Trevors, G. J., & Azevedo, R. (2013). Clustering and profiling students according to their interactions with an intelligent tutoring system fostering self-regulated learning. Journal of Educational Data Mining, 5(1), 104--146.

Brusilovsky, P. (2001). Adaptive hypermedia. User Modeling and User-Adapted Interaction, 11, 87--110. doi:10.1023/A:1011143116306

Brusilovsky, P. (2007). Adaptive navigation support. In P. Brusilovsky, A. Kobsa, & W. Nejdl (Eds.), The adaptive web. Methods and strategies of web personalization (pp. 263--290). Berlin Heidelberg: Springer. doi:10.1007/978-3-540-72079-9

Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences, 6(3), 271--315. doi:10.1207/s15327809jls0603 1

Clarebout, G., & Elen, J. (2006). Tool use in computer-based learning environments: Towards a research framework. Computers in Human Behavior, 22, 389--411. doi:10.1016/j.chb.2004.09.007

Clarebout, G., Elen, J., Collazo, N. A. J., Lust, G., & Jiang, L. (2013). Metacognition and the use of tools. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (pp. 187--195). Springer Science. doi: 10.1007/978-1-4419-5546-3 13

Cocea, M., & Weibelzahl, S. (2009). Log file analysis for disengagement detection in e-Learning environments. User Modelling and User-Adapted Interaction, 19, 341--385. doi:10.1007/s11257-009-9065-5

Dent, A. L., & Hoyle, R. H. (2015). A framework for evaluating and enhancing alignment in self-regulated learning research. Metacognition and Learning, 10, 165--179. doi:10.1007/s11409-015-9136-4.

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, Mass.: MIT Press.

Ge, X. (2013). Designing learning technologies to support self-regulation during III-structured problem-solving processes. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (pp. 213--228). New York: Springer. doi:10.1007/978-1-4419-5546-3 15

Greene, J. A., Dellinger, K. R., Tüysüzoglu, B. B., & Costa, L.-J. (2013). A two-tiered approach to analyzing self-regulated learning data to inform the design of hypermedia learning environments. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (pp. 117--128). New York: Springer. doi:10.1007/978-1-4419-5546-3 8

Günther, C. W., & van der Aalst, W. M. P. (2007). Fuzzy mining -- Adaptive process simplification based on multi-perspective metrics. Business Process Management - Lecture Notes in Computer Science, 4714, 328--343. doi:10.1007/978-3-540-75183-0

Hämäläinen, W., & Vinni, M. (2010). Classifiers for educational data mining. In C. Romero, S. Ventura, M. Pechenizkiy, & R. Baker (Eds.), Handbook of Educational Data Mining (pp. 57--74). Boca Raton, FL: Chapman&Hall/CRC.

Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York: Routledge.

Jeong, H., Gupta, A., Roscoe, R., Wagster, J., Biswas, G., & Schwartz, D. (2008). Using hidden markov models to characterize student behaviors in learning-by-teaching environments. Lecture Notes in Computer Science, 5091, 614--625. doi:10.1007/978-3-540-69132-7 64

Johnson, A. M., Azevedo, R., & D'Mello, S. K. (2011). The temporal and dynamic nature of self-regulatory processes during independent and externally assisted hypermedia learning. Cognition and Instruction, 29(4), 471--504. doi:10.1080/07370008.2011.610244

Kinnebrew, J. S., Segedy, J. R., & Biswas, G. (2014). Analyzing the temporal evolution of students' behaviors in open-ended learning environments. Metacognition and Learning, 9, 187--215. doi:10.1007/s11409-014-9112-4

Kramarski, B., & Gutman, M. (2006). How can self-regulated learning be supported in mathematical e-learning environments? Journal of Computer Assisted Learning, 22, 24--33. doi:10.1111/j.1365-2729.2006.00157.x

Lajoie, S. P., Naismith, L., Poitras, E., Hong, Y.-J., Cruz-Panesso, I., Ranellucci, J., ldots Wiseman, J. (2013). Technology-rich tools to support self-regulated learning and performance in medicine. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (pp. 229--242). New York: Springer. doi:10.1007/978-1-4419-5546-3 16

Lehmann, T., Hähnlein, I., & Ifenthaler, D. (2014). Cognitive, metacognitive and motivational perspectives on preflection in self-regulated online learning. Computers in Human Behavior, 32, 313--323. doi:10.1016/j.chb.2013.07.051

Lester, J. C., Mott, B. W., Robison, J. L., Rowe, J. P., & Shores, L. R. (2013). Supporting self-regulated science learning in narrative-centered learning environments. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (pp. 471--483). New York: Springer. doi:10.1007/978-1-4419-5546-3 30

Lin, X., Hmelo, C., Kinzer, C. K., & Secules, T. J. (1999). Designing technology to support reflection. Educational Technology Research and Development, 47(3), 43--62. doi:10.1007/BF02299633

Malmberg, J., Järvelä, S., Järvenoja, H., & Panadero, E. (2015). Promoting socially shared regulation of learning in CSCL: Progress of socially shared regulation among high- and low-performing groups. Computers in Human Behavior, 52, 562--572. doi:10.1016/j.chb.2015.03.082

Martin, T., & Sherin, B. (2013). Learning analytics and computational techniques for detecting and evaluating patterns in learning: An introduction to the special issue. Journal of the Learning Sciences, 22(4), 511--520. doi:10.1080/10508406.2013.840466

Molenaar, I., & Chiu, M. M. (2014). Dissecting sequences of regulation and cognition: Statistical discourse analysis of primary school children's collaborative learning. Metacognition and Learning, 9(2), 137--160. doi:10.1007/s11409-013-9105-8

Molenaar, I., & Järvelä, S. (2014). Sequential and temporal characteristics of self and socially regulated learning. Metacognition and Learning, 9(2), 75--85. doi:10.1007/s11409-014-9114-2

Molenaar, I., & Roda, C. (2008). Attention management for dynamic and adaptive scaffolding. Pragmatics & Cognition, 16(2), 224-271. doi: 10.1075/pc.16.2.04mol

Reimann, P., Markauskaite, L., & Bannert, M. (2014). E-Research and learning theory: What do sequence and process mining methods contribute? British Journal of Educational Technology, 45(3), 528--540. doi:10.1111/bjet.12146

Reimann, P., & Yacef, K. (2013). Using process mining for understanding learning. In R. Luckin, P. Puntambekar, P. Goodyear, B. Grabowski, J. Underwood, & N. Winters (Eds.), Handbook of Design in Educational Technology (pp. 472--481). New York: Routledge.

Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Improving students' help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction, 21(2), 267--280. doi:10.1016/j.learninstruc.2010.07.004

Schoor, C., & Bannert, M. (2012). Exploring regulatory processes during a computer-supported collaborative learning task using process mining. Computers in Human Behavior, 28, 1321--1331. doi:10.1016/j.chb.2012.02.016

Schwonke, R., Hauser, S., Nückles, M., & Renkl, A. (2006). Enhancing computer-supported writing of learning protocols by adaptive prompts. Computers in Human Behavior, 22, 77--92. doi:10.1016/j.chb.2005.01.002

Sitzmann, T., Bell, B. S., Kraiger, K., & Kanar, A. M. (2009). A multilevel analysis of the effect of prompting self-regulation in technology-delivered instruction. Personnel Psychology, 62, 697--734. doi:10.1111/j.1744-6570.2009.01155.x

Song, M., & van der Aalst, W.M.P. (2007). Supporting process mining by showing events at a glance. In K. Chari & A. Kumar (Eds.), Proceedings of 17th Annual Workshop on Information Technologies and Systems (WITS 2007) (pp. 139--145). Montreal, Canada.

Sonnenberg, C., & Bannert, M. (2015). Discovering the effects of metacognitive prompts on the sequential structure of SRL-processes using process mining techniques. Journal of Learning Analytics, 2(1), 72--100.

Thillmann, H., Künsting, J., Wirth, J., & Leutner, D. (2009). Is it merely a question of ``what'' to prompt or also ``when'' to prompt? Zeitschrift für Pädagogische, 23(2), 105--115. doi:10.1024/1010-0652.23.2.105

Trčka, N., Pechenizkiy, M., & van der Aalst, W. M. P. (2010). Process mining from educational data. In C. Romero, S. Ventura, M. Pechenizkiy, & R. Baker (Eds.), Handbook of Educational Data Mining (pp. 123--142). Boca Raton, FL: Chapman&Hall/CRC.

Van der Aalst, W. (2011). Process mining: Discovery, conformance and enhancement of business processes. Berlin, Heidelberg: Springer.

Van der Aalst, W., Weijters, T., & Maruster, L. (2004). Workflow mining: Discovering process models from event logs. IEEE Transactions on Knowledge and Data Engineering, 16(9), 1128--1142. doi:10.1109/TKDE.2004.47

Veenman, M. (1993). Metacognitive ability and metacognitive skill: Determinants of discovery learning in computerized learning environments. University of Amsterdam.

Veenman, M. V. J. (2007). The assessment and instruction of self-regulation in computer-based environments: A discussion. Metacognition and Learning, 2(2-3), 177--183. doi:10.1007/s11409-007-9017-6

Walker, E., Rummel, N., & Koedinger, K. (2011). Designing automated adaptive support to improve student helping behaviors in a peer tutoring activity. International Journal of Computer Supported Collaborative Learning, 6(2), 279-306. doi: 10.1007/s11412-011-9111-2

Winne, P. H. (1996). A metacognitive view of individual differences in self-regulated learning. Learning and Individual Differences, 8(4), 327--353.

Winne, P. H. (2014). Issues in researching self-regulated learning as patterns of events. Metacognition and Learning, 9, 229--237. doi:10.1007/s11409-014-9113-3

Winne, P. H., & Baker, R. S. (2013). The potentials of educational data mining for researching metacognition, motivation and self-regulated learning. Journal of Educational Data Mining, 5(1), 1--8.

Winne, P. H., & Hadwin, A. F. (2008). The weave of motivation and self-regulated learning. In D. Schunk & B. J. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297--314). NY: Taylor & Francis.

Wirth, J. (2009). Promoting self-regulated learning through prompts. Zeitschrift für Pädagogische Psychologie, 23(2), 91--94. doi:10.1024/1010-0652.23.2.91

Wittwer, J., & Renkl, A. (2008). Why instructional explanations often do not work: A framework for understanding the effectiveness of instructional explanations. Educational Psychologist, 43, 49--64. doi:10.1080/00461520701756420

Yeh, Y. F., Chen, M. C., Hung, P. H., & Hwang, G. J. (2010). Optimal self-explanation prompt design in dynamic multi-representational learning environments. Computers and Education, 54, 1089--1100. doi:10.1016/j.compedu.2009.10.013

Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1), 166--183. doi:10.3102/0002831207312909


  • There are currently no refbacks.