Going Online: A Simulated Student Approach for Evaluating Knowledge Tracing in the Context of Mastery Learning
Proceedings of the Fourteenth International Conference on Educational Data Mining
2021
Abstract
Knowledge tracing algorithms are embedded in Intelligent Tutoring Systems (ITS) to keep track of students’ learning process. While knowledge tracing models have been extensively studied in offline settings, very little work has explored their use in online settings. This is primarily because conducting experiments to evaluate and select knowledge tracing models in classroom settings is expensive. To fill this gap, we introduce a novel way of using machine learning models to generate simulated students. We conduct experiments using agents generated by the Apprentice Learner Architecture to investigate the online use of different knowledge tracing models (Bayesian Knowledge Tracing, the Streak model, and Deep Knowledge Tracing). An analysis of our simulation results revealed an error in the initial implementation of our Bayesian knowledge tracing model that was not identified in our previous work. Our simulations also revealed a more fundamental limitation of Deep Knowledge Tracing that prevents the model from supporting mastery learning on multi-step problems. Together, these two findings suggest that Apprentice agents provide a practical means of evaluating knowledge tracing models prior to more costly classroom testing. Lastly, our analysis identifies a positive correlation between the Bayesian knowledge tracing parameters estimated from human data and the parameters estimated from simulated learners. This suggests that model parameters might be initialized using simulated data when no human-student data is yet available.
BibTeX
@inproceedings{zhang-edm-2021,
title = {Going Online: A Simulated Student Approach for Evaluating Knowledge Tracing in the Context of Mastery Learning},
author = {Zhang, Qiao and MacLellan, Christopher J.},
booktitle = {Proceedings of the Fourteenth International Conference on Educational Data Mining},
year = {2021},
}