Gaming the System
Project Description
Some students who use interactive learning environments "game the system", attempting to succeed in an educational task by systematically taking advantage of properties and regularities in the system used to complete that task, rather than by thinking through the material. In Cognitive Tutor software, gaming the system includes systematic guessing, and repeatedly asking for an additional hint until the software gives the student the answer.
In 2003, using human observations of student behavior in cognitive tutor classrooms, my colleagues and I determined that students who game the system learn significantly less than other students. We have replicated this finding across a variety of classrooms and tutor lessons.
Data mining showed that gaming students split into two groups. One group appears to learn reasonably well. The other group shows extremely poor learning. The group that shows poor learning appears to repeatedly game on specific poorly-known skills. The other group appears to game more sporadically and on easier skills, as part of a broader pattern of focusing time and attention on skills they know poorly.
We have developed a detector of gaming which could accurately identify which students gamed, and when they gamed. We have also validated that this detector can transfer between cohorts of students, and specific curricular material.
We have also conducted two studies to determine what motivations underlying the choice to game, determining that gaming students are more likely to dislike computers, dislike the tutoring software, dislike the subject matter of the tutor, and to tend not to be self-driven in educational settings. Other motivations such as anxiety or performance goals appear not to be associated with the choice to game. Our colleagues Neil Heffernan and Jason Walonoski have also studied what motivations lead students to game, and integrating their findings with our findings is an important area of future work.
We have developed a system which uses a software agent, Scooter the Tutor, to respond to when students game. Scooter uses emotional expressions to signal to the student and their teacher that the student is gaming; Scooter also gives supplementary exercises on exactly the material that students bypass by gaming. Scooter reduces the frequency of gaming; Scooter's supplementary exercises have been shown to significantly improve gaming students' learning.
Neil Heffernan and Jason Walonoski have also recently developed a system to reduce gaming and improve learning, using visualizations of student behavior over time and text prompts about gaming. At the Intelligent Tutoring Systems conference in 2006, my paper on Scooter won best paper, and their poster on remediating gaming won best poster, showing the strength of interest in this emerging area of research.
Publications
Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference
on Intelligent Tutoring Systems, 392-401. [Won Best Paper Award] [pdf]
Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Roll, I. (2006)
Generalizing Detection of Gaming the System Across a Tutoring Curriculum. Proceedings of the 8th International Conference on Intelligent
Tutoring Systems, 402-411. [pdf]
Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students
"Game The System". Proceedings of ACM CHI 2004: Computer-Human
Interaction, 383-390. [pdf]
Baker, R.S., Roll, I., Corbett, A.T., Koedinger, K.R. (2005) Do
Performance Goals
Lead Students to Game the System? Proceedings of the International
Conference on Artificial Intelligence and Education (AIED2005), 57-64.
[pdf]
Baker, R.S., Corbett, A.T., Koedinger, K.R. (2004)
Detecting Student Misuse of Intelligent Tutoring Systems .
Proceedings of the 7th International Conference on Intelligent Tutoring
Systems, 531-540.
[pdf]
Baker, R.S., Corbett, A.T., Koedinger, K.R. (2006) Responding
to Problem Behaviors in Cognitive Tutors: Towards Educational Systems
Which Support All Students . National Association for the
Dually Diagnosed (NADD) Bulletin, 9 (4), 70-75.
[draft pdf]
Baker, R.S., Corbett, A., Koedinger, K., Roll, I. (2005) Detecting
When Students Game The System, Across Tutor Subjects and Classroom
Cohorts . Proceedings of User Modeling 2005, 220-224. [pdf]
Baker, R.S. (2005) Designing Intelligent Tutors That Adapt to When
Students Game the System. Doctoral Dissertation. CMU Technical Report
CMU-HCII-05-104. [pdf]
Collaborators and co-authors
Albert Corbett
Kenneth Koedinger
Ido Roll
Angela Wagner
Neil Heffernan
Jason Walonoski
Didith Rodrigo
Tanja Mitrovic
Shelley Evenson
Tom Mitchell
Daniel Baker
Joseph Beck
Jay Raspat
Meghan Naim
Other researchers who have studied gaming the system
(if I've mistakenly left you off the list, please let me know)
Ivon Arroyo
Joseph Beck
Mingyu Feng
Janet Schofield
Bev Woolf
Jeff Johns
R. Chas Murray
Vincent Aleven
Julita Vassileva
Ran Cheng
Rebecca Crowley
Michael Yudelson
Carole Beal
Lei Qu
Hyokyeong Lee
Ari Bader-Natal
Carolyn Rose
Gahgene Gweon
Rikke Magnussen
Morten Misfeldt
Jack Mostow