Discussion
Incorporating only three ARS questions into a 30 min lecture significantly improved participants' scores on a knowledge assessment by roughly 10% immediately after the lecture and 2 weeks later. In addition, the rate of knowledge loss decreased for content tested using ARSs. Using ARSs during the lecture also increased participants' self-reported engagement, learning and enjoyment. Therefore, we support the use of ARSs as one strategy to increase learner knowledge, retention and engagement during lectures.
Several similar studies have shown conflicting results. Two single-site studies reported improved retention with ARSs, while a larger multi-institutional study involving physicians showed no improvements in knowledge or retention. Hoyt and colleagues reported that medical students' final examination scores did not improve after the introduction of ARSs compared with the previous year's students, while another study involving medical students found increased understanding of the material without any increase in retention. These differences may be accounted for, in part, by inherent limitations of the study designs. For instance, several of these studies examined the use of ARSs within existing curricula. In this setting, participants were inclined to independently study the material, thereby improving their knowledge regardless of educational methodologies used, introducing a significant confounding factor. These studies also used live lectures; thus, variables introduced by lecturers themselves could not be controlled. Judson and Sawada reviewed 33 years of studies involving clicker use in college lecture halls, and concluded that the improvements in student comprehension initially attributed to ARS use were probably due to individual instructors' efficacy. We searched several databases, including PubMed, and were unable to find any studies that controlled for these factors. We uniquely controlled these variables through two key features in our study design, which increased the accuracy and validity of our results: (1) testing material outside of the normal curriculum and (2) using a recorded lecture.
However, the generalisability of our results is limited because of the use of a single cohort of students at a single school, the relatively small sample size, and the coverage of a single topic. An additional limitation is the structure of the knowledge assessment, as some items included 'all of the above' and 'none of the above' answer choices, as well as negatives in the question stems. However, the purpose of this study was to evaluate a teaching method, and the assessment items had appropriate reliability, difficulty and discrimination.
Further study is needed to determine the effect of asking various types of question with ARSs, as this may better define how to use ARSs during lectures. For instance, testing foundational material at the knowledge or comprehension level of Bloom's Taxonomy may be more likely to provide benefits earlier in a lecture, especially for students who need to correct misunderstandings before learning advanced material. Similarly, asking a difficult application question towards the end of the lecture would allow students to review some of the key concepts presented earlier in the lecture. The impact of ARSs may also be related to the number of questions asked during a lecture, as there may be a point of maximum benefit, after which, additional questions have a neutral or negative effect on learning. Furthermore, students' learning styles and preferences could contribute to or detract from ARS benefits, and should therefore be examined in conjunction with educational interventions.
As discussed in the Introduction, three current theories that explain the beneficial impact of ARSs on learning are (1) the testing effect, (2) correcting early misunderstandings, and (3) restarting the attention clock. The effects of ARSs in this study can be attributed to the testing effect because there was a statistically significant difference in knowledge assessment scores between content that was specifically tested using ARSs and content that was covered but not tested via ARSs. This implies that the process of thinking about a question and choosing an answer provides an additional learning benefit for that content. Chan et al found that testing also facilitates retention of related but non-tested information, a result that could possibly be attributed to the second theory. However, our results did not show a statistically significant improvement in knowledge for the seven non-ARS questions and did not support this theory. Furthermore, if ARSs were simply an attention-grabbing mechanism, there would be no significant difference in scores between tested and non-tested content.
Because ARS benefits can be explained by the testing effect, lecturers might be able to use less expensive ways than electronic devices to ask questions during lectures which may provide similar effects on learning. In fact, Stoddard and Piquette showed that there was no difference in educational outcomes between lecturers that incorporated ARSs and lecturers that simply put the same multiple choice questions to the class. In contrast, Mayer et al found that using ARSs to answer questions during a lecture increased examination scores relative to answering the same questions without ARSs. Other methods of asking questions during the learning process include administering short quizzes at the end of class and online quizzing. Both of these have been shown to be effective and may be as convenient as ARSs, but these methods assess students after the material has been presented rather than during the learning process. Another study involving undergraduate students found that using ARSs increased student participation, honesty while answering questions, and positive emotion during a lecture compared with hand-raising and response cards, so there may be additional benefits of using ARSs. Ultimately, it is our view that ARSs provide a convenient and effective way to improve the learning experience.