Assessing cognitive presence using automated learning analytics methods

Abstract

With the increasing pace of technological changes in the modern society, there has been a growing interest from educators, business leaders, and policymakers in teaching important higher-order skills which were identified as necessary for thriving in the present-day globalized economy. In this regard, one of the most widely discussed higher order skills is critical thinking, whose importance in shaping problem solving, decision making, and logical thinking has been recognized. Within the domain of distance and online education, the Community of Inquiry (CoI) model provides a pedagogical framework for understanding the critical dimensions of student learning and factors which impact the development of student critical thinking. The CoI model follows the social-constructivist perspective on learning in which learning is seen as happening in both individual minds of learners and through the discourse within the group of learners. Central to the CoI model is the construct of cognitive presence, which captures the student cognitive engagement and the development of critical thinking and deep thinking skills. However, the assessment of cognitive presence is challenging task, particularly given its latent nature and the inherent physical and time separation between students and instructors in distance education settings. One way to address this problem is to make use of the vast amounts of learning data being collected by learning systems. This thesis presents novel methods for understanding and assessing the levels of cognitive presence based on learning analytics techniques and the data collected by learning environments. We first outline a comprehensive model for cognitive presence assessment which builds on the well-established evidence-cantered design (ECD) assessment framework. The proposed assessment model provides a foundation of the thesis, showing how the developed analytical models and their components fit together and how they can be adjusted for new learning contexts. The thesis shows two distinct and complementary analytical methods for assessing students’ cognitive presence and its development. The first method is based on the automated classification of student discussion messages and captures learning as it is observed in the student dialogue. The second analytics method relies on the analysis of log data of students’ use of the learning platform and captures the individual dimension of the learning process. The developed analytics also extend current theoretical understanding of the cognitive presence construct through data-informed operationalization of cognitive presence with different quantitative measures extracted from the student use of online discussions. We also examine methodological challenges of assessing cognitive presence and other forms of cognitive engagement through the analysis of trace data. Finally, with the intent of enabling for the wider adoption of the CoI model for new online learning modalities, the last two chapters examine the use of developed analytics within the context of Massive Open Online Courses (MOOCs). Given the substantial differences between traditional online and MOOC contexts, we first evaluate the suitability of the CoI model for MOOC settings and then assess students’ cognitive presence using the data collected by the MOOC platform. We conclude the thesis with the discussion of practical application and impact of the present work and the directions for the future research.