Author: Brent Thoma
Thoma, Brent, 2023 Leveraging assessment data to improve medical education, Flinders University, College of Medicine and Public Health
Terms of Use: This electronic version is (or will be) made publicly available by Flinders University in accordance with its open access policy for student theses. Copyright in this thesis remains with the author. You may use this material for uses permitted under the Copyright Act 1968. If you are the owner of any included third party copyright material and/or you believe that any material has been made available without permission of the copyright owner please contact copyright@flinders.edu.au with the details.
Background: Large amounts of data are collected during medical training, but they are rarely utilized by
stakeholders such as learners, assessors, faculty, programs, institutions, and specialties. More often,
these data remain disconnected from other data. They are not used to support data-driven decision
making throughout our organizations and systems. This gap deprives our organizations of information
that could support the growth and development of their people, programs, and systems.
Methodology: The research incorporated in this thesis spans eight articles (six studies and two
editorials) published between 2020 and 2022. The first study leverages the expertise of an international
group of medical educators to identify barriers to the implementation of learning analytics within their
organizations. Four of the six studies use a design-based research methodology to determine the needs
of a key stakeholder (resident, competence committee, faculty, and program) for the analysis and
visualization of the assessment data collected within a competency-based education program.
Interactive dashboards were created to illustrate the findings. The final study amalgamates assessment
data from 15 training programs to explore the fidelity of the implementation of competency-based
medical education (CBME) in Canadian emergency medicine programs. The two editorials explore the
challenges and potential of the use of competency-based assessment data to inform the development of
medical education organizations.
Results: This body of work investigates the analysis and visualization of competency-based assessment
data to support the development of various stakeholders within medical education. It has been
translated into practice in part through the integration of the dashboards created for these stakeholders
within a learning management system that is used by most Canadian residency programs. This work is
contextualized within the CBME, learning analytics, and medical education systems literature and
proposes future directions in the exploration of the use of this assessment data across the system of
medical education.
Conclusion: This thesis informs the usage of competency-based assessment data by multiple
stakeholders. Moving forward, concerns related to the privacy, security, amalgamation, and appropriate
use of this data will need to be addressed in a way that is both morally justifiable and acceptable to
learners. Further work will result in increased sophistication of medical education learning analytics, the
exploration of additional use cases, and synergies with clinical analytics that will inform the growth of
the overall systems of medical education and health care.
Keywords: Medical education assessment, Competency-based medical education, Competency-based education, Faculty development, Program evaluation, Quality improvement, Learning analytics, Data visualization, Organizational development
Subject: Health Education thesis
Thesis type: Doctor of Philosophy
Completed: 2023
School: College of Medicine and Public Health
Supervisor: Dr. Lambert Schuwirth