Leveraging assessment data to improve medical education

Author: Brent Thoma

Thoma, Brent, 2023 Leveraging assessment data to improve medical education, Flinders University, College of Medicine and Public Health

Terms of Use: This electronic version is (or will be) made publicly available by Flinders University in accordance with its open access policy for student theses. Copyright in this thesis remains with the author. You may use this material for uses permitted under the Copyright Act 1968. If you are the owner of any included third party copyright material and/or you believe that any material has been made available without permission of the copyright owner please contact copyright@flinders.edu.au with the details.


Background: Large amounts of data are collected during medical training, but they are rarely utilized by

stakeholders such as learners, assessors, faculty, programs, institutions, and specialties. More often,

these data remain disconnected from other data. They are not used to support data-driven decision

making throughout our organizations and systems. This gap deprives our organizations of information

that could support the growth and development of their people, programs, and systems.

Methodology: The research incorporated in this thesis spans eight articles (six studies and two

editorials) published between 2020 and 2022. The first study leverages the expertise of an international

group of medical educators to identify barriers to the implementation of learning analytics within their

organizations. Four of the six studies use a design-based research methodology to determine the needs

of a key stakeholder (resident, competence committee, faculty, and program) for the analysis and

visualization of the assessment data collected within a competency-based education program.

Interactive dashboards were created to illustrate the findings. The final study amalgamates assessment

data from 15 training programs to explore the fidelity of the implementation of competency-based

medical education (CBME) in Canadian emergency medicine programs. The two editorials explore the

challenges and potential of the use of competency-based assessment data to inform the development of

medical education organizations.

Results: This body of work investigates the analysis and visualization of competency-based assessment

data to support the development of various stakeholders within medical education. It has been

translated into practice in part through the integration of the dashboards created for these stakeholders

within a learning management system that is used by most Canadian residency programs. This work is

contextualized within the CBME, learning analytics, and medical education systems literature and

proposes future directions in the exploration of the use of this assessment data across the system of

medical education.

Conclusion: This thesis informs the usage of competency-based assessment data by multiple

stakeholders. Moving forward, concerns related to the privacy, security, amalgamation, and appropriate

use of this data will need to be addressed in a way that is both morally justifiable and acceptable to

learners. Further work will result in increased sophistication of medical education learning analytics, the

exploration of additional use cases, and synergies with clinical analytics that will inform the growth of

the overall systems of medical education and health care.

Keywords: Medical education assessment, Competency-based medical education, Competency-based education, Faculty development, Program evaluation, Quality improvement, Learning analytics, Data visualization, Organizational development

Subject: Health Education thesis

Thesis type: Doctor of Philosophy
Completed: 2023
School: College of Medicine and Public Health
Supervisor: Dr. Lambert Schuwirth