Using Instructor Insights to Generate Scalable Metrics

Current models of student evaluations of teaching (SETs) have well documented shortcomings including demographic, discipline, and personality response biases, a focus on student satisfaction rather than learning, reliability issues, and ignoring implementation of evidence-based best practices. We present on our NSF funded project connecting change theory to practice through validating a SET designed to address these shortcomings. Our project is based on the Concerns-Based Adoption Model (CBAM), a popular change theory in K-12 schools adapted for higher education. CBAM supports change efforts by illuminating the challenges stakeholders perceive to inhibit changes and allows for a better understanding of strategies to help address these challenges. Using the CBAM model, we have implemented our SET, known as the S-IMPACT in 14 courses (N=691) and have been measuring congruent validity with the complementary F-IMPACT, a self-report faculty evaluation of teaching instrument, and classroom observations using the Classroom Observation Protocol for Undergraduate STEM (COPUS) instrument. We will present measures of reliability and congruent validity and plans to develop an instrument to easily capture instructors' concerns. These two instruments will provide higher education administrators and stakeholders with valuable information about both the types of practices being used and the instructors' perceived barriers to implementation across a large number of courses with a relatively low cost of deployment.