It depends how you slice it: How measurement and analysis methods affect sensitivity in detecting instructional change following professional development

To assess whether any change initiative has succeeded, it is important to be able to measure the resulting change. Our project draws on data from an established professional development initiative that has offered workshops for mathematics instructors to learn to use inquiry-based learning (IBL). In this talk, we will use observational data from a subset of workshop participants to show how differences in observation and analysis methods affect our ability to detect whether their instructional practices changed after attending a workshop.

We collected video observations from 15 participants teaching a course prior to the workshop and the same course the year after the workshop. We coded the videos using two observation protocols, our own Toolkit for Assessing Mathematics Instruction-Observation Protocol (TAMI-OP) (Hayward et al., 2017) and the Reformed Teaching Observation Protocol (RTOP) (Sawada et al., 2002), a protocol used in numerous other studies of instructional change. We place each into Hora and Ferrare's (2012) framework of observation protocols and show how their differing designs (segmented/descriptive vs. holistic/evaluative) provide different measurement sensitivities.

Two methods of analysis and visualization of the segmented, descriptive TAMI-OP data provide different levels of detail from the same dataset. We use cumulative proportions of various activities to measure change via central tendency. We also use a "heatmap" approach to help visualize patterns in how classroom activities vary from day to day across a whole course. Paired side-by-side comparisons of these two analyses allow us to detect changes in the sequencing of instruction that may be significant even if the overall proportions of instructional methods do not change. We will also share matched results from RTOP scores. We will engage the audience in a broader discussion about what change, if any, we can detect with different measurement and analysis methods.