Presentations: Session I

STEM educational change projects are increasingly drawing on change theory to consider how and why change occurs. Change theory is a framework of ideas, supported by evidence, that explains some aspect of how or why change occurs. Change theory relevant to higher education comes from diverse areas and change work is often siloed by STEM discipline. Therefore, we conducted a systematic review of how change theory has been used in STEM higher educational change between 1995-2019.

Our analysis of 97 peer-reviewed articles revealed a notable lack of theoretical coherence in the relatively narrow domain of STEM higher educational change. The reviewed articles used 40 distinct change theories, more than half of which were used in just one or two articles. Eight change theories were used in three or more articles, with the vast majority using one of two change theories: Communities of Practice (n = 26 articles) and Diffusion of Innovations (n = 19 articles). Eleven articles created new theories. Though each change context is unique, this enormous diversity in theoretical grounding may be a major barrier to generalization.

We also analyzed the way in which change theories informed efforts and found that most research drew upon theory in a superficial fashion. Work that does not substantively draw on theory often cannot contribute to theory, limiting what can be transferred to new contexts. Additionally, the majority of articles focused on change at the level of individuals without considering the larger system in which individuals exist, yet changing systems may be key to achieving sustainable change.

This presentation will describe the methods and results of this review, provide space for discussing implications, and highlight a new free, online resource to help change agents and researchers learn about change theories.  

 The School of Science and Technology at Georgia Gwinnett College attracts a highly diverse student population, many from groups traditionally underrepresented in STEM education. However, there is disparity in diversity between the students we serve and those that we graduate. To address that disparity, we received NSF funding to develop and assess a comprehensive systems model aimed at improving student retention, progression, and graduation rates.

Our systems model addresses the complexity and challenges of reforming STEM education by focusing on the entire institution rather than targeted courses, faculty, or departments. Our model includes three interconnected levels: (1) a macro-level that focuses on the institution's objective to increase graduation rates in STEM fields; (2) a meso-level that addresses curriculum reform; and (3) a micro-level focused on faculty development and student engagement and learning. Continuous interactions and assessments occur between the levels, promoting an informed systems approach that positively influences student learning and engagement, faculty development, curricula reform, and decision making processes all at the same time.

What lessons have we learned after six years? There have been many successes and challenges. Our curricula are redesigned, faculty are more engaged in high-impact teaching strategies, institutional support is growing, and as a consequence, our students have benefitted. Assessment is ongoing and results indicate our model leads to improved teaching and student engagement, as evidenced by: (1) improved STEM skills in students, (2) improved student GPAs; (3) increased STEM retention, especially for minority and underrepresented groups; and (4) improved student attitudes about STEM. Additionally, faculty have honed their teaching skills, redesigned curricula, increased collaborations, and published SoTL research. Challenges also exist; project assessment and evaluation is overwhelming, and the work requires a "village". In this presentation, we share the impact, efficacy, successes, and challenges of the model, along with some key findings to date.

Student evaluation surveys are not valid instruments for evaluating teaching, student learning, or the curriculum. For almost a century, researchers have found that not only are these tools highly correlated with course grades, but they are also correlated with grade inflation and lower levels of student learning.

Further, overreliance on student evaluation surveys can lead to fraud or extortion because instructors feel compelled to play school and inflate grades in order to keep their job and get a promotion. Researchers have found that these instruments are unfair to teachers because these they are based on the subjective emotions and beliefs of students, which are largely beyond an instructor's control. This is especially damaging to quality teachers with high standards because they often receive the lowest evaluation scores, even though they produce the most learning.

These instruments are also discriminatory. Student ratings are correlated with a vast array of prejudices based on teacher characteristics that have nothing to do with teaching, such as race, nationality, gender, and attractiveness.

Stark and Freishtat (2014) argued, "there's general agreement that student evaluations of teaching don't mean what they claim to mean" (Kamenetz 2014, para. 10). Student evaluations tell us about the biased and emotional mind of the student. They tell us very little, if anything about the instructor, teaching, student learning, or the curriculum. It can be useful to understand student perceptions, but we need to be clear about what is actually being measured and why.

Unfortunately, few are having this conversation. Stark and Freishtat (2014) conclude: "It's totally valuable to ask [students] about their experience, but it's not synonymous with good teaching". Thus, 18 scholarly associations have all stated that student surveys should not be relied upon "as a measure of teaching quality" (Supiano 2019, para. 2). And yet, these invalid tools have proliferated. How do we move beyond these flawed metrics? What are better tools?

The Instructional-Teams Project (I-TP) at the University of Arizona is an NSF-funded effort (DUE-1626531) that supports high-quality active-learning instruction in large-enrollment STEM courses through building high-functioning instructional teams with diversified roles and high-quality instructional tasks that create opportunities for formative assessment of student thinking. The I-TP has leveraged multiple institutional-level change initiatives and existing instructional resources to directly support instructors' use of evidence-based instructional practices in their classrooms. Since Spring 2017, 38 instructors from 19 different departments (including 17 STEM departments) have participated in the project, representing a significant impact on students across our campus. To better characterize how I-TP participation has influenced instructors' practice, we have followed up with instructors on their motivations and goals for participating in the I-TP, and whether and how participation in the I-TP has led to lasting instructional change. Participants include those teaching in-person, asynchronous online, and synchronous remote courses, giving us insight into how the I-TP has impacted the ways instructors implement instructional change with the support of instructional teams in these different teaching modalities. In this talk, we will present the model underlying the I-TP and describe how our institutional resources and initiatives influenced the genesis of the I-TP. We will then present lessons learned about factors that influenced instructors' engagement with the I-TP and whether and how the I-TP has had lasting impacts on their teaching. We will discuss the role the I-TP has played in both in-person and online teaching environments and close with examples of how participating in the project prior to Spring 2020 impacted how instructors negotiated the switch to remote teaching during the pandemic.