Presentations: Session III

This paper discusses the case of implementing a system for student evaluation of courses (SEC) within a faculty (school) of engineering within a Scandinavian traditional university. The process for implementation took approximately two years (from conceptualization to operational) and 10 years in terms of being culturally embedded (Roxå et al. 2021). It is argued that an analysis of this process requires analytical tools for different levels applied with sensitive timing. In short, several change theories and theories of change are applicable for different critical phases in order to avoid distortion of the overall process, as often happens (Senge 2006). In addition to this, a concrete principle for change guides the process while it becomes messy, and from the standpoint of change managers appears overwhelmingly chaotic. SEC has now been in place for 17 years and has transformed the faculty and is used for evaluative, managerial, and developmental purposes, it is also used in teachers' dossiers while engaging in the faculty based incentive system for excellence in teaching and by the student union.

The need for multiple change theories to navigate institution wide change is advocated in Transforming institutions (White et al. 2020), published by the ASCN, and by Kezar and Holocombe (2019). In this paper we differentiate between change theory, wider frameworks; theory of change, a particular approach for a specific change initiative (Reinholz and Andrews 2020); and principles for change (Quinn Patton 2018), intellectual tools functioning as aiding compasses during complex and chaotic change processes.

The analysis displays instances when various change theories (Kezar 2018) are balanced: scientific management initiatives call for cultural influence, which in turn show the need for professional development (social cognition) leading to neo-institutional change. The need to balance these and to spot the need for a changed approach is especially emphasised.

As a Hispanic Serving University (HSI), Texas State University is working towards not only to graduate students but also to provide culturally enhancing educational experiences to minoritized learners. Our STEM Communities Project, a 5-year NSF-funded project, strives to implement sustainable student-centered instructional changes in Science, Technology, Engineering, and Mathematics (STEM) courses. Our approach consists of providing College of Science and Engineering (CoSE) faculty with opportunities to engage in professional development events, assess historical student outcomes in target courses, and redesign those courses by incorporating culturally relevant instructional practices. These activities are designed to create communities among faculty and students within and across the CoSE departments. We will present the progress and challenges of the early stages of the project. In particular, the talk will focus on a component of our project called department instructional assessment (DIA), which is a first step to transforming classrooms based on course-level student data. The DIA process involves analyzing historical data of student outcomes in several STEM courses, spanning from introductory to major-level, to understand inequities across various student demographics. Through the sharing of these course-level data, we encouraged faculty teams to engage in discussions regarding students' learning outcomes and course experiences to recognize areas of instructional improvement with an equity cognitive frame. We recorded and analyzed these faculty discussions to gain insights into the supports that bring about equity thinking. We will share some preliminary findings of this analysis as well as an update on the course redesign workshops.

Over many decades, education researchers and practitioners have developed validated instruments for studying traditional in-person classroom-based teaching practices. These instruments are not easily adaptable to the virtual classrooms. The use of technology – particularly online instruction – alters instructional realities and practices (Major, 2015).
The recent and rapid increase in online teaching presents new challenges in studying teaching practices, including the development and validation of new instruments. This project developed and validated two instruments for measuring online teaching approaches: an observation protocol and a self-report questionnaire.

Key drivers for this study were the ideas that measurement produces improvement (The National Academy of Engineering, 2009), and the potential to promote more effective teaching practices in undergraduate STEM courses. The online environment provides a unique learning space that has the potential to be either more or less inclusive for diverse learners. Our goal is to engage in discussion about the potential for this type of work in initiating conversations that could lead to more welcoming learning spaces for online undergraduate STEM courses.

Data were collected through multiple phases and included both systematic observations and self-report methodologies. The final phase includes a sample of 250 instructors.

Preliminary results suggest that both instruments can be used to reliably collect non-evaluative data for the description and study of online courses. This research was supported by a National Science Foundation grant, and the instruments will be shared with participants and are free to use.