Oral Presentations: Session D

The Partnership for Undergraduate Life Science Education (PULSE), a non-profit organization, provides life sciences departments at all types of US higher education institutions with training and resources to implement best educational practices for undergraduates. PULSE Fellows, who are faculty and administrators, have helped more than 200 departments diagnose their current conditions, enhance communication effectiveness, develop action plans, and access resources to support change since 2012. PULSE recognized the need in 2017 to articulate explicitly the implicit theory of change (ToC) underlying its work in order to document PULSE impact and expand its reach.During a 3-day retreat, a small team immersed itself in ToC and organizational change literature, re-examined PULSE assumptions, goals, activities, and outcomes, and articulated a first draft of a PULSE ToC derived from the Annie E. Casey Foundation model1 (core capacities, impact and influence strategies, and outcomes) and grounded in Positive Organizational Scholarship2. The draft has since been reviewed by PULSE Fellows and iteratively revised to reflect insights gained from prior work with departments. A penultimate version of the ToC was approved in Fall 2018. Since then, efforts to validate the ToC have commenced using assessment data from one major PULSE activity (2-day workshops with whole departments). Once validated, the ToC will be used to reanalyze PULSE assessment data to determine impact of PULSE programs and to inform ongoing PULSE efforts to help academic departments improve undergraduate biology education.
In this session, we will describe the ToC development process, theories and key components of the PULSE ToC, how it informs PULSE change efforts, and how PULSE uses assessment data to empirically validate and refine the ToC. We will demonstrate how assessment data from PULSE activities can be used to validate and iteratively revise a ToC. This session is related to ASCN Working Groups 1 and 4 and will interest those who wish to learn from an actual experience of developing a ToC and applying a ToC to demonstrate the impact of change.

The DeLTA project aims to promote comprehensive change in thinking and culture related to undergraduate STEM education at the University of Georgia. Our approach is multi-level; we are convening and supporting the work of faculty, departmental leadership, and upper administrators. Faculty will work in Instructional Action Teams to expand the use of evidence-based teaching practices in introductory courses across STEM. Faculty in these teams will engage in long-term, collaborative teaching professional development. Department leaders will collaborate across units as a Leadership Action Team to develop and enact new ways of supporting, evaluating, incentivizing, and rewarding evidence-based teaching at the department level. Strategic Action Teams of administrators will work opportunistically to align university incentive structures with teaching reform. Our approach is informed by social cognition and cultural theories of change, while taking into account political, institutional, and scientific management theories of change. We are investigating the degree to which thinking and actions relevant to STEM education change over the course of the five-year initiative among faculty, departments, and administration. We are gathering multiple lines of evidence to examine patterns and drivers of change across levels. We will examine teacher thinking using interviews and surveys. We will study teacher practice using classroom observations and analysis of course-based assessments. We will examine shifts in thinking at the department level through interviews and surveys probing perceptions of departmental culture and practices. We will use artifact collection, participant observation, and interviews to document changes in practices, such as those related to hiring, tenure & promotion, and training and collaboration related to teaching. In this presentation, we will describe the initiative, relating specific components to theories of change and highlighting early successes and challenges. We will also discuss research we are conducting on the change process within this initiative and seek audience insights.

The Teaching Quality Framework (TQF) Initiative seeks to create a process and tools for systemic transformation of departmental evaluation of teaching. As part of the Bay View Alliance and NSF funded TTEVAL collaboration, TQF is a response to national calls (e.g., Association of American Universities, National Academies) and draws on decades of research on higher education change and faculty evaluation. TQF objectives include: 1) documenting the iterative process of developing and implementing quality measures of teaching excellence and procedures for enacting these measures within academic units, 2) aligning and sharing resources, processes, and values across departments, 3) adopting these quality measures campus wide, and 4) enhancing the value of teaching campus wide. Here we present the theoretical background, our framework model, and the general three-phase process; we then explore these in more depth within the context of three departmental case studies. In Phase I, departments express interest, which is cultivated via one-on-one meetings with TQF team members and attendance at regular campuswide stakeholder discussions. During Phase II, the TQF team and departmental leadership coordinate to define timelines, processes, and members for Departmental Action Teams (DATs) using an opt-in model: departments choose to participate and determine who participates and how participants will be rewarded. In Phase III, DAT teams engage in regular facilitated meetings, review the framework, externalize values around teaching, identify/create tools to better assess quality teaching, and pilot the use of these tools. Fourteen departments across three colleges are currently engaged in this process (6 in Phase I, 2 in Phase II, 6 in Phase III). In addition to an improved evaluation system and increased valuation of teaching, predicted TQF outcomes include: externalization of departmental/institutional values around teaching and learning, improved instruction and student outcomes, and a shift in culture toward a scholarly approach to teaching.