Working Group 2: Costs and Benefits
Change initiatives related to pedagogical innovation are typically analyzed in terms of financial costs to the institution and initial time investments by the faculty, while the benefits are framed in terms of improved student outcomes (generally at the course level). What is generally missing from these analyses is the potential financial benefit to the institution of improved student performance, e.g. improved retention, persistence in a major, and graduation rates, among other possible measures. Typical analyses of pedagogical change initiatives also fall short in capturing the ongoing cost of faculty time investment to use improved teaching methods beyond the initial learning curve. This working group will explore the financial and human costs and benefits of scaling and sustaining instructional changes in undergraduate STEM. In particular, techniques and expertise from economics and related social sciences will be applied to STEM higher education in ways that resonate with network members.
Knowledge Claims and Established Ideas
Much of the research on change initiatives related to pedagogical innovation has been focused on costs and benefits to individuals (in particular, instructors - see in particular work by Charles Henderson, Andrea Beach, Melissa Dancy and others in the Resources section) and groups of students (e.g. changes in learning outcomes in a class section or course), although there is a growing literature on efforts to promote, implement, and assess pedagogical innovation at the department level (see in particular, work by the Carl Wieman Science Education Initiative at the University of British Columbia and the CU Science Education Initiative at the Univ. of Colorado-Boulder, among others).
There has been relatively less work done on costs and benefits of change at the institution level. See the Resources section for an initial set of annotated resources on this topic. As the group moves forward we will continue to add to this list, summarize what is known on this topic, and determine directions for fruitful investigation that will add to the knowledge base.
Key QuestionsDuring the July, 2016 ASCN workshop workshop members developed the following "key questions" to be addressed in the coming months. We expect that this list will continue to be modified over time with continued feedback from the "post-secondary STEM education change-agent" community.
- How do we capture costs and benefits of "student success and learning" – in terms of students, faculty, staff, administrators, and institutions?
- How can we design a cost/benefit analysis that incorporates a "triple bottom line" approach (see Wikipedia and The Economist for definitions of "triple bottom line") that includes the overall impacts of implementing educational change on student learning experiences, faculty and staff experiences, and educational outcomes? A (single) "bottom line" approach (a business-related term) focuses only on financial gains/losses (profits), while a "triple bottom line" approach includes broader social and sustainability factors as well. In this case, a "triple bottom line" approach would include not just financial gains and losses to the institution, but also the impacts on instructors and students, as well as broader social issues (e.g. representation of under-represented minorities in a discipline).
- What is the relationship between traditional notions of "cost" (primarily financial) and broader "economic values" (which can include social costs and benefits) to students and institutions?
- What changes should we consider in designing more navigable outcomes for students? How can we become more deliberate and intentional in helping students navigate courses and curricula to reach institutional goals such as graduation, meaningful employment, and entry into post-baccalaureate degree programs.
- What kinds of information can inform different approaches to education with different expected outcomes? How can we accommodate the needs for different measurements/data based on different expectations and values?
- What kinds of Institutional data inform and influence change and in what ways? How does this inform what kinds of data should be collected to promote institutional change?
- What data do institutions collect that they rarely use to evaluate the costs/benefits of instructional reform? How can they use available data more effectively to guide decisions?
- What fiscal and human benefits and costs should be considered to assess whether an innovation is worthwhile?
- How do faculty experience their day-to-day teaching work and how does that experience affect faculty members' interest in adopting new, evidence-based teaching practices and promoting them within their department? How is teaching (and the improvement of teaching) valued by the department and institution? What is the incentive to improve teaching and learning? What constraints do faculty feel (in terms of incentives, alternate uses of time, colleagues' approval, etc.) when considering making changes to their teaching practices? How can change leaders make use of this knowledge to promote change?
- What role do part-time and/or contingent teaching staff play in creating institutional change related to teaching and learning?
- How does the "change process" differ across institutional types - community colleges, public four-year institutions, private four-year institutions, R1 and comprehensive regional institutions - and how does this impact the costs and benefits of change? To what extent are they similar or different and how does this inform "change initiatives" within and across institutions?
- How do decision makers decide about investing in institutional change initiatives and how do those decisions affect all the stakeholders in the system?
- How do we evaluate alternative choices when making decisions about how to improve student learning outcomes?
- What role do new technological tools play in mediating the costs/benefits of pedagogical change - at both the individual and institutional levels?
Workshop participants emphasized that when considering the impact of change in higher education, it is important to include a long time frame and both economic and social costs/benefits, including effects on equity and inclusion. Participants also emphasized that to ensure the broadest possible understanding of issues related to change, it is important that each of the working groups incorporate input from the full spectrum of higher education institutions, including two-year colleges and minority serving institutions.
Potential Outcomes or Products
Workshop members developed the following prioritized list of potential outcomes for the group:
- Developing a research plan for analyzing how faculty members' decisions about implementing evidence-based teaching practices are impacted by resources and incentive/reward structures. In addition,determining how faculty members perceive the costs of changing educational practices, how these perceptions influence time allocation in the short run, and the manifestation of these decisions in terms of long term time utility
- Developing a review article on cost benefit analysis in higher education, with a focus on issues relevant for changes in STEM education. Issues to be addressed:
- Developing an annotated list of references, with examples from other disciplines, areas, etc. The Resources page for this working group serves as a starting point for this activity.
- Determining the relevant"concepts" to look at from the various "institutional change literatures": exploration vs. efficiency, six-sigma, etc.
- Identifying tools and people with relevant expertise (e.g. those involved in infrastructure planning, social scientists - in particular, economists, etc.) in cost-benefit analysis. When translating this work for STEM education, we will need to emphasize clarity of languageand conceptual framing to enhance quality of communication, especially for those with little prior background in cost/benefit analysis.
- How can an institution differentiate itself from others in terms of student success along a continuum of outcomes (initial jobs, flexibility over time as jobs change in the future, critical thinking skills, etc.)
- How do institutions help consumers make "informed choices" about college education?
- What are the explicit (and hidden) costs of "doing nothing" to promote changes in institutional/instructional practices?
- What are the costs/benefits of "change initiatives" that have already been started? What data/evidence is being collected? How is that data/evidence informing "next steps"?
Action Plan for Moving Forward
The working group developed the following action plan for moving the work of the group forward:
In the immediate future (the next two months):
- Plan and convene the next meeting (virtual) for the group (September, 2016)
- Develop a list of cost and benefits associated with institutional change related to improved instructional practices and student learning
- Accounting for both economic and human/social costs/benefits
- Developing a matrix with types of costs/benefits and for each, listing related resources (working papers, articles, reports, etc.), a brief summary of the how the resource is related to a particular cost/benefit and the knowledge claims made, and any additional questions remaining
- Identifying additional contributors to the cost/benefit group, perhaps by leading a nationally-advertised webinar on the group's purpose, plans, and goals; disciplinary listservs; social media; etc.
- Develop a small research proposal to initiate projects listed under "Potential Outcomes" above (e.g. NSF EAGER funding - see information on where to submit potentially transformative research proposals and the EAGER funding mechanism specifically)
- Contact NACUBO (National Association of College and University Business Officers) for information about business officers' evaluation of costs associated with institutional change and/or restructuring
- Identify common presuppositions of faculty members, staff, and administrators related to institutional and instructional change
- Initiate conversations with instructors regarding costs/benefits of instructional and institutional change
- Survey (formally or informally) presidents, chancellors, and provosts on their views related to instructional and institutional change
- Develop a set of "commissioned papers" on the topic of costs/benefits of institutional change related to improving student learning outcomes via evidence-based teaching practices
- Develop a proposal for a small workshop grant to further the work of the cost/benefit working group (or a series of workshops, each focusing on one of the workgroups)
- Develop and implement a variety of communication/engagement strategies for multiple levels of stakeholders
- Lorne Whitehead, University of British Columbia
- Scott Simkins, North Carolina A&T State University
- Tom Rudin, National Academy of Sciences, Engineering, and Medicine
- Judith Ramaley, Portland State University
- Jay Labov, National Academy of Sciences, Engineering, and Medicine
- Clif Poodry, HHMI
- Sarah Simmons, HHMI
- Karl Smith, University of Minnesota–Twin Cities
- Eric Brewe, Florida International University
- Lorne Whitehead, University of British Columbia
- Linda Slakey, AAU, AACU/PKAL
- Amy Chang, American Society of Microbiology
- Scott Simpkins, North Carolina A&T State University
- Heidi Schweingruber, National Academy of Sciences, Engineering, and Medicine
- Rita Kirshstein, George Washington University
- Sylvia Hurtado, UCLA