How can we help change leaders understand how measurement and data can be used?
ASCN Working Group 4: Demonstrating Impact is trying something new. This group's mission is to identify, explain, and disseminate information on metrics that hold the potential to document, foster, accelerate, and communicate systemic change. Good questions are a great way to share and expand knowledge. Each month a question of interest and value to the higher education community will be sent to the working group members. Responses will be collated and posted on the ASCN blog. We hope that this will lead to beneficial collaborations not just among the members of the working group, but also across the network, and will reach the larger higher education community interested in systemic change.
The assumption behind this group is that measurement and data are effective mechanisms for facilitating change. The question for this month has two parts.
How can we help change leaders understand how measurement and data can be used? Can you give an example from your own experience where this has happened?
Below are the first three responses received. Please use comment section to respond to the question and to engage in a discussion about the current responses. If there is a link or citation that you think would be of value to other readers, please include this as well.
In addition, if there are any questions you would like Demonstrating Impact Working Group to address, please email those to Inese, the ASCN Project Manager.
There are two key approaches I've found to help change leaders better understand the use of data in education.
One is simply to make them aware of education research and the ability to assess student learning in a more objective and reliable way. I've found that when leaders become aware of tools (such as concept inventories), they become advocates for using these tools to collect data and assess interventions.
The second approach is to create an institution-specific "dashboard" of data that change leaders can easily access and play with. Our institution has created a tool that allows leaders to look at grade histograms for each course, displayed for various demographic groups. This tool has fostered intense discussion about disparities in these distributions and driven change to reduce these disparities.
The ASCN Working Group 4 can help change leaders understand how measurement and data can be used by explaining it in terms of Return on Investments (ROI's). Measurements provide institutions with relevant information on whether or not an intervention or program is working. Data from assessments provide a quantifiable account of the universities strengths and weaknesses as it relates to student academic success. It may also reveal common themes of how money is spent and potential areas of improvement. In my experience, change leaders and leaders in higher administration want to know how resources are being used and if the institution, investors and funding agencies are getting a ROI (i.e., increased persistence and graduation rates).
Over the last year, I have worked with my research team to identify measurements that would assess factors (i.e., faculty mentoring research experiences, REU's, mentoring programs) and other High Impact Practices that have been found to predict student success. For example, my team collected data from students that participated in a Biomedical/Behavioral Science Research Program and those who did not. We found that students that participated in our Research Program were more likely to graduate and pursue a career in research. This data helped us to advocate for additional resources for our program. It also showed the leaders in the institution how funds from an external funding agency were used to accomplish our programs goals and how it benefited not only the students but also the university.
In sum, measurements and data are effective tools in helping change leaders both understand and invest in these efforts at an institutional level and grasp the ROI.
Like a good politician, instead of answering the question as asked, let me try to answer a slightly different question: what happens if change leaders are too convinced that measurement and data can be useful? I am referring to data analytics and predictive models of student success. I'd like to stay generic, though, since no particular example would be fair to those involved. Given our ability to develop quantitative models of student success and to create risk factors in the model that flag when a student might have a higher than usual risk of X (dropping out, getting poor grades, etc.), how do we deal with the Minority Report conundrum? Stats are not clinical, and they do not predict for an individual; they just indicate a potential for higher risk. While with sufficient resources we can now follow up and serve individual better, less resourced situations can use this information to select, track, label (think health insurance companies). There is also the risk to remove people from the risk pool in order to get better overall stats. I have no answer to the question other than to suggest that we have to be conscientious and aware that with knowledge comes power and potential for abuse.
Which brings me to the next good use: of course we can now clearly see which section of what course shows better results in the short and long term. Again, knowledge is first and foremost a good thing. We want to understand what works and what does not. But how do we respond? What is our strategy for dealing with this knowledge? Do we have one when we are generating the knowledge? If not, will knowledge be perceived as a threat, as a means to harm and hurt? In my not so hypothetical case, we would first want to know why there are differences between sections; do they attract different students? Are they using a different approach? Is the person teaching it the only difference? If so, what will we do? Can we give that person support? Do these reflections mean that we need to think through the entire value chain around knowledge to action before we start a conversation based on data. Probably not. But we ought to be aware that measurement and data are double-edged swords...Comment? Start the discussion about How can we help change leaders understand how measurement and data can be used?