Insights into systemic reform from K-12 research on data driven decision-making

In the Spring of 2015 I was part of a research team that visited three California universities, as part of a study on the prevalence of data driven decision-making (DDDM) in STEM departments. We spoke with people about student evaluations, exams, hallway conversations, learning analytics, ABET, and increasing pressure from institutional accreditors, finding that these issues were on everybody's minds. This was unsurprising, because it's safe to say that higher education has entered a data-focused accountability and reform phase not unlike the K-12 sector in the 1990s.

But as we gathered accounts of how people used these forms of data "in the wild" of their departments and offices, the study took a surprise turn. We simply couldn't escape an issue that we hadn't anticipated being so central to our research – that of organizational systems and how they function (or not). This is because the production, retrieval, and use of data and information implicate a complex ecology of policies, norms, tools, and behaviors that function at multiple levels within a university.

A study that I had initiated with my colleague Jana Bouwma-Gearhart, with whom I shared a curiosity (and dread) about how the wave of accountability was hitting higher education – quickly led me to revisit the systems literatures that we'd amassed in graduate school and old projects. Because ASCN is focused on systemic change in higher education, I thought to share some insights from what I found; specifically, how research on data use in K-12 settings holds great promise and implications for the work of ASCN members.

As I delved into the literature it quickly became apparent that there was a lot to learn, not just because of the voluminous writings on the topic, but because I was working out of field. Systems research extends across many fields: cybernetics, biology, organizational science, cognitive science, human development, engineering, management, sociology and anthropology, and on and on. To gather insights I spoke to colleagues in systems engineering, development, organizational behavior and situated cognition, and began to piece together an elementary understanding of the subject.

Some of the key issues that scholars of organizational systems kept returning to, and that seemed critical for our study of data use to keep in mind, were the following:

• Determining whether the system is closed or open (i.e., permeability to external forces);
• Accounting for how different units interact with and influence one another (e.g., tightly or loosely coupled);
• Deciding whether organizational systems are being viewed as changeable (by managerial fiat) or not;
• Acknowledging that system operations are non-linear and often unpredictable;
• Making sure to not anthropomorphize the organization;
• Clarifying the constituent parts of the system across multiple dimensions including policies, cultural norms, routines, structures, artifacts, and practices;
• Articulating which units or levels of analysis (e.g., micro, meso, and macro) are being studied, which leads to;
• Reconciling the long-standing issue in social science of the relationship between structure and agency.

Fortunately, there was a group of scholars who had considered these issues in relation to educational reform, especially drawing on theory and method from organizational and cognitive science. Researchers such as James Spillane, Cynthia Coburn, Erica Turner, and Judith Little launched research investigating the processes whereby data-related reforms were adopted, adapted, or even rejected in real-world settings.

Much of this work – known as "practice-based" research – was a response to the dominance of scholarship that focused on evaluating the effects of data-based reforms or were simply how-to guides for enacting DDDM projects and initiatives. This approach was problematic given mounting evidence that reforms were having little impacts, which led to the realization that the field had been relying on a "black box" to reform – focusing on inputs (reform strategies) and outputs (student outcomes) – while ignoring how complex organizational systems inevitably impact the processes of reform implementation in an often non-linear and unpredictable fashion.

These new studies viewed data use as an interactive affair that implicated technology, policies and rules about data, cultural norms, leadership, and routinized practices. They adopted a view of organizations as complex, multi-level socio-technical systems. But these scholars placed individual (and small group) cognition at the center, arguing that the sense-making process as it unfolds in these complex systems was the key to understanding DDDM.

One of the implications of this stance was that a common strategy in systems-oriented research – to list influential contextual factors or create charts of inter-connected systems components – were less than useful. Instead, scholars such as Coburn and Turner aimed to "move beyond a list of contextual conditions to begin to specify the relationship between these contextual conditions on the one hand and the process of data use on the other."

We took these insights and ran with them, tracing how STEM faculty and administrators actually used data in their daily work, and most importantly, the factors that most impinged upon their decision-making processes. This resulted in an incomplete accounting of a system, but one that illuminates those elements – across dimensions and levels – that are most influential (whether in a positive or negative sense) to educational practice. (For an example of our approach, see: http://stemeducationjournal.springeropen.com/articles/10.1186/s40594-014-0008-2). This approach fit our purposes for this study on DDDM, and I suggest that it also holds great promise for the types of systemic reforms that the ASCN community is interested in facilitating in colleges and universities across the country.

References

Coburn, C. E., & Turner, E. O. (2011). Research on data use: A framework and analysis. Measurement: Interdisciplinary Research & Perspective, 9(4), 173-206.

Hora, M. T., & Hunter, A. B. (2014). Exploring the dynamics of organizational learning: identifying the decision chains science and math faculty use to plan and teach undergraduate courses. International Journal of STEM Education, 1(1), 1.

Suggested citation

Hora, M. T. (2017, January 6). Insights into systemic reform from K-12 research on data driven decision-making [Blog post]. Retrieved from http://serc.carleton.edu/ASCN/posts/decision_making.html