Initial Publication Date: August 9, 2023

Initiatives Editorial Board and Rubric

The Teaching Evaluation Change Initiatives repository serves as a searchable collection of teaching evaluations evaluated on how well they are aligned with systemic change. This repository of teaching evaluation change programs and innovations is intended to support the national community of change agents working toward changing the evaluation of faculty teaching to better align with best practices for inclusive and equitable STEM higher education. Teaching initiatives are reviewed regularly by the Repository Editorial Board.

Curious to learn more about the development and purpose of the repository? Learn more about the perspectives of the editorial board on the ASCN blog.

Repository Editorial Board:


Christine Broussard
University of La Verne
Stephanie Salomone
University of Portland
Sharon Homer-Drummond, PhD
Tri-County Technical College
Carlos Goller
North Carolina State University


Evaluation Process

The editorial board uses a rubric to evaluate each teaching evaluation. Each teaching evaluation change initiative is rated out of a total of 18 points.

There are six areas of evaluation criteria, each with...

  1. Content quality - by what metrics has the effort impacted change?
  2. Inclusiveness - materials use asset-based language and include all types of learners and teachers
  3. Alignment between identified audience and the materials shared
  4. Scalability and Context-Dependence: can materials or approach be translated to other institutions?
  5. Alignment with ASCN Working Group 6 mission. Has an identifiable change theory basis?
  6. Is this a stand-alone resource? Can it be used as is, with embedded links to needed documents, rubrics, resources, etc? Includes resources for others

Rubric

Criteria Level 3 (Exceeds Expectations) Level 2 (Meets Expectations) Level 1 (Not yet, please incorporate feedback and resubmit)
Content quality - by what metrics has the effort impacted change? Successful or reflection on why it didn't work? Use case/context (size of class, setting) information is complete? Current relevance?

1- Resource has demonstrated impact: data have been collected in a systematic fashion (implemented >1 section/term, >1 user, data gathered), analyzed, and review by peers/experts in a formal process (e.g. publication by conference or journal)

2- Reflection on why intervention did or did not work: intervention did have an impact, reflection on critical elements of success; intervention did not have a positive impact, resource includes reflection and analysis of possible elements that contributed to failure;

3- Complete context is described: size of class, setting, institutional type(s);

4- resource relevance is clear to the aims of WG 6 mission [effective evaluation and reward of teaching].

1- Resource has demonstrated impact: data have been collected in a systematic fashion, analyzed, but not yet vetted by peers/experts

2- Reflection on why intervention did or did not work: missing or incomplete

3- Complete context is described:

4- resource relevance can be inferred

Missing one or more of the following:

1- Resource has demonstrated impact:

2- Reflection on why intervention did or did not work:

3- Complete context is described:

4- resource relevance is clear or can be inferred

 

OR

Not presented in a clear and professional manner
Inclusiveness -materials use asset-based language and include all types of learners and teachers  Directly addresses how the actions and resources promote belonging and inclusion. (i.e. has a section that specifically addresses this). Directly and comprehensively recognizes a wide variety of learners and educators, purposefully developing an inclusive environment for them. Indirectly addresses inclusiveness. Does not have a specific section, but contains DEI approaches. (e.g. references belonging, loosely focuses on marginalized students).  Resource describes creating a welcoming environment, but doesn't recognize more than a subset of learners and educators. Resources do not address inclusion yet, or do not use approaches to maximize belonging. 
Alignment between identified audience and the materials shared Clear and correct alignment between the audience identified by the submitter and the materials, with rationale for that choice of audience. Clear and correct alignment between the audience identified by the submitter, and the intended audience of the materials. No clear rationale for that choice of audience included. Audience is not identified or is clearly misidentified. Feedback to the submitter should include which audiences should/should not be included and why, with an ask to amend their identification.
Scalability and Context-Dependence: can materials or approach be translated to other institutions? Can all elements be used by others or only selected elements? Materials and/or approach can be scaled and adopted in different institutional contexts. There is detailed information about ways in which adoption and/or scaling might be accomplished. Materials and/or approach can be scaled and adopted in different institutional contexts. There is limited information about how to accomplish adoption and/or scaling in different institutional settings.do so. Materials and/or approach are deeply tied to context, so they cannot be scaled or adapted to different institutional cultures, sizes, or contexts. Feedback to the submitter should include questions about context-dependence, possibly including suggestions for other publication options such as a blog post or case study.
Alignment with ASCN Working Group 6 mission. Has an identifiable change theory basis? Focuses on improving teaching of instructors in STEMM; specifically and directly acknowledges foundation in change or pedagogical theory; has actionable and measurable policies or practices toward changing culture of teaching; acknowledges a diverse array of instructor/faculty responsibilities   Appears to have some aspects that relate to systemic change and change theory, but is not clearly stated or cited Focuses on punitive or proforma ("checkbox") evaluation of teaching; does not include implicit or explicit mention of change theory or systemic change at departmental or institutional level; needs significant improvement to demonstrate the value of continuous improvement of teaching 
Is this a stand-alone resource? Can it be used as is, with embedded links to needed documents, rubrics, resources, etc? Includes resources for others The learning resource is a stand-alone resource that can be readily transferred to different courses, learning designs and contexts without modification. It operates effectively with a broad range of learners by adapting content or providing adjunctive content such as glossaries and summaries of prerequisite concepts. The learning resource in its current form requires refinement in one or more of the following characteristics: 1) It refers to the module, course or instructor for which it was originally designed; 2) Its use is critically dependent on specific, external learning resources; and 3) It can only be used by a small group of learners with highly specialized prerequisite knowledge. The potential to nurture the submitted learning resource to maturity is significantly compromised by major deficiencies in the following characteristics: 1) Dependence upon specific, external learning resources. 2) Requirement of highly specialized knowledge. 3) lacks specific implementation instructions.