SUMMATIVE EVALUATION PLAN University of Virginia (U.VA.) CHARGE Prepared by External Evaluator, Emorcia V. Hill, PhD February 13, 2015

Program Goals The U.VA. Charge initiative includes five goals and associated initiative as indicated below:   

 

Goal 1: Departmental Diversity: To strengthen and support departmental efforts to create a positive environment for all faculty with an emphasis on women Goal 2: Recruitment & Hiring: To increase the gender diversity of STEM/SBS departments Goal 3: Tournament of Ideas: To generate & implement creative, grassroots solutions from across grounds to barriers preventing the hiring & retention of women STEM/SBS faculty Goal 4: Enhancement Grants: To provide departments and women STEM/SBS faculty with resources to support hiring, recruiting, retention and professional development Goal 5: Voices & Visibility: To increase the sense of belonging of STEM/SBS women faculty among their schools and departments

Purpose of the Summative Evaluation The proposed summative evaluation will be conducted between February 1, 2015 and August 2017, which is during the later stages of the program’s life cycle as it becomes an established and mature program. Given its stage, the evaluation will focus on program products, results or impact that includes outcome and impact evaluation. For outcome, the overarching question is whether the program is achieving its goals and objectives. For impact, the overarching question is what are the program’s predicted and unpredicted impacts. Summative questions relate to the results and benefits produced for participants as a consequence of the project’s stated goal. The summative evaluation will provide evidence about the program’s worth or merit, demonstrate results to funders and describe the quality and effectiveness of the program. Thus, the summative evaluation is distinct from the formative evaluation with its ongoing focus on program activities; outputs and short-term outcomes for progress monitoring and mid-course 1

corrections; and overall program improvement. With its attention to process and implementation, its overarching question is whether the program is operating as planned. Formative process questions generally relate to project performance in terms of implementation factors, project management and administration and outputs generated. Of necessity, to contextualize and more fully understand the findings, the summative evaluation will draw on the documentation produced by the formative evaluation activities.

Conceptual Framework Developmental Evaluation

Developmental Evaluation supports innovation development to guide adaptation to emergent and dynamic realities in complex environments. While Patton 2006 positions the roles and relationships of this approach as largely an internal team function, which is more appropriately aligned with the formative (internal) evaluation, the summative evaluation will, as feasible, adopt standards that are supported by this approach. These standards will include systems thinking; creative and critical thinking balance; team and people skills; and ability to facilitate rigorous evidence-based perspectives. Theory of Change (from program document)

Change at U.Va. is premised on 3 concepts: 1) Change happens in 5 domains—faculty demographic diversity, recognition and attention to historical legacy, psychological, behavioral, and structural— (Hurtado, Milem, Clayton-Pedersen, & Allen, 1999; Milem, Chang, & Antonio, 2005), and the program will operate in these domains. 2) The transformation of male privilege demands cultural change and the engagement of STEM/SBS men as allies (Heifetz and Laurie, 1997). 3) This type of cultural change may disrupt strongly held values and beliefs, does not always have immediately discernible solutions, and requires mobilizing diverse constituents in non-linear and adaptive strategies best accomplished through selfreflection and discovery (Heifetz and Laurie, 1997; Kezar, 2001). Self-reflection, discovery, and reorganization processes take longer and require non-linear evaluation frameworks and intense qualitative evaluation data.

Evaluation Framework and Technical Approach A mixed-method evaluative approach will be used that combines both quantitative and qualitative methodologies, and primary and secondary data sources. The evaluation will be conducted conterminously with the remaining implementation period of the project goals and related activities. The evaluation incorporates a multi-level system approach to anticipate institutional transformation and change.

2

In terms of applying the theory-of-change, the evaluation will treat Hurtado et. al. 5 levels as domains of change and as the primary/overall perspective. The cultural changes suggested by Heifetz and Laurie will be positioned within the domains as secondary and that provide a model for effecting change. The evaluation framework for U.VA. provides examples of guiding questions, multi-level (individual, departmental, institutional and program) measures and indicators, data collection and respondents for each of the five goals and activities supporting those goals. Note that the measures and indicators will reflect the dimensions of the theory-of-change and become the basis for items on the data collection instruments. Measures and indicators that capture changes in faculty demographic diversity, recognition and attention to historical legacy, psychological, behavioral, and structural will be included. These cross-cutting measures will also relate to the circumstances and context in which the program is being implemented and the results are produced. Context includes factors related to the U.VA. culture as expressed via its policies, programs, practices, and procedures, and the lived experiences of its faculty. Context also includes the external factors that may affect the culture and/or implementation of the program through priorities at the university. Goal 1: Departmental Diversity: To strengthen and support departmental efforts to create a positive environment for all faculty with an emphasis on women.

  

Objective 1: STEM/SBS department policies and practices for recruitment, hiring, and retention of women faculty reflect best practices that address issues faced by these faculty Objective 2: STEM/SBS departmental cultures are flexible and welcoming to diverse faculty Objective 3: Build the capacity of STEM/SBS Chairs & Deans in creating and managing more gender diverse departments

Evaluation Questions: What are the separate and combined effects of policies and practices to

create a positive and welcoming environment for all faculty with an emphasis on women? What new capacities (affective, cognitive behavioral and administrative/operational) do chairs and deans demonstrate? What individual, departmental, institutional and contextual factors are at play and what is the interrelationship among them? Measures and indicators:

Individual results—Women describe their departments as equitable. Newly hired women faculty describe the department as welcoming and equitable. New faculty report the P&T process is transparent and equitable. Deans and chairs enhanced capacities and changes in departments. 3

Department/College results—Policies and practices that support women faculty. Search committees increase the diversity of their interview pool. Search committees increase the diversity of their applicant and candidate pools. Search committee members increase their bias literacy. More offers made to female candidates and the content of the offers. Sex and URM ratios are balanced in departments. Institution results—Policies and practices that support women recruitment. Gender representation in recruitment pool. Comparable recruitment numbers for women and men. Cultural shifts in attitudes towards women faculty. Program results—Technical assistance and protocols for search committees. New policies and practices are followed by P&T committees. Faculty and administrators increase their bias literacy. Goal 2: Recruitment & Hiring: To increase the gender diversity of STEM/SBS departments.

 

Objective 1: STEM/SBS department policies and practices for recruitment, hiring, and retention of women faculty reflect best practices that address issues faced by these faculty Objective 2: Build the capacity of STEM/SBS Chairs & Deans in creating and managing more gender diverse departments

Measures and indicators:

Individual results—Women describe their departments as equitable. Newly hired women faculty describe the department as welcoming and equitable. New faculty report the P&T process is transparent and equitable. Deans and chairs enhanced capacities and changes in departments. Department/College results—Policies and practices that support women faculty. Search committees increase the diversity of their interview pool. Search committees increase the diversity of their applicant and candidate pools. Search committee members increase their bias literacy. More offers made to female candidates and the content of the offers. Sex and URM ratios are balanced in departments. Institution results—Policies and practices that support women recruitment. Gender representation in recruitment pool. Comparable recruitment numbers for women and men. Cultural shifts in attitudes towards women faculty. Program results—Technical assistance and protocols for search committees. New policies and practices are followed by P&T committees. Faculty and administrators increase their bias literacy. Special Note: There is substantial overlap between the elements of Goals 1 and 2, which will

need to be addressed.

4

Goal 3: Tournament of Ideas: To generate & implement creative, grassroots solutions from across grounds to barriers preventing the hiring & retention of women STEM/SBS faculty.



Objective 1: Tournaments will identify the primary reasons why women STEM/SBS faculty do not accept positions at U.VA. or remain at U.VA. Objective 2: Tournament solutions will foster coalitions of faculty and staff to implement and resolve the identified issue



Evaluation Questions: What creative, grassroots solutions to barriers preventing the hiring &

retention of women STEM/SBS faculty were generated from across Grounds and implemented? What coalitions were established and what are their impacts? What individual, departmental, institutional and contextual factors are at play and what is the interrelationship among them? Measures and indicators:

Individual results—Change in perception of faculty and staff that this problem is solvable. Department/College results—Types of ideas adopted for implementation and effects on department policies and practices. Increased number of women STEM/SBS candidates accept positions and associated factors. Higher retention of women STEM/SBS faculty and associated factors. Sex and URM ratios in departments balanced. Institution results—Enhanced sensitivity and receptivity to women’s voices in policies and practices. Program results—Key elements for promoting and engaging in tournaments defined as a replicable “model” and /or best practice. Goal 4: Enhancement Grants: To provide departments and women STEM/SBS faculty with resources to support hiring, recruiting, retention and professional development.

 

Objective 1: Identify and recruit highly qualified women TTT candidates in STEM/SBS Objective 2: Advance women STEM/SBS faculty careers

Evaluation Questions: What types of grants were provided to support hiring, recruiting, retention

and professional development? What was the level of utilization by whom? What were the effects/impact on hiring, recruiting, retention, professional development and overall career advancement? Measures and indicators:

Individual results—Greater faculty retention among women who received grant. Increased number of women faculty receiving grants receive tenure. Work produced has impact on their field. Department/College results—More offers made to female candidates. Sex & URM ratios are balanced in departments. Institution results—Professional visibility of women. Valuing of women’s scholarship. Program results—Expansion and institutionalization of enhancement grants. 5

Goal 5: Voices & Visibility: To increase the sense of belonging of STEM/SBS women faculty among their schools and departments.

 

Objective 1: Reshape the sense of place to one of inclusion, recognition for STEM women faculty Objective 2: Identify and limit barriers to women’s sense of belonging, empowerment, and participation on grounds

Evaluation Questions: To what extent and in what ways was the sense of place reshaped to

one of inclusion, recognition for STEM women faculty? What barriers to women’s sense of belonging, empowerment, and participation on grounds were identified and how were they limited? Measures and indicators:

Individual results—Women STEM/SBS faculty report feeling recognized and empowered after participating in OH or photo exhibit or symposium. Women STEM/SBS faculty report increased productivity (use of labs, etc. after improvements to the physical environment. Department/College results—Barriers are provided to equity consultants, structured dialogues, and Deans to generate solutions and discussion. Increased retention of female faculty in STEM/SBS and factors to which attributed. Institution results—University takes action to improve sense of belonging (social science research: the physical environment for women in STEM/SBS areas of grounds). Women STEM/SBS faculty feel a greater sense of belonging at U.VA.. Program results—Key elements for promoting and engaging visibility of women in STEM defined as a replicable “model” and /or best practice. Evaluation for Social Science Study Evaluation Questions: What types of data were collected using what tools and how was the

data utilized? To what extent were rigorous research methodologies and methods applied? Measures and indicators:

Individual results—Participation in surveys and /or interviews. Familiarity with summary reports. Benefits derived from knowledge produced by survey findings. Department/College and Institution results—Level of faculty participation. Level of administrative involvement and support. Awareness of issues facing women and proactive solutions. Utilization of data for action agenda. Program results—Overall quality/rigor of research study. Institutional participation. Dissemination and utilization of results. New research questions formulated. Longitudinal data base. New area of research and variables for STEM women’s success and advancement in the academy.

6

Data Collection Instruments As noted above a mixed-method evaluative approach will be used that combines both quantitative and qualitative approaches, and primary and secondary data sources. Primary will include  Survey  Structured Interview Guide  Focus Group Guide Secondary will include  Extant/existing Program Data In the design of instruments, the evaluation will utilize data collected and/or reports produced by the internal and external evaluator, and the U.VA. program management team. Items from prior summative activities and resources will be utilized, as deemed feasible and appropriate. Data Collection and Respondents

Primary data will be collected annually, during the spring semester (or during scheduled site visits by the external evaluator) via surveys hosted by Survey Monkey or a comparable platform. Secondary data sources will include data collected by the U.VA. program management team, the internal evaluators, and NSF programmatic reporting data sources. Data collection and respondents include surveys and interviews with faculty, university administrators and program personnel. Review of relevant documents and other extant data including technical notes. Medium include:  Online/electronic for surveys  Site visits at scheduled intervals, likely annually. Data Analysis

Separate and combined quantitative data analysis will be conducted for each of the five components. To understand the U.VA. context, the quantitative analysis will examine relationships and findings across the five goals and initiatives. The data aggregation process will reflect the theory-of-change approach. Thus, an important first step will be the extraction and categorization of the relevant contextual factors that may be involved in producing the observed results and outcomes for the various stakeholder constituencies. Univariate analysis will look at distributions and patterns among the various measures and indicators. Bivariate analysis will examine the relationships among the measures and indicators and relevant outcome variables. Multivariate analysis will consider such issues as the relative 7

contribution of different components and activities to outcomes. As appropriate, regression analysis and HLM techniques will be used to model the relationships among variables. Reports

Annual and cumulative reports will be produced based on the data collected over the remaining period of evaluation and program implementation activities. The external evaluation report will integrate the quantitative and qualitative results and findings, as well as relevant data produced by the internal evaluation. The report will include:  Introduction  Program background  Methods and evaluation questions  Findings for each initiative  Recommendations  Conclusion Utilization of Findings

Program results and findings will be used to demonstrate results, outcomes, and impact, as benefits all stakeholder participants, to ascertain the extent to which results and outcomes can be attributed to program initiatives. In addition, the core elements of the U.VA. CHARGE model can be identified for dissemination and replication/adaptation. Responsibilities

The summative evaluation will be conducted by evaluation consultant, Emorcia V. Hill, PhD. Dr. Hill has extensive experience and expertise in the design, implementation, evaluation and research in the areas of diversity, inclusion and access; workforce development; institutional and system change. Dr. Hill’s evaluation and research work has focused on the high demand science, technology, engineering and mathematics, as well as biomedical and behavioral sciences.

ADDITIONAL TASKS    

Refine and/or add measures and indicators so that they are consonant and aligned with the theory of change. Adjust the logic model to more prominently and directly reflect the social science study. Consider for discussion “Additional Strategies to Demonstrate Impact”. Create a time line for the summative activities.

8