Articles

Reconceptualizing Evaluator Roles

American Journal of Evaluation Volume 30 Number 3 September 2009 275-295 # 2009 American Evaluation Association 10.1177/1098214009338872 http://aje.sagepub.com hosted at http://online.sagepub.com

Gary J. Skolits Jennifer Ann Morrow Erin Mehalic Burr University of Tennessee

The current evaluation literature tends to conceptualize evaluator roles as a single, overarching orientation toward an evaluation, an orientation largely driven by evaluation methods, models, or stakeholder orientations. Roles identified range from a social transformer or a neutral social scientist to that of an educator or even a power merchant. We argue that these single, broadly construed role orientations do not reflect the multiple roles evaluators actually assume as they complete the activities encompassing an external evaluation. In contrast to the current literature, this article suggests that typical evaluation activities create functional demands on evaluators, and that evaluators respond to these demands through a limited number of specified evaluator roles. This depiction of a set of specific multiple evaluator roles, generated in response to particular evaluation activities and their associated demands, has implications regarding how evaluation is conceptualized, practiced, and studied. This article concludes with a discussion of these implications. Keywords:

roles; evaluators; practice; research

T

he conceptualization of evaluator roles as currently portrayed in the evaluation literature is rooted in the development of the evaluation discipline. Beginning with the emergence of evaluation as a distinct discipline, much of the evaluation literature continues to focus on evaluation methods, models, and their applications (Alkin, 2004; Stufflebeam, 2001). Prescriptive evaluation models are still being offered to suggest the desired focus of an evaluation (Hopson, 2002), and evaluation methods literature continues to promote the design and application of rigorous research methodologies needed to establish credible evidence (Donaldson, Christie, & Mark, 2008). Evaluation models and methodologies and their applications are and will remain critical foundational resources providing evaluators with important guidance on focusing an evaluation project and enhancing its credibility. Correspondingly, the existing conceptualizations of evaluator roles, as addressed in the literature review below, have been heavily influenced by the substantial body of evaluation models and methods literature. Many evaluation theorists have identified evaluator roles based on the focus of a particular evaluation model or research methodology. Unfortunately, the selection and use of an evaluation model, or the application of a particular rigorous research methodology, does not encompass the breadth or scope of the many activities required for an external evaluation. Evaluation is a complex web of multiple activities and relationships, and even a relatively small external evaluation project entails a broad and interrelated array of events and interactions Authors’ Note: Gary J. Skolits, Department of Educational Psychology and Counseling, 1122 Volunteer Blvd, Jane and David Bailey Education Complex, Knoxville, TN 37996; e-mail: [email protected].

275

276

American Journal of Evaluation / September 2009

that create specific types of demands on the evaluator. Therefore, conceptualizing evaluator roles primarily based on a particular evaluation model, evaluation method, or stakeholder orientation does not realistically account for many of the activities for which an evaluator is responsible. Current evaluation role orientations identified in the literature are typically not well defined, and they tend to promote a single overarching evaluation role orientation for the evaluator across an entire evaluation event. However, we find that vaguely defined roles or roles that represent a single evaluation orientation are insufficient representations of the actual orientations evaluators assume when they respond to typical evaluation activities and their demands. Practicing evaluators understand that many critically important evaluation processes and activities must precede an initial determination of an evaluation’s focus or methodology, and many critical evaluation activities must occur after data are collected, analyzed, and summarized into an evaluation report. Although the evaluation literature is now expanding to addresses evaluator roles based on evaluator stakeholder relationships, this perspective also tends to promote a single evaluator role orientation that is also not highly defined or operationalized. A more ‘‘situational’’ perspective of roles is also receiving some attention, but this literature does not attempt to match contextual situations with associated role responses. Broadly construed, singularly focused evaluator role conceptualizations or the undefined situational role perspectives offered in the literature are limited in their ability to guide evaluators in the many activities and associated demands of an evaluation. The problem underlying the examination of roles addressed in this article is the need for a more realistic and useful conceptualization of evaluator roles, roles evaluators actually tend to assume in response to the specific activities common to external evaluations. In response to this problem, the specific purpose of this examination is to reconceptualize roles based on evaluator responses to the common activities of an evaluation and the particular demands these activities tend to place on an evaluator. Three objectives guided this examination: (a) delineation of the common activities and associated demands of a typical external evaluation event, (b) reconceptualization of roles based on evaluator responses to evaluation activities and demands; and (c) specification of how this more realistic understanding of evaluator roles can help guide future evaluation practice, training, and research. In addressing these objectives, the authors drew upon decades of experience in conducting evaluations and related responsibilities for the classroom and field training of novice evaluators. Although we argue that the specification of multiple evaluator roles based on actual evaluation activities offers a more complete and realistic depiction of evaluator orientations to the evaluation process, the approach underlying this examination of evaluator roles is also acknowledged to be experientially based. The authors’ previous experiences with dozens of evaluations offered insight into evaluation activities and the demands they placed on us as the evaluator. In our practice, we observed how evaluation activities and their demands led us to activate similar patterns of responses. Over time, we began to recognize our responses to evaluation activity demands as regularly occurring patterns of behavior that we adopted in the form of roles. Although our positing of specific evaluation roles is driven by experience instead of empirical testing, our methodological approach to roles is consistent with the larger body of literature addressing roles; most existing evaluator role specifications in the literature are not empirically based. However, we hope our framework of specifically defined roles based on the demands of evaluation activities will warrant and promote future evidenced-based investigation.

Literature on Evaluator Roles Ryan and Schwandt (2002) suggested that ‘‘the concept of the ‘role of the evaluator’ is central to the theory and practice of evaluation’’ (p. vii), and there is substantial evaluation

Skolits et al. / Reconceptualizing Evaluator Roles

277

literature reinforcing this perspective. Evaluation theorists have offered a variety of perspectives on evaluator roles, and many of these evaluator role perspectives have continued to evolve to reflect new evaluation methods, models, and an increasing orientation toward greater stakeholder involvement. Although the concept of a role is central to many disciplines, it is primarily associated with the field of sociology. Turner (2006) defined the particular type of roles of most interest for this current study, functional group roles, as ‘‘un-formalized patterns that emerge spontaneously as individuals acquire situational identities during sustained interaction in a group setting. They include such roles as ‘leader,’ ‘follower,’ ‘counselor,’ ‘mediator,’ and ‘devils advocate’’’ (p. 247). He further asserted that ‘‘while roles can be viewed organizationally and societally as at least a vague framework, real persons must learn and hold role conceptions and enact roles . . . .’’ (p. 247). Recent sociological scholarship (Lynch, 2007; Montgomery, 2000) suggested that individuals are accustomed to playing a variety of roles, they change roles more frequently and fluidly than originally thought, and they are quite adept at switching roles and managing role conflicts. This more recent understanding of an individual’s ability to assume and successfully navigate multiple roles is consistent with the experiences underlying our proposed conceptualization of evaluator roles. From the perspective of evaluation, four perspectives of evaluator roles dominate the current evaluation literature. These include the identification of evaluator roles as orientations resulting from evaluators’: (a) choice of evaluation methods, (b) choice of evaluation models, and (c) desired relationships with stakeholders, and (d) contingent reactions to evolving evaluation circumstances. Common among the first three of these perspectives is the identification of a single overarching role for an evaluator, a single role that is often not particularly well defined or stated in terms that can help guide an evaluator through the many activities of an actual evaluation. The fourth perspective acknowledges the potential for multiple evaluator roles but offers few details beyond suggesting that these roles derive primarily from the changing contextual factors an evaluator faces.

Roles Driven by Evaluation Methods Some evaluation theorists suggested that traditionally an evaluator’s choice of an evaluation method determined his/her evaluator role (Greene, 2000; Mark, 2002; Weiss, 1998). Weiss (1998) asserted that the traditional and still dominant role conceptualization of evaluators is methods-based and representative of a neutral, detached social scientist: ‘‘The traditional role of the evaluator has been one of detached objective inquiry . . . . She puts her trust in methodology’’ (p. 98). Weiss also noted that role conceptualizations of evaluators were beginning to shift toward viewing the evaluator as more of a collaborator with program staff. Mark (2002) cited Campbell’s (1969) view of an evaluator’s role as a ‘‘technical servant’’ to an experimenting society as an example of the traditional methods-based understanding of an evaluator’s role. Noblit and Eaker (1987) suggested that several evaluation methodologies actually lead to specific evaluator role orientations, including positivism (credible expert role), interpretivism (social network broker role), critical theory (emancipator role), aesthetics researcher (connoisseur/critic role), collaborative researcher (broker of interests role), and action research (expert in solidarity with practitioners role). The methodological perspective of evaluator roles continues to expand as recent writers identify new roles for evaluators based on proposed new evaluation methodologies. Hopson (2002), for example, suggested a role of ethno-evaluators based on the application of ethnographic methods for evaluation.

278

American Journal of Evaluation / September 2009

Roles Driven By Evaluation Models Evaluation models help evaluators establish a focus and purpose for evaluators, and to some evaluation writers these models suggest an overarching evaluator role orientation. Traditional and more recent evaluation models tend to suggest role orientation for evaluators as judge (Scriven, 1991), supporter of decision making (Stufflebeam, 2003), one who empowers (Fetterman, Kaftarian, & Wandersman, 1996), as well as the roles of a critical friend, facilitator, and problem solver (Weiss, 1998). The role of evaluator as a supporter of organizational decision making has received substantial attention. For example, the Context, Input, Process, and Product (CIPP) evaluation model clearly establishes a role for evaluators to produce information of relevance for program decision makers. Stufflebeam (2003) suggested specifically that the evaluator’s overall orientation should largely be centered on a process of creating information to support managerial decisions. Wholey (2001) argued that emerging results-oriented management practices in public and nonprofit organizations establish a new role orientation for evaluators. Wholey suggested that an evaluator’s role should encompass ‘‘overcoming challenges that arise in results-oriented management’’ (p. 343), such as: (a) developing agreement on goals and strategies, (b) measuring and evaluating performance, and (c) using performance measurement and evaluation information. Preskill and Torres (1999) presented an organizational learning model of evaluation and suggested an overarching role for evaluators as facilitators of organizational inquiry, specifically identifying evaluator roles of collaborator, facilitator, interpreter, mediator, coach, and educator. Owen and Lambert (1998) suggested that the expanding needs of organizational leaders for expertise in organizational development have blurred evaluators’ traditional orientations toward an organizational development consultant role. Several evaluation models establish an evaluator role of change agent and societal transformer. These models promote evaluator roles that focus on issues of social improvement, social justice, and democracy (Greene, 2000; House, 1993; Mark, Henry, & Julnes, 2000; Mertens, 2002; Segerholm, 2002). A recent edited book addressing evaluator roles, Exploring evaluator roles and identities (Ryan & Schwandt, 2002), presented some of the more recent evaluation approaches that seek to advance a role for evaluators oriented toward social criticism and social transformation. For example, King and Stevahn (2002) presented a chapter that offered a model representing a continuum of evaluator purposes and associated role orientations toward social betterment and transformation. Sullins and Youker (2006) suggested that several chapters in the Ryan and Schmant (2002) text (e.g., Dahler-Larsen, 2002; Mertens, 2002; Segerholm, 2002; Stronach, Halsall, & Hustler, 2002) identified the traditional role of evaluators as unsuspecting supporters of organizational oppression.

Roles Focusing on Evaluator Relationships with Stakeholders Some evaluation models tend to focus on the role of the evaluator from the perspective of evaluation stakeholders. In this perspective, the overarching role of the evaluator is to identify relevant stakeholders for the promotion of stakeholder participation in the evaluation and subsequent use of evaluation results (Cousins & Earl, 1992; Greene, 1988). Fetterman et al. (1996) view the evaluator’s role as one of empowering stakeholders. Other writers (Cartland, RuchRoss, Mason, & Donohue, 2008; Cousins, 2001) have suggested that stakeholder involvement evaluation models tend to promote role sharing between the evaluator and stakeholders. Mark (2002) offered a typology of evaluator roles based on the relationship between the evaluator and stakeholders that identified roles such as philosopher king, technical geek, and public relations agent. Patton (2007) suggested that evaluators adopt roles from the perspective

Skolits et al. / Reconceptualizing Evaluator Roles

279

of the relationship between the evaluator and the stakeholders. For example, Patton indicated that if the primary users are funders/decision makers then the evaluator’s primary role is that of judge; for policy makers (auditor), program staff (consultant), diverse stakeholders (facilitator), program design teams (team member with evaluation expertise), program staff and participants (collaborator), program participants and community members (empowerment facilitator), ideological adherents (support of cause), and evaluation planners/users (synthesizers; p. 210-211).

Situational Generated Roles There is some recent and potentially more helpful literature suggesting a situational perspective on evaluator roles. Although Patton (2007) acknowledged the large variety of role options in the literature, identifying ‘‘collaborator, trainer, group facilitator, technician, politician, organizational analyst, internal colleague, external expert, methodologist, information broker, communicator, change agent, diplomat, problem solver, and creative consultant’’ (p. 213), he also suggested that the role of an evaluator is contingent in response to the situational conditions of actual practice: ‘‘The roles played by an evaluator in any given situation will depend on the evaluation’s purpose, the unique constellation of the conditions with which the evaluator is faced, and the evaluators’ own personal knowledge, skills, style, values and ethics’’ (p. 213). Patton also cited Morabito (2002) who suggested that evaluators select and implement multiple role orientations, as ‘‘it is unlikely that an evaluator could succeed in the implementation of an evaluation by selecting one role. Rather, the evaluator should begin a process of reflection . . . to determine the combination of appropriate roles and strategies that appear to be the most indicative of an influential evaluation process’’ (p. 328). Although these authors break new ground in suggesting multiple roles that occur in response to evaluation contexts, they do not specify which multiple roles are activated in response to particular situations. Moreover, Patton also cautions the reader that the idea of evaluators activating multiple roles remains controversial in the evaluation discipline.

Conclusion and Deficiencies in the Literature Some evaluation writers suggested that evaluator roles need further investigation (Schwandt, 2002; Segerholm, 2002). Segerholm (2002) argued that the current concept of evaluator roles is unrealistic: ‘‘To think about the roles of the evaluator as pre-ordained or desirable behavior or ways to relate to people affected by the evaluation does not bring forward the necessity for the evaluator to make a critical analysis of the evaluation, the evaluation context, and her own part and position in the evaluation process and contexts.’’ (p. 98). This assertion lends support to the concept of multiple evaluator roles and reflects some of the major weaknesses in the current roles literature. Most of the current conceptualization of roles suggests broad evaluator orientations driven solely by evaluators’ decisions regarding an evaluation model, method, or targeted stakeholder involvement level. Evaluator roles are generally assumed to remain constant throughout an evaluation. Existing evaluator role depictions are typically not well defined or operationalized. Although more recent literature suggests that evaluators adopt multiple roles in response to particular evaluation contexts, it provides little guidance in linking evaluation contexts to any specific roles. Moreover, none of the role approaches reviewed specified how evaluation activities may lead to specific evaluator roles; yet, in practice, evaluators often find that they change and enact specific roles in response to the demands of specific evaluation activities.

280

American Journal of Evaluation / September 2009

Table 1 Assumptions of the Evaluation Activity-Based Conceptualization of Evaluator Roles    

 

  



Evaluation is a complex process, and a realistic role conceptualization of external evaluation should encompass the broad array of interrelated, multifaceted activities that are part of the evaluation process. Common to all external evaluations is a focus on improvement (formative perspective) and/or the determination of worth existing in a program (summative perspective). To address the common purposes of an external evaluation, evaluations are comprised of typical evaluation phases (i.e., the three phases of pre, during, and post evaluation). Each phase of an external evaluation incorporates a set of evaluation activities that need to be accomplished for an evaluation to progress to the next phase and on through eventual fruition. These activities can be anticipated to some extent during each phase of an evaluation Each evaluation activity presents specific types of demands on the evaluator. These demands can be technical, interpersonal, and/or intrapersonal. Evaluators address each these evaluation activity demands through a specific dominant evaluator role. While one role is dominant in response to a particular evaluation activity demand, other secondary roles may also need to be active activated. (For example, an evaluator is always responsible for the administrative functions of an evaluation, regardless of which demands, and associated roles, are active at any one time). Evaluators are likely to assume multiple roles at any time during an evaluation due to the complexity of the requirements of a single evaluation activity and/or the overlapping of evaluation activities. Evaluation roles actually begin when an individual prepares to offer evaluation services, at which time they encounter the initial and continuing demands for managing an evaluation process. The role of manager includes two functions: 1) a traditional management function for supporting the evaluation and 2) a role coordination function determining and managing role responses to all the demands of evaluation activities. While evaluations typically proceed through anticipated phases and activities in a sequenced manner, changing evaluation circumstances may require changes in the anticipated sequences of evaluation activities. Regardless of when evaluation activities occur in the sequence of an evaluation, these activities will create demands requiring corresponding role responses.

An Evaluation Activity-Based Conceptualization of Evaluator Roles Instead of viewing roles from a macro orientation such as an evaluator’s overall purpose, choice of methodology, or an intended relationship to stakeholders, this article suggests that roles are more realistically viewed from the perspective of the common activities that an evaluator engages in throughout an external evaluation process. This conceptualization suggests that the inherent common activities of an evaluation create specific demands on evaluators. In response to these demands, evaluators tend to assume specific role responses. Accordingly, this understanding of roles is built on a framework with three primary elements: 1.

2.

3.

Evaluation Activities—Most evaluations are based on a common set of evaluation activities. (These evaluation activities ultimately drive evaluator roles, and as such, the delineation of specific common activities in an external evaluation forms the basic determinants of roles). Evaluation Demands—Each common evaluation activity creates specific demands on an evaluator. The nature of these demands sets the stage for the types of role responses the evaluator can be expected to activate. Evaluator Role Responses—In response to each evaluation activity demand, evaluators assume a particular dominant role response; although other secondary roles will likely be activated in support of the dominant role. This process results in a limited and definable set of role responses common to all evaluations.

Table 1 presents several specific assumptions of this evaluation activity-based perspective of roles. A key assumption underlying this proposed conceptualization of evaluator roles is that

Skolits et al. / Reconceptualizing Evaluator Roles

281

regardless of an evaluator’s macro orientation to an evaluation (e.g., change-agent, empowerer, compliance officer, neutral social scientist, etc.) or choice of methodology, common evaluation activities occur in most external evaluations. It is these common activities and their associated demands that drive multiple evaluator roles. The model also realistically acknowledges the inherent complexity of an evaluation in that multiple evaluation activities can occur at anytime in an evaluation and therefore so can multiple roles. Moreover, even for single evaluation activities, this approach to roles acknowledges that evaluators tend to assume a primary role as well as activate one or more secondary roles in response to specific activity demands.

Evaluation Activities Evaluation can be viewed as a process that occurs within 3 phases (pre-evaluation, active evaluation, post-evaluation) and 10 common, sequenced evaluation activities. The designation of 3 evaluation phases and most of the 10 activities identified is generally consistent with the literature (Fitzpatrick, Sanders, & Worthen, 2004; Kellogg Foundation, 1998; Preskill & Torres, 1999; The Joint Committee on Standards for Educational Evaluation, 1994). However, in our depiction the pre-evaluation phase activities and associated demands actually begin with an individual’s preparation to conduct evaluations (which we specifically address below). The pre-evaluation phase activities lead to an agreement between the evaluator and a client for an evaluation to be conducted, while the active evaluation phase activities reflect the core activities of concluding evaluation reporting. Finally, the post-evaluation phase activities conclude the evaluation process with the promotion of evaluation use and subsequent evaluator reflection on the evaluation experience. This depiction includes the following: Pre-Evaluation Phase 1. Evaluator preparation to conduct an evaluation 2. Initial contact 3. Evaluation planning 4. Evaluation contracting Active Evaluation Phase 1. Initial evaluation implementation 2. Evaluation data collection 3. Evaluation judgment 4. Evaluation reporting Post-Evaluation Phase 1. Promoting evaluation use 2. Evaluation reflection

Evaluation Demands Each common evaluation activity generates demands, and evaluators seek to address these demands through specific role responses, including one primary role and potentially several secondary roles (Table 2). The identification of the specific demands resulting from each common evaluation activity provides insight into the types of evaluator roles generated to address these demands. For external evaluations, a typical set and sequence of common evaluation activities can be anticipated, along with an associated set of corresponding demands created

282

Negotiator

Evaluation contracting

Detective

Designer

Pre-Evaluation Phase

Initial client contact

Manager

Primary Role Responsesa

Evaluation planning

All Phases

Evaluation Phases

Evaluation managementb

Evaluation Activity

Creating agreement on an evaluation contract by evaluator and client

Developing a realistic and responsive evaluation design to address the client’s need

Determining evaluation need and alignment with evaluator skills, competencies, and interests

Addressing evaluation administration and role coordination

Primary Role Brief Description

Table 2 Evaluation Activities, Phase, and Evaluator Role Responses

Manager Diplomat Use advocate Manager Researcher Reporter Use advocate Other roles Manager Designer Other roles

          

(continued)

Diplomat All roles

 

Potential Secondary Rolesa

282 American Journal of Evaluation / September 2009

Reporter

Use Advocate

Learner

Evaluation reporting

Evaluation use advocacy Post-Evaluation Phase

Evaluation reflection

Reflecting on the evaluation to improve future practice

Promoting and supporting improvement and change

Sharing evaluation results with appropriate stakeholders

Making evidence-based judging of worth/opportunities for improvement

Collecting/analyzing reliable and valid data

Establishing trust and rapport with stakeholders

a. Each primary role can also be activated as a secondary role in support of primary roles required for each evaluation activity. b. The role of evaluator manager/role coordinator is active throughout the evaluation.

Judge

Evaluation judgment

Diplomat

Researcher

Active Evaluation Phase

Data Collection/analysis

Initial implementation

Manager Researcher Reporter Use advocate Manager Reporter Diplomat Manager Researcher Manager Diplomat Researcher Manager Reporter Diplomat Other roles All roles

                

Skolits et al. / Reconceptualizing Evaluator Roles 283

283

284

American Journal of Evaluation / September 2009

by these activities. For some evaluation activities, there are numerous broadly focused demands to be addressed, and this is especially the case with regard to the activity of evaluation management. In contrast, other evaluation activities, such as evaluation planning, introduce specific demands that have a much narrower focus. Similar to the predictability of evaluation activities, the demands associated with each activity can also be predicted to some extent, and it is the predictability of the evaluation activities and their specific demands that establish the foundation for a common set of evaluator roles.

Evaluation Role Responses to Evaluation Activities and Their Demands It should be noted that for the purpose of clearly presenting and explicating the basic common evaluation activities, demands, and role responses, the following discussion is presented in the form of a relatively straightforward evaluation process that follows a predictable, sequential path of activities. However, the evaluator role conceptualization we are proposing is not at all restricted to a simple sequential evaluation event. Each basic evaluation activity, whether it occurs in an unanticipated sequence or in conjunction with one or more other evaluation activities, creates its own set of demands that generate an associated evaluator role response. Evaluations that do not follow a predictable sequence or that have many overlapping activities create more complexity for the evaluator, but the basic mechanisms of the model (evaluation activity, demands, and evaluator role response) still apply, albeit in a manner that will reflect the complexity of the simultaneous multiple tasks at hand. Evaluators in these instances will need to manage and juggle a greater number of roles. For this reason, the very first evaluator role activated, and the only role that remains active throughout the entire evaluation process, is that of manager. The following section will describe each of these activities, the associated demands, and the associated primary and likely secondary role responses.

Pre-Evaluation Activities, Demands, and Role Responses Evaluation Activity One (Preparation to Serve as an Evaluator) and its Demands The first evaluation activity we define as an evaluator’s initial preparation and readiness to conduct an evaluation. Preparation to conduct evaluation begins the demands related to the overall coordination of an evaluation, which lasts throughout all evaluation phases. This activity creates two types of demands on the evaluator: (a) demands for executing the administrative elements of an evaluation; and (b) demands for the coordination of specific evaluator role responses to each evaluation activity/demand encountered. Primary Role Response—Manager. As soon as individuals begin to prepare and become ready to provide services as an evaluator, they encounter demands that they respond to in the role of manager. Two types of demands drive this overarching management role. The first set of demands presents the evaluator with responsibility for the traditional types of administrative support required of any organized, complex, and multifaceted project. These traditional management functions include personnel acquisition and supervision, budgeting and the provision of accounting and payroll services, provision of physical resources, as well as ensuring that the quality of products and services promised are developed and delivered on a timely basis. While different elements of this demand will occur through the evaluation, the activity of preparing to be available as an external evaluator initiates these demands.

Skolits et al. / Reconceptualizing Evaluator Roles

285

While managerial activities are critical functions underlying an evaluation, they are not sufficient for the successful completion of an evaluation. Evaluators encounter initial demands when they prepare to conduct evaluations for ensuring that the multiple roles required to address the evaluation activity demands encountered throughout an evaluation are activated and coordinated. It is the responsibility of the role coordinator function of the manager to determine, activate, and manage the particular role responses to meet the demands of specific evaluation activities. Because multiple roles (i.e., a primary role and one or more secondary roles) will be active in response to the demands of each evaluation activity, the manager’s role needs to coordinate and integrate multiple evaluator roles throughout an evaluation. Both administrative and role coordinator functions of the manager are essential elements of an evaluation. Although an individual can possess the skills of a competent manager, they cannot be a competent evaluator unless they can also provide the specialized evaluator skills, knowledge, and other attributes of evaluator roles required to fulfill the specific demands of each evaluation activity. Secondary Role Responses. Because the manager’s role is integrated with role activation and coordination, the manager’s role supports every evaluator role, and every evaluator role in Table 2 can be activated in support of the manager’s role.

Evaluation Activity Two (Initial Client Contact) and its Demands The second common evaluation activity, initial contact by a potential evaluation client, generates demands for the evaluator to conduct an investigation of the client’s evaluation needs to determine whether there is a potential for a productive evaluator–client relationship. Primary Role Response—Detective. The primary role the evaluator assumes in response to initial client contact activity demands is that of detective. In this role, the evaluator seeks to understand the evaluation problem and related contextual circumstances such as, the program’s operations, staffing, the relevant stakeholders and beneficiaries, as well as the potential client’s expectations. The evaluator also seeks to determine whether there is a potential for viable evaluation with appropriate standards for making meaningful formative or summative judgments. Beyond assessment of a program’s evaluability, the evaluator seeks to analyze the needs and expectations of the client from the perspective of their congruence with the evaluator’s interests, competencies, and capacities. Similar to a detective, the evaluator must probe for and piece together relevant pieces of evidence, some of which are not always readily apparent. In this detective role, the evaluator asks questions, reviews documents, makes observations, and seeks other clues that will provide insight regarding the program, its context, and its suitability for an evaluation. This detective role may be brief or it may take substantial time and entail a more detailed review of program data and documents as well as multiple interactions with program managers, staff, and relevant stakeholders. Because determining how a program operates and what related contextual elements are relevant can be extremely challenging, the role of detective is necessary. More often, this role requires substantial investigative skill and effort as clients and stakeholders either do not fully understand their evaluation need and context, or are not able to articulate these clearly. When the demands associated with this role are addressed, the evaluator should have substantial knowledge about the program’s context and operations, key stakeholders, the specific evaluation need or problems (i.e., formative or summative) to be addressed, appropriate standards for making judgments, as well as the client and stakeholders’ expectations and likelihood of their use of evaluation results. The evaluator should also have a basic sense of expected evaluation

286

American Journal of Evaluation / September 2009

deliverables, time frames, and available evaluation resources. This is often an incredible amount of information to piece together, analyze, and use in what is typically a short period of time. Potential Secondary Roles. Several potential secondary roles may need to be activated in support of the detective role at the initial client contact stage. Certainly a level of diplomacy exemplified by the diplomat role can be useful during the initial client and program stakeholder interactions. The manager’s role is also likely to be activated because part of the discovery process must address the sufficiency of resources available to adequately address the evaluation problem and expectations. Another potential secondary role would be the evaluation use advocate role, as the evaluator seeks clues regarding the potential receptiveness of stakeholders to appropriate evaluation use.

Evaluation Activity Three (Evaluation Planning) and its Demands The third common evaluation activity, evaluation planning, establishes demands on the evaluator to develop a specific, detailed evaluation plan for addressing the particular evaluation needs and expectations of the client. Primary Role Response—Designer. To address evaluation design demands established by the activity of evaluation planning, the evaluator assumes the role of a designer. In this role, the evaluator seeks to design an evaluation plan that would enable the evaluator to provide the client with the types of information and services needed to address the evaluation problem to either improve operations (formative perspective) or determine program value or worth (summative perspective). In this role, the evaluator designs an evaluation study, including the specific purpose and guiding evaluation questions, the associated research methods to be used, the data to be collected, and the related strategies for data analyses, judgment, and reporting. The evaluator as designer would seek to ensure that sufficient human resources with appropriate skill sets are available to conduct the evaluation. As part of this design role, the evaluator creates a preliminary budget and timeline along with an identification of specific evaluation deliverables. At this stage of the evaluation process, the design being developed is cast in the form of an initial or draft evaluation plan to responsively address the evaluation problem. The evaluation design is then submitted for client review and subsequent negotiation. Potential Secondary Roles. The potential secondary role that would most likely be activated during the design phase of the evaluation is that of manager, because the initial evaluation plan must reflect realistic human resource, budgetary, and scheduling commitments. Elements of the researcher’s role would also be appropriate in support of the research and data collection aspect of the evaluation. However, several other secondary roles may need to be activated, such as the reporter and evaluation use advocate roles because evaluation reporting and use of evaluation results must be given consideration at this design phase of the evaluation.

Evaluation Activity Four (Evaluation Contracting) and its Demands The fourth common evaluation activity, evaluation contracting, creates demands on an evaluator to negotiate with the evaluation client to codify the evaluation design, deliverables, and timetables into a formal agreement.

Skolits et al. / Reconceptualizing Evaluator Roles

287

Primary Role Response—Negotiator. In response to these demands, the evaluator assumes the role of negotiator. The negotiator role is foundational to an evaluation event. Without an agreement, there is no basis for an evaluation. The agreement specifies commitments of the evaluator as well as the client throughout the evaluation process. The draft evaluation plan prepared through the designer role is the starting point for the negotiations. Should the initial draft plan be acceptable, little negotiation may be necessary. However, every element of the plan is subject to client negotiation and modification for a final mutually acceptable evaluation contract. The negotiation process may address more business-related elements such as the evaluation fee, evaluation deliverables, and the time frame and/or it may address more technical issues of evaluation questions, research design sharing and collection, access to beneficiaries, and other issues. Sometimes negotiations may require substantial revisions to the proposed evaluation design, depending on the clients’ perspective of the suitability of the plan. Oftentimes, clients are surprised to see the specification of client responsibilities in the evaluation plan such as the provision of specific types of program data to the evaluation, as well as other support in the form of access to program staff and stakeholders. If an evaluation contract is not carefully constructed to reflect the responsibilities of both parties, both or either partners may not understand the other parties’ expectations, which could lead to problems and conflicts later in the evaluation process. Thus evaluation activity is based on active negotiation of both parties seeking to promote their interests. Beginning with the initial evaluation plan, the evaluator and the client negotiate to adjust and formalize the proposed evaluation design and timelines, budgets, and deliverables into a final and mutually acceptable evaluation contract agreement. Because each party may have needs and interests that are not necessarily compatible (e.g., client concern for minimal cost vs. evaluators’ concern for larger samples for representativeness), the negotiation process can be rigorous before a mutually acceptable plan is determined. Sometimes negotiations can fail and the evaluation process terminates. Potential Secondary Roles. Several potential secondary roles may be active in support of the negotiator role. The manager’s role may be needed to ensure that the evaluation costs and deliverables are capable of being met, given available human and financial resources. The designer role may also be active as plans are redesigned in response to negotiations. All other roles may need to come into play depending on the specific issues raised and addressed during negotiation efforts.

Active Evaluation Phase Evaluation Activity Five (Initial Implementation) and its Demands The fifth common evaluation activity, initial implementation, is a critical milestone juncture for the evaluator and the evaluation. The demands created by initial implementation require that the evaluator establish a base of rapport, trust, and cooperation with the client and the stakeholders who will be participating in the evaluation. Primary Role Response—Diplomat. In response to initial implementation demands, the evaluator responds to the role of diplomat. Most evaluations require the participation of key stakeholders for data collection purposes, and the level and quality of stakeholder participation can have a substantial impact on the accumulation of credible data. The level of trust stakeholders eventually develop in the evaluator can also impact the likelihood of subsequent use

288

American Journal of Evaluation / September 2009

of evaluation results. Similar to most social interactions, initial impressions are critical for establishing positive longer term relationships; however, ongoing awareness and sensitivity to the needs and concerns of all stakeholders suggest this role continues at some level beyond initial implementation activity. In the diplomat role, the evaluator communicates with stakeholders and helps them understand the purpose and scope of the evaluation, addresses the role of evaluator, and establishes expectations for participation in the evaluation by program staff and stakeholders. Because evaluations can have a major impact on program staff and stakeholders, a legitimate initial high level of stakeholder uncertainty and concern can be anticipated. It is within this context that the evaluator initially enters the setting and begins interacting with staff and stakeholders to begin data collection, making observations, conducting surveys and interviews, and reviewing existing operational data and records. Before effective data collection can proceed, the evaluator in the diplomat role seeks to establish buy-in on the part of stakeholders. Failure to engage the diplomatic role early in the evaluation can adversely affect the willingness of stakeholders to participate throughout the evaluation and it may influence their level of openness when they do participate. Potential Secondary Roles. Several secondary roles may be activated in support of the diplomat role. For example, the researcher role could come into play through the provision of reassurance through protections for human subjects, providing for some level of documentation regarding the safeguarding of human subjects. Similarly the reporter and use advocate roles may be helpful for this activity as well, because these roles support communications and appropriate use of assessment results by stakeholders.

Evaluation Activity Six (Data Collection and Analysis) and its Demands The sixth common evaluation activity, data collection and analysis, creates demands for the collection and analysis of credible data needed to address the evaluation questions and substantiate evaluation judgments. Fortunately, the demands for data collection and analysis activities play to the traditional strengths and competencies of most evaluators. Primary Role Response—Researcher. The evaluator assumes the role of researcher to address the data collection and analysis activity demands of an evaluation. A variety of technical tasks will be represented within the researcher role. The objective is to collect reliable and valid data that are capable of providing answers to the overarching evaluation questions and enable evaluation judgments. Credible information comes from an appropriate methodology, the design or use of reliable/valid instruments, and the implementation of appropriate data collection procedures. The researcher role also includes the rigorous analysis of data and the designation of substantiated findings. The researcher role also seeks to produce relevant data summaries that are accurate and representative of target populations. Fortunately, much of evaluators training focuses on the technical elements of an evaluation such as the specification of evaluation questions and research designs as well as the collection and analysis of valid and credible data. Essential research skills also include knowledge of, and skill with, proper procedures in analyzing evaluation data and drawing appropriate conclusions within the scope of methodological limitations. Weaknesses in these data collection and analysis activities of the researcher role can have serious impacts on the overall quality and credibility of findings. As the expectations for greater methodological rigor in evaluation increases, the researcher role can be more challenging. For example, as contextual conditions can

Skolits et al. / Reconceptualizing Evaluator Roles

289

change, the researcher role may be required to flexibly adapt to the unexpected challenges that can easily occur with field-based research. Potential Secondary Roles. A secondary role of reporter may come into play for this evaluation activity, as the data collection elements of the research process will need to be represented in the evaluation report. Aspects of the diplomat role may also be relevant as much of the data collection effort will continue to require interaction with program staff, program participants, and other stakeholders.

Evaluation Activity Seven (Judgment) and its Demands The seventh common evaluation activity, making a judgment of merit or worth and/or defining areas for improvement, creates demands on an evaluator to make judgmental decisions based on the evidence resulting from data collection and analysis. Guiding the decision are the agreed upon performance standards underlying the evaluation. Primary Role Response—Judge. In response to the judgment activity of an evaluation, the evaluator assumes the role of a judge. Based on the available evidence (i.e., the data collected and analyzed for the evaluation), the evaluator has to determine the extent to which the agreed upon standards of the evaluation have been achieved. For formative evaluations, judgments address the designation of areas of weaknesses with an emphasis on recommending improvements. For summative evaluation, the judgments address the overall value and worth of the program ultimately supporting client decisions regarding program continuation. However, judgment can often require some level of introspection, as the evaluator seeks to minimize acknowledged personal values and perspectives to make a judgment based on the findings and the evaluation standards (a defensible, research-based judgment). For an evaluation, judgment reflects a decision by the evaluator. The evaluator does not simply present facts and findings and then expect the reader to draw his/her own conclusions. Judgments are often not easy, even when the relevant data have been effectively collected and properly analyzed. Often standards are less than specific and concrete and there are always inherent limitations in any research design. Measurement, data collection, and analysis processes are never perfect. Given the existence of some uncertainty in methods and standards, judgments are, therefore, always made in less than ideal conditions. Regardless, all evaluation requires judgment. Potential Secondary Roles. The role of diplomat may be active in response to the judgment activity of an evaluation. Some researcher role elements may also be active, as the conclusions of the data analysis establish a critical element of the framework on which judgments are made.

Evaluation Activity Eight (Reporting) and its Demands The eighth common evaluation activity, evaluation reporting, establishes a demand for the evaluator to share the results of the evaluation with the client and other interested stakeholders. The demands can be limited to the issuance of a single report or it can include multiple interim written reports and verbal presentations about the status and results of an evaluation across phases and activities.

290

American Journal of Evaluation / September 2009

Primary Role Response—Reporter. In response to the demands of this activity, the evaluator assumes the role of a reporter. At this stage in the evaluation, the evaluator records and disseminates the findings of the evaluation. While data collection, analysis, and judgment evaluation activities must be completed before final reporting can be completed, some communication of findings can occur at multiple points in the evaluation process, often requiring that reporting activities have some overlap with data collection and analysis activities. While, some elements of the reporter role have been specified in the evaluation plan (e.g., how many reports are to be prepared, when they are to be submitted, and to whom), stakeholders’ interest in the evaluation process can create unanticipated expectations for interim reporting. Similar to a reporter, the evaluator seeks to tell a specific story (i.e., results of the evaluation) to a defined, targeted audience. However, the evaluator has some additional challenges to address in the reporting process. Different stakeholders may have different interests, perspectives, and levels of understandings of the program being evaluated. Many aspects of evaluation data collection and analysis can be technical and jargon laden. While some audiences will be interested in the reporting on these technical aspects, others will be more interested in a condensed summary of the final results. Moreover, written reports may need to be formally presented to the client and various stakeholder groups, requiring some level of presentation skills. Reporting can also be critical in creating a foundation for subsequent use of evaluation results. The evaluator seeks to communicate evaluation findings to multiple audiences in a way that promotes use of the results in a meaningful manner. This requires more than one-way communication, the reporter role may require various levels of two-way communication. Potential Secondary Roles. Some secondary evaluator roles may come into focus in support of the reporter role. The skills of the diplomat role can be helpful in supporting evaluation reporting. Depending on stakeholders’ questions, the researcher role can also be helpful in explaining the study rationale, methods, and findings.

Post-Evaluation Phase Evaluation Activity Nine (Evaluation Use Advocacy) and its Demands The ninth common evaluation activity, evaluation use advocacy, creates demands to promote meaningful stakeholder use of evaluation results. The demands of this activity, requiring an evaluator to promote the use of evaluation results, partially emanates from the client who usually can be expected to want to use the results. However, it also is partly driven by the inherent responsibilities of an evaluator to promote use. Primary Role Response—Evaluation Use Advocate. In response to evaluation use demands, the evaluator assumes the role of evaluation use advocate. This role addresses the evaluation demand to promote and support the stakeholders’ efforts to use evaluation results in a responsible and meaningful manner. For this responsibility to be accomplished, substantial evaluator success with prior roles will need to have been achieved. There will need to be something of value resulting from the previously completed activities that is credible, relevant, and useable in terms of the evaluation problem being addressed. Evaluations are conducted to address a real problem or need, and the substantial cost and effort required to conduct an evaluation can only be justified in terms of the creation of useful, actionable information. For use to occur, stakeholders will need to believe the overall credibility of the findings. Stakeholders will often need some additional direct support by the evaluator to understand how to use the findings to

Skolits et al. / Reconceptualizing Evaluator Roles

291

their advantage. For example, while the use of some findings may be straightforward, other findings may offer the potential multiple strategies for improvement. Potential Secondary Roles. The reporter role is likely to be activated at this stage of the evaluation, but many other roles, including the roles of reporter and diplomat, may be required depending on the use challenges and issues that may need to be addressed. Evaluation Activity 10 (Evaluator Reflection) and its Demands The 10th common evaluation activity, evaluator reflection, represents an internal demand on the evaluator for introspection and professional self-improvement. The demands of evaluation reflection are driven by the professional responsibility of an evaluator to continuously improve their knowledge and performance as an evaluator. Primary Role—Learner. In response to the demands of this professional activity, evaluators assume the role of learner. In this role, the evaluator seeks to improve their practice through reflection on a recently completed evaluation event. As professionals, evaluators are expected to reflect on their practice and learn from their experiences. Every evaluation experience presents the evaluator with an opportunity for targeted improvement through professional growth and advancement. In this role, the evaluator seeks to learn what worked well, what was less effective about the evaluation than anticipated, and how future evaluations could be improved. Through this role, an evaluator may recognize skill sets or other professional attributes that may need to be enhanced. Except for meta-evaluations, there is no external demand by a client that the evaluator engages in critical reflection, self-examination, and improvement. This is one element of an evaluation that represents an evaluator-driven responsibility. However, reflecting on one’s experiences does not have to remain a solo activity. Involving stakeholders through systematic feedback can also help the evaluator better understand and reflect upon their performance. A meaningful strategy for self-reflection would be a systematic review of each activity of the evaluation, the demands that resulted, and the effectiveness of the associated role responses. Potential Secondary Roles. All roles could be activated at this stage of an evaluation, at least from the perspective of considering how effective they were in addressing demands.

How the Proposed Activity-Based Role Conceptualizations Relate to the Literature This depiction of evaluator roles addresses some of the limitations within the existing literature on evaluator roles. While current macro evaluator role orientations depicted in the literature (e.g., reformer, judge, empowerer, internal/external management resource, neutral scientist, philosopher king) suggest the options for a larger overall evaluator framework or orientation, they offer evaluators little guidance on how these orientations are expected to influence the multiple activities that are required for an evaluation. Our role framework directly relates roles to evaluation activities, better promoting the evaluator’s ability to acknowledge, understand, and activate responsive role orientations for the many evaluation activities that they will encounter. Macro level evaluator roles appear to address higher level orientation to an evaluation. In contrast, the activity-based roles framework provides guidance

292

American Journal of Evaluation / September 2009

at the level of major evaluation activities. Future research may suggest that both approaches may prove to be complementary and compatible.

Model Limitations and Implications Model Limitations All conceptual models are limited as they attempt to simplify reality for illustrative and instructional purposes. An evaluator typically does not adopt and implement an evaluation model as if it was a formula or a script, but an evaluation model helps the evaluator conceptualize the overall focus of the evaluation. The same is true for our role conceptualization of evaluation but with a greater level of detail. The activity-based role conceptualization frames the evaluation process as a fairly common evaluation event reflecting predictable phases, typical activities/demands, and associated roles. However, in the actual conducting of evaluations, there are probably few evaluation events that would completely follow this generic evaluation path. For example, some unique evaluation activities or multiple reiterations of evaluation activities in unanticipated sequences may be encountered. Other evaluations may be highly specialized and particularly narrower in scope. Moreover, each evaluation will be always be somewhat unique given its particular contexts, and thus a generic framework of roles will not fit every occasion. Accordingly, some evaluations may require activities that will establish unanticipated activities and demands that may require new roles beyond those that have been addressed in this role framework. In these instances, the managing of role responses to address more unique and rarely encountered evaluation activities/demands may require another set of skills that will need to be embedded in the manager function of role creation. Reflecting all possible permutations and combinations of evaluation activities/demands and attendant roles occurrences in a single schema or theoretical framework would not be possible or particularly useful. The complexity in such a depiction would cloud the clarity and utility of the model, and little would be gained by the complexity that would be required. Fortunately, many external program evaluations follow predictable sequences, and the specific roles and responsibilities delineated in this conceptualization of roles are consistent with our experiences with external evaluation. Implications Despite these limitations, the conceptualization of evaluator roles as responses to evaluation activity demands suggests important implications for evaluator practice, training, and future research. While the role framework presented here is not a cookbook approach to evaluation, it does present a schema to help an evaluator conceptualize an evaluation process from the perspective of activities, demands, and roles. For new evaluators, this evaluator roles framework can help them better prepare for an upcoming evaluation from the perspective of anticipated roles and their likely occurrence in response to evaluation activities and demands. For experienced evaluators, this role framework can also serve as a reference to promote the review and identification of areas of knowledge and skill that may need to be enhanced through targeted professional development experiences. Moreover, should an evaluator be faced with a unique, larger scale evaluation project and need to establish an evaluation team, consideration of the anticipated evaluation activities, their demands and potential role requirements may suggest the types of knowledge and skill sets that will need to be

Skolits et al. / Reconceptualizing Evaluator Roles

293

recruited to the team. This role framework may also help evaluators anticipate potential role type conflicts that might be encountered in a particular evaluation. Implications for future research suggest the need to determine an empirical base for the particular roles identified. Empirical examination of the roles described in this article is required to determine the extent to which they are consistent with the way other evaluators respond to evaluation activities and their demands. Moreover, it would be helpful for researchers to examine each of the evaluator roles to determine how they may vary based on the type of evaluation being addressed (e.g., needs assessment, formative evaluation, summative evaluation) and/or the type of evaluation client being served (e.g., private organization, nonprofit organization). Moreover, research addressing the dynamics of specific roles would be helpful. For example, much needs to be learned about how the evaluation manager (role) activates, manages, and integrates multiple roles, and how conflicts among multiple roles are adjudicated. Greater understanding of role conflicts especially needs to be targeted for future research efforts. Given the number, complexity, continued occurrence of multiple evaluator roles (primary and secondary roles) as identified in this conceptualization of evaluator roles, there is an extremely strong potential for multiple role conflicts during an evaluation process. At the current time, role conflict issues are not well addressed in the evaluation literature. Possibly, this conceptualization of evaluator roles may also provide a tool for furthering our understanding of the types and nature of evaluator role conflicts. Five questions in particular seem highly appropriate and amenable for further analysis of role conflict: (a) At what point in the evaluation process do role conflicts tend to occur? (b) Which specific roles are involved in these conflicts? (c) What is the basis for the conflicts (i.e., which demands are creating the conflict between or among roles)? (d) What additional demands are being placed on the evaluation management role to address role conflicts? and (e) How does the manager’s role seek to resolve role conflicts? Further study of role conflicts seems especially warranted if evaluator training is going to prepare future practitioners to minimize role conflicts and/or effectively manage them when they do occur There are also implications for evaluator training suggested by this evaluation activity based conceptualization of evaluator roles. This role conceptualization offers students a fairly comprehensive conceptual road map of an entire evaluation process; it depicts and links all major elements of an evaluation across an entire evaluation event (from pre-evaluation to post-evaluation). It also presents students with a more realistic picture of the evaluation process by highlighting the absolute necessity for evaluators to possess managerial and interpersonal acumen along with the knowledge and skill sets traditionally associated with evaluation (e.g., evaluation methods and models). For students seeking a comprehensive set of evaluator skills and competencies, these evaluator roles suggest a potential framework students could use to identify, categorize, and monitor their developing evaluation skills and competencies. Moreover, these evaluator roles suggest the value and necessity of experienced-based approaches to evaluator training that would mimic what evaluators actually experience as they engage in common evaluator roles. Finally, this conceptualization of evaluator roles could be used to support the enhancement of evaluation training curricula through the mapping and alignment of training content with professional practice expectations from the perspectives of roles.

References Alkin, M. C. (Ed.) (2004). Evaluation roots: Tracing theorists’ view s and influences. Thousand Oaks, CA: Sage. Campbell, D. T. (1969). Reforms as experiments. American Psychologist, 24, 409-429.

294

American Journal of Evaluation / September 2009

Cartland, J., Ruch-Ross, H. S., Mason, M., & Donohue, W. (2008). Role sharing between evaluators and stakeholders in practice. American Journal of Evaluation, 29, 460-497. Cousins, J. B. (2001). Do evaluator and program practitioner perspectives converge in collaborative evaluation? Canadian Journal of Program Evaluation, 16, 113-133. Cousins, J. B., & Earl, L. M. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis Theory, 14, 397-418. Dahler-Larsen, P. (2002). Evaluation in an organizational world. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 57-86). Greenwich, CT: Information Age Publishing. Donaldson, C. I., Christie, C. A., & Mark, M. M. (2008). What counts as credible evidence in applied research and evaluation practice? Thousand Oaks, CA: Sage Publications. Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program evaluation (3rd edition). Boston: Pearson Education, Inc. Fetterman, D. M., Kaftarian, S. J., & Wandersman, A. (1996). Empowering evaluation: Knowledge and tools for selfassessment and accountability. Thousand Oaks, CA: Sage. Greene, J. C. (1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 18, 574-591. Greene, J. C. (November, 2000). Reconsidering roles. Paper presented at the Annual Meeting of the American Evaluation Association, Honolulu, HI. Hopson, R. K. (2002). Making (more) room at the evaluation table for ethnography: Contributions to the responsive constructivist generation. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 37-56). Greenwich, CT: Information Age Publishing. House, E. R. (1993). Professional evaluation: Social impact and political consequences. Newbury Park, CA: Sage. Kellogg Foundation. (1998). W.K. Kellogg Foundation Evaluation Handbook. Retrieved January 7, 2008, from http:// www.wkkf.org/pubs/Tools/Evaluation/Pub770.pdf King, J. A., & Stevahn, L. (2002). Three frameworks for considering evaluator role. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 1-16). Greenwich, CT: Information Age Publishing. Lynch, K. D. (2007). Modeling role enactment: Linking role theory and social cognition. Journal for the Theory of Social Behavior, 37, 379-399. Mark, M. M. (2002). Toward better understanding of alternative evaluator roles. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 17-36). Greenwich, CT: Information Age Publishing. Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco: Jossey-Bass. Mertens, D. M. (2002). The evaluator’s role in the transformative context. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 103-118). Greenwich, CT: Information Age Publishing. Montgomery, J. D. (2000). The self as a fuzzy set of roles: Role theory as a fuzzy system. Sociology Methodology, 30, 261-314. Morabito, S. M. (2002). Evaluator roles and strategies for expanding evaluation process influence. American Journal of Evaluation, 23, 321-330. Noblit, G. W., & Eaker, D. J. (1987, April). Evaluation designs as political strategies. Paper Presented at the Annual Meeting of the American Educational Research Association, Washington, DC. Owen, J. M., & Lambert, C. L. (1998). Evaluation and the information needs of organizational leaders. American Journal of Evaluation, 19, 355-365. Patton, M. Q. (2007). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage. Preskill, H., & Torres, R. T. (1999). Building capacity for organizational learning through evaluative inquiry. Evaluation, 5, 42-60. Ryan, K. E., & Schwandt, T. A. (Eds.). (2002) Exploring evaluator role and identity. Greenwich, CT: Information Age Publishing. Schwandt, T. A. (2002). Traversing the terrain of role, identify, and self. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 193-207). Greenwich, CT: Information Age Publishing. Scriven, M. (1991). Evaluation thesaurus. Thousand Oaks, CA: Sage. Segerholm, C. (2002). Evaluating as responsibility, conscience, and conviction. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 87-102). Greenwich, CT: Information Age Publishing. Stronach, I., Halsall, R., & Hustler, D. (2002). Future imperfect: Evaluation in dystopian times. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 167-192). Greenwich, CT: Information Age Publishing. Stufflebeam, D. (2001). Evaluator models. New Directions for Evaluation, 89, 7-106. Stufflebeam, D. L. (2003, October). The CIPP model for evaluation. Paper Presented at the Annual Conference of the Oregon Program Evaluators Network Conference, Portland, OR.

Skolits et al. / Reconceptualizing Evaluator Roles

295

Sullins, S., & Youker, B. (2006). Book review: Evaluator role and identify. American Journal of Evaluation, 27, 130-133. Turner, J. H. (2006). Role theory. In J. H. Turner (Ed.), Handbook of sociology (pp. 233-254). New York: SpringerScience. The Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage. Weiss, C. H. (1998). Evaluation: Methods for studying programs and policies (2nd edition). Upper Saddle River, NJ: Prentice-Hall. Wholey, J. S. (2001). Managing for results: Roles for evaluators in a new management era. American Journal of Evaluation, 22, 343-347.