Policies shaping evaluation of psychological and educational interventions reflect a naive

understanding of the broad nature and function of evaluative research. As a result, such policies are overemphasizing the evaluation of immediate intervention outcomes and undermining comprehensive intervention research. The presentation highlights the danger of this trend and illustrates how the focus of evaluation should be expanded not only to make a greater contribution to practice but to aid in the development of basic knowledge about the nature of interventions. To these ends, evaluation is explored as one of four basic intervention problems; specific examples are offered to underscore the type of basic intervention questions currently ignored and in need of scholarly attention.

INTERVENTION THEORY AND EVALUATING EFFICACY HOWARD S. ADELMAN University of California,

Los

Angeles

effective ow The

are psychological and educational interventions? question is directed at one of the key concerns faced by those who apply psychological theory and research. The concern is not new. Interveners have long been expected to carry out formal evaluations-at least of a formative nature and usually for purposes of

summative decisions. What is relatively

new

troubling is the primary emphasis efficacy evaluation (see Banta and Saxe, and

policymakers giving 1983). Increasingly the focus is narrowed to immediate outcomes rather than broadened to encompass at least a few of the major intervention problems that must be resolved in order to improve efficacy. As a result, current evaluation policies and practices are undermining intervention research. After a cursory introduction to this problem, the intent here is to illustrate how the focus of evaluation should be expanded not only to make a greater practical contribution but also to aid in evolving theory and basic knowledge about the nature of interventions. To these ends, evaluation is analyzed in its context as one of four fundamental and inextricably related intervention problems. are

to

65

66

THE UNDERMINING OF INTERVENTION RESEARCH

Increased demand for evaluation has lead to major advances in the way intervention evaluation is conceived (e.g., Chen and Rossi, 1983; Gottfredson, 1984; Rossi and Freeman, 1982; Tharp and Gallimore, 1979). However, despite the scholarly activity, widespread demands for accountability continue to have the impact of narrowing the way

professionals, clients, policymakers, underwriters, and the general public think about evaluation. The cry is for specific evidence of efficacy-usually in terms of readily measured immediate benefits-and for cost containment. (No one should be surprised by this in light of the unfulfilled promise of so many psychological and educational services and the troubled economic climate.) Although understandable, the prevailing perspectives do tend to ignore the complexities of intervention. The problem is well exemplified by the narrow focus seen in the recent flurry of reviews, analyses, reanalyses, and debates related to data on psychotherapy and behavior change (e.g., Glass and Kliegl, 1983; Office of Technology Assessment, 1980; Wilson and Rachman, 1983). Besides accountability pressures, current evaluation trends in psychology and education seem to rest on two unfounded presumptions; that is, (1) if an intervention is in widespread use, it must be at a relatively evolved stage of development and thus warrants the cost of summative evaluation; and (2) major conceptual and methodological problems associated with evaluating treatment efficacy have been resolved. The truth, of course, is that interventions are frequently introduced prior to adequate validation with a view to evolving them based on what is learned each day; moreover, many well-institutionalized approaches remain relatively unvalidated. As to the process of evaluation, every review of the literature outlines comprehensive, unresolved concerns. Given this state of affairs, current accountability demands are often unreasonable and chronically reflect a naive view of research and theory. Consequently, policies mandating evaluation in pursuit of costeffective intervention run the risk of generating evaluative practices that neither cost-effective nor wise. Premature efforts to carry out summative evaluations clearly are not cost-effective. And cost-effective outcomes cannot be achieved in the absence of cost-effective research and development. The inevitable impact of recent policies has been to increase competition for dwindling resources. In the battle for dollars, interveners are

comprehensive

67

constantly find themselves having to agree to underwriters’ demands for

easily understood cost-effectiveness data. And as always when resources become scarce, academicians are told that future-oriented, ivory-tower pursuits should give way to work focused on the pressing problems of today. Under such conditions, both intervention and scholarly goals often are jeopardized by marketplace dictates. There can be little doubt that the overemphasis on evaluating efficacy draws attention away from the type of intensive research programs necessary for advancing intervention knowledge and practice. To underscore the danger of this trend and as a step toward countering it, the following discussion places evaluation into a framework of other intervention concerns in order to clarify how the focus of evaluation should be expanded.

EVALUATION AS ONE OF FOUR FUNDAMENTAL PROBLEMS FOR INTERVENTION THEORY The complexity of evaluating psychological and educational interventions is perhaps best appreciated when viewed in the context of other fundamental problems associated with intervention theory. For our purposes, intervention theory is described as concerned with four basic problems: (1) the classification problem-how best to differentiate individuals into relevant subgroups (conceptually and methodologically) for purposes of planning, implementing, and evaluating intervention; (2) the underlying rationale problem-how to decide about intervention purposes and procedures (intended outcomes and the means for accomplishing them); (3) the planning and implementation problem-how to implement intended intervention in an optimal way; and (4) the evaluation problem-how to evaluate interventions. Each of the above has major implications for efforts to document intervention efficacy. Each must be reasonably well addressed before satisfactory summative evaluations can be made. Exploration related to each can help evolve basic understanding about the nature, scope, and impact of interventions. Several basic implications can be appreciated by noting the sequential relationship among the four problems. Logically, if valid differentiation is not made among individuals’ intervention needs, appropriate decisions about processes and outcomes usually cannot be made. If

68

appropriate process and outcome decisions cannot be made, appropriate planning and implementation are jeopardized. Thus, even if valid evaluation practices are available and feasible, the logical prerequisites for a sound summative evaluation of efficacy are absent. Even under such circumstances, however, research of an evaluative nature is essential in order to advance knowledge about each of the four fundamental problems. A few examples can help further illustrate (1) the general relationship among the four basic problems and (2) a variety of implications for both evaluating intervention efficacy and advancing knowledge about intervention. Note to begin with, as Figure I graphically suggests, that the intervention concerns are so interrelated that work done on any of them may contribute to increased understanding of the others. THE CLASSIFICATION PROBLEM

approaches to classifying psychological and educational categorize them in terms of current dysfunctioning, causal factors, prescriptive implications, or some combination of all these. The variables and criteria used in defining a category usually are chosen because they have immediate relevance for research, intervention, administrative, or policy matters. When the emphasis is on current dysfunctioning, both severity and pervasiveness of dysfunction are stressed. That is, a narrow or broad range of areas of human functioning may be affected (e.g., developmental areas; school, family, and job performance) and the severity of dysfunction may range from mild to profound depending on normative expectations related to age, intelligence, socioeconomic, and subcultural status. As graphically suggested in Figure 2, the combination of pervasiveness and severity, if treated as discrete categories rather than continuous variables, yield nine classification groups. When the duration of dysfunction (i.e., recent/ acute to longstanding/ chronic) is added as a third variable, the schema jumps to 27 groups. One clear implication of the relatively simple nine-group classification is that outcomes are likely to be harder to achieve with the Z&dquo;’ category than for those in the X’ group. With regard to efficacy, then, it is minimally essential to evaluate groups separately in terms of severity and pervasiveness of dysfunctioning at the onset of intervention. Causal classification demonstrates other complexities. Classification in terms of cause may be done with reference to primary instigators, Common

maladies

69

Figure

1: Four Fundamental Problems for Intervention

Theory

secondary contributing factors, or both. To illustrate how complicated such classification can be, Table I uses a reciprocal deterministic paradigm for understanding human behavior (see Bandura, 1978) to outline major types of factors causing learning problems. The complexity is increased by the way the secondary causal factors interact with primary instigators. In general, the degree to which secondary factors exacerbate problems is determined by type of primary instigator and the degree of dysfunction produced by it. This is exemplified when physiological &dquo;insult&dquo; causes a major CNS disorder and the effect on behavior is so severe and pervasive that resultant dysfunctioning cannot be significantly worsened. In contrast, when CNS disorders are minor, a great many secondary variables can aggravate existing dysfunctions and create other problems. As this analysis suggests, there are times when causal factors and their effects logically are important indicators for intervention decisions and are potential predictors of outcome. Thus, classification of causal factors and their immediate effects and current manifestations can be of great significance in evaluating outcomes; outcomes should differ for groups who differ in the degree to which the pathological causal factors can be compensated for or reversed.

impact

of

70

Figure 2: Classification tioning

in Terms of

Severity and Pervasiveness

of Current

Dysfunc-

Outcome evaluations of remedial reading interventions provide a contemporary and representative example of the classification problem’s impact on evaluating efficacy. A subset of individuals who have difficulty reading are diagnosed as dyslexic or learning disabled. Unfortunately, there are serious deficiencies with these diagnostic categories. For one, there are major disagreements over definition. One approach stresses neurological causality (e.g., National Joint Committee for Learning Disabilities, 1981). Another emphasizes current dysfunctioning, usually in terms of a severe discrepancy between achievement and intellectual ability (Federal Register, 1977: 65083). Because attention to clarifying relevant differences in &dquo;LD&dquo; samples has been sparse, discussions of intervention efficacy often are misleading. And unfortunately, even if there was conceptual agreement about relevant subgroup differences, efforts to evaluate efficacy would be confounded by the low validity of prevailing diagnostic classification procedures (see Arter and Jenkins, 1979; Coles, 1978). It is widely recognized that because of the conceptual and methodological problems associated with categorizing individuals as dyslexic or learning disabled, treatment programs contain individuals whose reading problems differ significantly with regard to severity and cause (see Keogh et al., 1982; McKinney, 1984). When the efficacy of such programs is evaluated, findings tend to show only a modest number of individuals to make good progress (e.g., Gottesman, 1979; Levin et al., 1985). Given such findings, the question arises: Do those who succeed

71

TABLE 1

Factors

Instigating Learning Problems Based on an Interactional Model

SOURCE: Adapted from Adelman and Taylor (1983a). Reproduced with the kind permission of the publisher, Scott, Foresman & Co. *May involve only one (P) and one (E) variable or may involve multiple combinations.

72

represent a cross section of the relevant subgroups in the program or are they primarily those whose reading difficulties are not caused by not start out with broad and skill deficits? profound A reasonable case may be made that positive outcomes are not associated with the full spectrum of individuals with reading problems. That is, as with other psychoeducational problems, the best outcomes are likely to occur for those with relatively mild problems; moreover, these individuals are likely to respond well to a variety of different treatments. In contrast, it is common to find a high proportion of individuals with severe and pervasive dysfunctions whose problems seem intractable; these often may be individuals whose dysfunctions are rooted in major biological disorders. To find a treatment that is generally effective with those who have major biological dysfunctions and/ or broad and profound skill deficits requires isolating subgroups. In this connection, it has been recommended that all investigators at least report a designated set of hypothesized marker variables to differentiate among samples (Keogh et al., 1982). A more ambitious proposal calls for classifying LD subgroups (see McKinney, 1984); unfortunately, the proposal for subtyping skips right by the prerequisite problem of validly differentiating LD from other learning problems. Thus, some investigators have proposed strategies of sequential assessments and corrective interventions as ways first to differentiate among learning problems and then among learning disabilities (Adelman and Taylor, 1983a; Wedell, 1970). Strategy preferences aside, the point is clear: Valid diagnostic classification of dyslexia or learning disabilities must be achieved before the efficacy of remedial reading approaches for such groups can be

neurological dysfunctioning and/ or did

appropriately validated. Besides the LD example, failure to identify appropriate subgroups has been a key factor in most controversies over intervention efficacy, for example, the Head Start program evaluations (see Lazar and Darlington, 1983), psychotherapy outcome studies (see Glass and Kliegl, 1983), and evaluation of alcoholism treatments (see Emrick and Hansen, 1983). That is, in each case, inadequate efforts were made to discriminate between important subgroups in analyzing findings, and as a result, premature conclusions were reached about efficacy or lack thereof. In general, then, as stressed by researchers who focus on the interaction between interventions and client characteristics, the question to be answered is not whether an intervention works-the question is the

73

degree to which a specific intervention works for a given subgroup. In

evaluating efficacy, classification can be seen as a process of specifying relevant subgroups that can help clarify important sources of variance in outcome data. More basically, the appropriateness of most intervention planning, implementation, and evaluation can be seen as determined by the way the classification problem is resolved. Unfortunately, in many key instances, the classification problem remains unresolved and represents a major area where evaluative research could make a significant contribution not only to improving efficacy evaluation but to evolving basic thinking about intervention. THE UNDERLYING RATIONALE PROBLEM

Intentional interventions are rationally based. That is, underlying such activity there is a rationale-whether or not it is explicitly stated. A rationale is a framework outlining and shaping the nature of intervention aims and practices. It consists of views derived from theoretical, empirical, and philosophical sources. It incorporates general orientations or &dquo;models&dquo; of the causes of problems, of tasks to be accomplished, and of appropriate processes and outcomes of intervention (Adelman and Taylor, 1985). Although rationales direct intervention, it is rare to find them systematically formulated and explicitly stated. Even when not explicitly stated however, there is an underlying rationale that guides the thoughts and actions of those involved. With regard to underlying rationale, Bruner (1966) has stated that instructional interventions reflect &dquo;a theory of how growth and development are assisted by diverse means. With regard to psychotherapy, Howard and Orlinsky (1972) indicate that the activity implies &dquo;some conception of human nature or personality (the ’material’ to be worked with), human fulfillment (the ideal to be sought), human vulnerability (psychopathology), of therapeutics, and of the therapeutic profession. Taken together, they comprise ... the Therapeutic Belief-Value Complex&dquo; (p. 617). Rossi and Freeman (1982) stress that interventions are based on hypotheses drawn from causal and predictive studies, from theories, and from clinical impressions. Because they both guide and limit the nature of subsequent intervention activity, rationales have major ramifications for outcomes. As Brickman and his colleagues (1982) suggest, &dquo;each set of assumptions has characteristic consequences for ... competence, status, and well-

74

being... (and) the wrong choice ... will undermine effective helping and coping&dquo; (p. 368). Not all rationales are equal. Some have a greater philosophical, theoretical, and empirical consistency; some reflect a higher level of scholarly sophistication. Thus, besides being an important factor in decisions related to evaluating outcomes, the quality of an intervention rationale is by itself an important concern for evaluative focus. And in this context, a rationale’s consistency and sophistication are not the only important considerations. There is also the factor of systematic bias in decisions about the nature and scope of the rationale. Concern about such bias has long been alluded to in evaluations of mental health and education interventions. With regard to potential bias, Strupp and Hadley (1977) have noted three &dquo;interested parties&dquo; in intervention decision making-the client, society, and the intervener. Adelman and Taylor (1983a, 1985) have aligned the interested parties into two groups, ( 1 ) those most directly involved (e.g., clients, interveners) and (2) those not directly involved but whose influence is felt (e.g., family, friends, third-party payers). The interested parties shape the focus of evaluation and much more. Their attitudes and perceptions about what causes problems and what constitute appropriate corrective strategies and outcome objectives also influence how intervention rationales are formulated. Given that there are often conflicts of interest among involved parties, systematic bias in intervention and evaluation decision making is a major concern. Take, for example, individuals who manifest deviant and devious behavior. The intervention literature overwhelmingly emphasizes strategies designed to &dquo;help&dquo; such individuals change and adapt to prevailing norms and standards. This underlying bias is not surprising given the perspectives of those who shape the rationale for such interventions. They generally are nondeviant &dquo;observers&dquo; of those who are behaving differently-and are not a widely representative range of observers at that. As attribution theory suggests, the psychological tendency is for observers to attribute causes of an actor’s problems to stable, withinperson dispositions (Jones and Nisbett, 1972; Monson and Snyder, 1977). And if those who are not behaving in keeping with expectations are a minority (i.e., in the sense that most other people do not stand out in the same way), this fact may lend a form of external validation to an observer’s belief that the behavior is a problem and the cause must be within the person. Prevailing social and political forces and the lack of power of individuals and subgroups designated as problems also have been identified as furthering this perspective (see Coles, 1978; Hobbs,

75

1975). Ryan ( 1971 ) goes so far as to label the bias as one of &dquo;blaming the victim.&dquo; Outcome evaluations based on biased rationales have the impact of reifying the rationale and downplaying alternative viewpoints. Thus, when the rationale designates individuals as problems, efforts to correct problems and measure change primarily focus on the individual. This results in a pervasive tendency to ignore alternative focal points-such as the environment-as the source of problems and the object of change (e.g., Hobbs, 1975; Kelly et al, 1977; Kessler and Albee, 1975). In evaluating a specific intervention, concern over rationale quality and bias highlights the importance of initiating evaluation processes prior to the onset of intervention. Such evaluation activity can aid in determining the appropriateness of decisions about who or what is to be the focal point of intervention and about the nature and scope of outcome objectives. Furthermore, given that there is an important rational relationship between means and ends, a priori judgments can be made about the compatability between procedures and projected outcomes. Also, well before it is time to analyze outcomes, it is useful to clarify whose objectives will be weighted most heavily in cases of conflicting interests. The above analysis of the underlying rationale problem illustrates the worth of an evaluative focus on intervention rationales. Evaluative investigations of the utility/ validity of intervention rationales should have an immediate impact on practice (e.g., should minimize perpetuating systematic biases) and a cumulative impact on basic knowledge and theory. With reference to the focal point bias, for example, evaluation could help clarify basic intervention questions such as: When should factors in the environment (home, school, workplace) be changed to accommodate (rather than modify) the behavior of a specific person or group of individuals? When should both the person and environment be the focal point for intervention?

THE PLANNING AND IMPLEMENTATION PROBLEM

Intervention rationales usually are abstract and in a state of continuous evolution. This, in part, accounts for difficulties and controversies that surround efforts to translate a rationale into a specific plan of action and to carry the plan out. In addition to controversial

76 0

theoretical and philosophical concerns, planning and implementation also enmesh interveners in major methodological and practical problems. Methodologically, there are difficulties arising from the limited validity of current approaches to assessment and treatment; practically, there are difficulties imposed by forces that resist change and by priorities and limited resources, for example, policies; personnel availability, motivation and competence; and financial support. In essence, the complexity of planning and implementation can be illustrated by discussing intervener efforts to start &dquo;where the client is&dquo; and to maintain an optimal ongoing &dquo;match&dquo; (see Hunt, 1961). Most interveners subscribe to the idea that interventions should be planned and implemented in ways that appropriately match the client (in the sense of establishing just the right amount of challenge to the client’s assimilated way of adapting to environmental circumstances). However, when this idea is articulated beyond the maxim stage, it becomes clear that interveners differ in their theoretical orientation regarding what characteristics to match. For instance, many interveners interpret the concept of the &dquo;match&dquo; as prescribing procedures that do not require responses beyond a client’s current developmental levels (e.g., see Miller, 1981). For others, the idea is understood to mean that both motivation and development should be accommodated (e.g., Adelman and Taylor, 1985; Hunt, 1961). A major implication of such variations in theoretical rationale is that different antecedent conditions are specified as important in planning procedures and outcomes. And thus, not surprisingly, the differences in rationale lead to major variations in decisions about intervention means and ends. Efforts to maintain an optimal match are further complicated by such factors as (a) inadequate assessment procedures for monitoring the continuous changes stemming from experience and maturation and (b) the limited amount of control interveners have over ongoing intervention transactions. In general, then, it is a truism that limitations of present intervention theory and practice make efforts to create and maintain an optimal match considerably less successful than most of us like. And the

complexity of planning and implementation is further highlighted by awareness that major interventions are meant to have lasting and fairly widespread effects. That is, they are supposed to produce significant maintenance and generalization of effects. Again, theoretical and practical limitations often make it unrealistic to predict very much in the way of maintenance and generalization for many interventions.

77

The implications of the planning and implementation problem for evaluating efficacy are so straightforward as to seem simple minded. If the antecedent conditions necessary for intervention

success are

not

actually present to a significant degree and/ or intervention processes are not implemented in a way that adequately approximates an appropriate match with antecedents, then the desired logical consistency between intended antecedents, processes, and outcomes is missing (Stake, 1967). Thus, ( 1 ) an intervention plan should be evaluated prior to implementation to determine the degree to which intended antecedents are present, and (2) processes should be monitored during implementation to determine if intended transactions are taking place. It is not reasonable to predict desired outcomes if intended antecedents and transactions are not well approximated. An extreme but commonplace example of this point occurs in instances when a client terminates treatment prematurely, for example, due to poor motivation or financial difficulty. By definition, treatment has not been successfully implemented in cases of inappropriately premature termination. Consequently, outcome data on such individuals obviously should be differentiated from findings on those for whom the intended antecedents and processes were logically consistent and carried out as planned. Apparently less obvious and less accepted in practice are instances when clients differ in the degree to which they are motivated to participate. Again, however, the same logic applies. Whenever treatment requires clients who have fairly high motivation, those not appropriately motivated should be differentiated from those who are (Adelman and Taylor, 1983b). Such differentiation is essential for proper planning and implementation of intervention and efficacy evaluation. The above examples suggest numerous basic questions for study related to the planning and implementation of interventions. For example: What are the essential facets of planning? What are the key processes that facilitate implementation of intended transactions and specific outcomes? What major factors commonly interfere with intended transactions? Clearly, more is at stake here than evaluating

immediate intervention

efficacy.

EVALUATING INTERVENTIONNOT JUST EFFICACY For fairly obvious reasons, most analyses of the evaluation problem have been concerned primarily with how to maximize the quality of data

78 .

variables. Although less emphasized, discussion also has pointed out the significance of data on unintended outcomes (especially negative side effects) in evaluating efficacy (Strupp and on

intended

outcome

Hadley, 1977). Because the primary focus has been on evaluating efficacy, the critical importance of evaluating other facets of intervention has been too little appreciated. Not only should other facets be evaluated, there are times when these facets should be the primary and even the only focus of evaluative research. Analyses of the evaluation problem (along with implications derived from analyses of the other three fundamental intervention problems) are meant to aid in improving both the evaluation of programs and basic knowledge about intervention as a general

phenomenon. PROGRAM EVALUATION

For new and evolving programs, program evaluation is used in a research and development fashion to test and improve interventions. For established programs, program evaluation involves describing and judging the overall impact and value of interventions. In either case, the framework of basic intervention problems underscores the importance of evaluating more than how well an intervention has accomplished projected outcomes. As the preceding analyses and examples point out, program evaluation should encompass procedures that allow a priori judgments about whether it is even worth implementing a particular intervention. Prior judgments also can be made about whether costs of evaluating outcomes are likely to outweigh the value of gathering comprehensive efficacy data. To be more specific, as the program evaluation literature stresses, there are many questions that logically should be evaluated before an intervention program is implemented. One such question is whether the underlying intervention rationale is coherent, logical, and well grounded theoretically and empirically. If it is not, there may be little justification for proceeding with the program. Prior to implementing processes, it should be determined whether key variations in antecedent conditions have been considered. That is, do the intended processes account for individual and subgroup differences relevant to intended outcomes? If not, is it feasible to do so? When such variations cannot be validly classified, cost-benefit concerns arise about the appropriateness of evaluating planned processes and

79

their efficacy. Indeed, if subgroups cannot be differentiated validly (e.g., in terms of motivation for overcoming problems and severity and pervasiveness of problems), it may not be possible to validate the impact of certain interventions. In instances when relevant antecedent conditions are considered and present, key evaluative questions include whether intended processes actually occur and whether potent unintended processes transpire. Findings related to such matters are basic to decisions about evaluating efficacy and to interpreting outcomes when they are assessed. Another major program evaluation question is whether projected outcomes and available measures are proper indicators of efficacy. If not, outcome evaluation may be inappropriate (e.g., not worth the costs). And, of course, when an intervention or key measures are in early stages of development, it is usually premature to expend limited resources on summative evaluations. The previous examples illustrate how much more is involved in program evaluation than is suggested by accountability demands that overemphasize outcome evaluation. Further examples that nicely capture the dynamic, spiraling nature of evaluative research are offered in presentations by Tharp and Gallimore (1979) on the ecology of program research and evaluation and by Gottfredson (1984) on program development evaluation (PDE). As these writers suggest, evaluating efficacy is but one facet of program evaluation, and program evaluation is but one aspect of evaluative research. IMPROVING BASIC KNOWLEDGE ABOUT INTERVENTION

Beyond expanding

awareness

of the nature and scope of program

evaluation, analyses of the evaluation problem underscore the role of evaluation research in testing and evolving hypotheses about the nature, scope, and impact of interventions. Many examples have been cited throughout this article. Another prototypical example is seen in systematic attempts to investigate intervention processes in order to (1) isolate &dquo;active ingredients&dquo; that produce specific program effects and (2) rule out placebo effects (see Critelli and Neumann, 1984). Evaluative research has contributed to a basic understanding of intervention (e.g., of fundamental intervention problems) and has the potential to do so to a greater extent. Such activity is essential if intervention practices are to meet society’s needs and expectations. As Campbell (1969) has suggested, evaluative research can be part of an

80 t

experimental approach to social reform &dquo;in which we try out new programs designed to cure specific social problems, in which we learn whether or not those programs are effective, and in which we retain, initiate, modify, or discard them on the basis of apparent effectiveness on the multiple imperfect criteria available&dquo; (p. 409). Obviously, knowledge-driven and decision-driven research are not necessarily incompatible (Masters, 1984). The danger in both cases arises when the fuel supply is scarce, and its distribution is tied to naive or biased accountability practices. On a theoretical level, evaluative research is advancing knowledge about classes of intervention (e.g, community, organizational, educational, mental health) and about intervention as a pervasive phenomenon. Increasingly, such research is guided by and contributes to model building about the general nature of intervention. Initial efforts to evolve a comprehensive model of intervention have focused on frameworks and concepts as aids to investigating significant commonalities and differences among interventions (e.g., see Adelman and Taylor, 1983a, 1985; Howard and Orlinsky, 1972; Sundberg et al., 1973; Urban and Ford, 1963, 1971 ). This work shows promise for stimulating more comprehensive and systematic research and theory designed to improve cross-intervention understanding of what works, what does not, and why. Examples of the types of questions for which answers are beginning to emerge are: What is the essence of intervention? Are there specific philosophical and theoretical concepts that underlie all intervention activity? What are the major elements to be found in all intervention practices? These and other questions raised throughout this article represent a range of basic researchable topics that can contribute to fundamental knowledge and theoretical thought regarding the phenomof intervention. Obviously, this brief discussion has barely tapped the range of intervention questions that should be the focus of evaluative research. The intent has been simply to outline a sample of concerns to highlight why such research must not be limited to program evaluation or, even worse, to a narrow efficacy focus. enon

CONCLUDING COMMENTS It is understandable that those underwriting interventions (legislators, insurance companies) want to ensure that costs do not escalate

81

inappropriately. They also seem to believe that desired benefits can be accrued by demanding and then monitoring outcomes. Thus, the overemphasis on evaluating efficacy. Existing policies are short-sighted. They channel dwindling resources into measurement of limited outcomes produced by limited interventions. What can be measured readily is reshaping intervention rationales. In various forms, interveners are pushed toward &dquo;teaching to the test.&dquo; To underwrite accountability, funds are drained away from research that could improve the power (effectiveness and efficiency) of psychological and educational interventions. Evaluative research must go beyond assessing outcomes if it is to improve practices and evolve theory and basic knowledge, and test and evolve hypotheses about the nature, scope, and impact of interventions. In this connection, current policies shaping intervention evaluation are seen as counterproductive. At best, they are a recipe for maintaining the status quo. At worst, they become another factor exacerbating the problems of those who are supposed to benefit from the interventions under evaluation.

REFERENCES ADELMAN, H. S. and L. TAYLOR (1983a) Learning Disabilities in Perspective. Glenview, IL: Scott, Foresman. ———(1983b) "Classifying in terms of inferred motivation to learn."Learning Disability

Q. 6: 201-206. ———(1985) "Toward integrating intervention concepts, research, and practice," pp. 57-92 in S. I. Pfeiffer (ed.) Clinical Child Psychology: An Introduction to Theory, Research, and Practice. New York: Grune & Stratton. ARTER, J. A. and J. R. JENKINS (1979) "Differential diagnosis-prescriptive teaching: a critical appraisal." Rev. of Educ. Research 49: 517-555. BANDURA, A. (1978) "The self system in reciprocal determinism." Amer. Psychologist 33: 344-358.

BANTA, H. D.

and L. SAXE (1983) "Reimbursement for psychotherapy: Linking and public policy making." Amer. Psychologist 38: 918-923. BRICKMAN, P., V. C. RABINOWITZ, J. KARUZA Jr., D. COATES, E. COHN, and L. KIDDER (1982) "Models of helping and coping."Amer. Psychologist 37: 368-384. BRUNER, J. S. (1966) Toward a Theory of Instruction. New York: Norton. CAMPBELL, D. J. (1969) "Reforms as experiments." Amer. Psychologist 24: 409-429. CHEN, H.-T. and P. H. ROSSI (1983) "Evaluating with sense: the theory-driven approach." Evaluation Rev. 7: 283-302. COLES, G. S. (1978) "The learning disabilities test battery: empirical and social issues." Harvard Educ. Rev. 48: 313-340.

efficacy research

82

CRITELLI, J. W. and K. F. NEUMANN (1984) "The placebo: conceptual analysis of a construct in transition." Amer.

Psychologist 39:

32-39.

EM RICK, C. D. and J. HANSEN (1983) "Assertions regarding effectiveness of treatment of alcoholism: fact or fantasy?" Amer. Psychologist 38: 1078-1088. Federal Register (1977, August) Education of Handicapped Children. 42: 65082-65085. GLASS, G. V and R. M. KLIEGL (1983) "An apology for research integration in the study of psychotherapy." J. of Consulting and Clinical Psychology 51: 28-41. GOTTESMAN, R. T. (1979) "Follow-up of LD children." Learning Disability Q. 2: 60-69. GOTTFREDSON, G. D. (1984) "A theory-ridden approach to program evaluation: a method for stimulating researcher-implementer collaboration." Amer. Psychologist 39: 1101-1112. HOBBS, N. (1975) The Futures of Children: Categories, Labels, and Their Consequences. San Francisco: Jossey-Bass. HOWARD, K. I. and D. E. ORLINSKY (1972) "Psychotherapeutic processes," pp. 615-668 in P. Mussen and M. R. Rosenzweig (eds.) Annual Review of Psychology (vol. 23). Palo Alto, CA.: Annual Reviews, Inc. HUNT, J. (1961) Intelligence and Experience. New York: Ronald Press. JONES, E. and R. NISBETT ( 1972) "The actor and the observer: divergent perceptions of the causes of behavior," pp. 79-94 in E. E. Jones et al. Attribution: Perceiving the Causes of Behavior. Morristown, NJ: General Learning Press. KELLEY, J. G., SNOWDEN, L. R. and R. F. MUÑOZ (1977) "Social and community intervention," pp. 323-361 in M. R. Rosenzweig and L. W. Porter (eds.) Annual Review of Psychology (vol. 28). Palo Alto, CA: Annual Reviews, Inc. KEOGH, B. K., S. MAJOR-KINGSLEY, H. OMORI-GORDON and H. P. REID (1982) A System of Marker Variables for the Field of Learning Disabilities. Syracuse, NY: Syracuse Univ. Press. KESSLER, M. and G. W. ALBEE (1975) "Primary prevention," pp. 557-591 in M. R. Rosenweig and L. W. Porter (eds.) Annual Review of Psychology (vol. 26). Palo Alto, CA: Annual Reviews, Inc. LAZAR, I. and R. DARLINGTON (1982) Lasting Effects of Early Education: A Report from the Consortium for Longitudinal Studies. Monograph of the Society for Research in Child Development, vol. 47. LEVIN, E. K., N. ZIGMOND and J. W. BIRCH (1985) "A follow-up study of 52 learning disabled adolescents." J. of Learning Disabilities 18: 2-7. MCKINNEY, J. D. (1984) "The search for subtypes of specific learning disabilities." J. of Learning Disabilities 17: 43-50. MASTERS, J. C. (1984) "Psychology, research, and social policy." Amer. Psychologist 39: 851-862. MILLER, A. (1981) "Conceptual matching models and interactional research in education." Rev. of Educ. Research 51: 33-84. MONSON, T. C. and M. SNYDER (1977) "Actors, observers, and the attribution process: toward a reconceptualization." J. of Experimental Social Psychology 13: 89-111. National Joint Committee for Learning Disabilities (1981) "A new definition of learning disabilities." Learning Disability Quarterly 4: 336-342. Office of Technology Assessment (1980) The Efficacy and Cost-Effectiveness of Psychotherapy. Background Paper 3: The Implications of Cost-effectiveness Analysis of Medical Technology. Washington, DC: Government Printing Office.

83

ROSSI, P. H. and H. E. FREEMAN (1982) Evaluation: A Systematic Approach (2nd ed.). Beverly Hills, CA: Sage. RYAN, W. (1971) Blaming the Victim. New York: Random House. STAKE, R. E. (1967) "The countenance of educational evaluation." Teachers College Record 68: 523-540.

STRUPP, H. H. and S. M. HADLEY (1977) "A tripartite model of mental health and

therapeutic outcomes with special Amer. Psychologist 32: 187-196.

reference

to

negative

effects in

psychotherapy."

SUNDBERG, N. D., L. E. TYLER and J. R. TAPLIN (1973) Clinical Psychology: Expanding Horizons. Englewood Cliffs, NJ: Prentice-Hall. THARP, R. G. and R. GALLIMORE (1979) "The ecology of program research and evaluation: a model of evaluation succession," pp. 39-60 in L. Sechrest et al. (eds.) Evaluation Studies Review Annual (vol. 4) Beverly Hills, CA: Sage. URBAN, H. B. and D. H. FORD (1963) Systems of Psychotherapy: A Comparative

Study. New York: John Wiley. ———(1971) "Some historical and conceptual perspectives on psychotherapy and behavior change," pp. 3-35 in A. E. Bergin and S. L. Garfield (eds.) Handbook of Psychotherapy and Behavior Change. New York: John Wiley. WEDELL, K. (1970) "Diagnosing learning disabilities: a sequential strategy." J. of Learning Disabilities 3 July: 311-317. WILSON, G. T. and S. J. RACHMAN (1983) "Meta-analyses and the evaluation of psychotherapy outcomes: limitations and liabilities." J. of Consulting and Clinical Psychology 51: 54-64.

Howard S. Adelman is Professor of Psychology and Director of the Fernald Laboratory and Clinic at the University of California, Los Angeles. Among his research interests is the basic clinical problem of how to enhance systematically an individual’s motivation to overcome psychological, social, and emotional problems.