Human Resource Management Review 20 (2010) 261–282

Contents lists available at ScienceDirect

Human Resource Management Review j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / h u m r e s

The role of pre-training interventions in learning: A meta-analysis and integrative review☆ Jessica Mesmer-Magnus a,⁎, Chockalingam Viswesvaran b a b

University of North Carolina Wilmington, United States Florida International University, United States

a r t i c l e

i n f o

Keywords: Instructional design Learning Meta-analysis Pre-training interventions Training

a b s t r a c t Learning is one of the main goals of any training program. Much research has focused on how learning may be enhanced through effective training design. We compiled the extant literature exploring the efficacy of five common pre-training interventions in promoting learning. Metaanalytic results (k = 159; total N = 13,684) reveal consistent positive effects for the role of such interventions in learning. Attentional advice and goal orientation (as compared with metacognitive strategies, advance organizers and preparatory information) yielded the most consistent learning gains. Results suggest intervention format, implementation, and match to learning outcome are important considerations. Recommendations are provided for interventions which are useful in promoting cognitive, skill-based, and affective learning gains. © 2010 Elsevier Inc. All rights reserved.

1. Introduction Training is one of the most universal methods for enhancing employee productivity and reinforcing organizational objectives (Arthur, Bennett, Edens, & Bell, 2003); as such, U.S. organizations spend approximately $134 billion annually on training (Noe, 2010). Given the importance of training to organizational effectiveness and the costs associated with its development and implementation, it is important that researchers and practitioners have a clear understanding of the factors which promote effective training initiatives. Much research has focused on ways in which training programs may be designed to improve learning, since learning is a logical pre-requisite to improved trainee job performance (Arthur et al., 2003; Colquitt, LePine, & Noe, 2000; Gagné, Briggs, & Wager, 1992; Goldstein & Ford, 2002; Noe, 1986), and practitioners are routinely advised that properly preparing trainees for learning is crucial to maximizing the learning outcomes of training initiatives (cf. Blanchard & Thacker, 2010; Gagné et al., 1992; Noe, 2010). Based on a qualitative review of the extant literature, Cannon-Bowers, Rhodenizer, Salas, and Bowers (1998) argued five types of pre-training interventions (attentional advice, meta-cognitive strategies, advance organizers, goal orientation, and preparatory information) have the potential to significantly enhance learning from training, and made some recommendations to practitioners regarding their use. However, the authors cautioned pre-training interventions likely vary in their effectiveness for different learning outcomes, and that certain moderators like intervention format, training content and training method may interact to yield varying results. They cautioned practitioners should “interpret these guidelines with caution…” until they could be empirically tested (p. 313). More than a decade has passed and neither the comparative effectiveness of these pre-training interventions in promoting learning nor their boundary conditions (i.e., moderators of their effectiveness) have been empirically confirmed. A key reason for this is that it is difficult to explore all five interventions in a single study; indeed most researchers find it feasible to examine only one intervention (and often only one format of that intervention) at a time. How, then, do we make

☆ This paper is based on the doctoral dissertation of Jessica Mesmer-Magnus completed under the supervision of Chockalingam Viswesvaran. ⁎ Corresponding author. Department of Management, UNC-Wilmington, 601 South College Road, Wilmington, NC 28403, United States. Tel.: +1 910 962 7193. E-mail address: [email protected] (J. Mesmer-Magnus). 1053-4822/$ – see front matter © 2010 Elsevier Inc. All rights reserved. doi:10.1016/j.hrmr.2010.05.001

262

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

broader conclusions regarding which interventions better promote which types of learning outcomes, which formats of these interventions are most effective and when, etc.? Meta-analysis permits a quantitative aggregation of the results of primary studies so that overall conclusions can be drawn. Further, meta-analyses make possible the assessment of relationships not explored in primary studies (Arthur et al., 2003). For instance, by cumulating results across studies, we can empirically investigate the effects of study characteristics on outcomes. Although many studies have examined the effectiveness in promoting learning of one or more of the pre-training interventions discussed by Cannon-Bowers et al. (1998), their results have been dispersed throughout diverse literatures of education, training, and psychology. Further, common recommendations made regarding the application of pre-training interventions are often based on studies of grade school children, the results of which may or may not generalize to adults (e.g., Luiten, Ames, & Ackerson, 1980). In this study, we meta-analytically integrate the extant literature on the role of the five pre-training interventions reviewed by Cannon-Bowers et al. (1998) in learning using only studies of adult trainees. We have organized the results using two wellaccepted frameworks as a lens to promote the application of these findings in both research and practice; specifically, pre-training interventions were organized using the Cannon-Bowers et al. (1998) framework and their impact on learning was assessed using the framework of learning outcomes proffered by Kraiger, Ford, and Salas (1993; i.e., cognitive, skill, and affective learning). Following key questions identified by Cannon-Bowers et al. (1998) in their review of the literature on pre-training interventions, we investigate the following questions regarding the role of pre-training interventions in learning: To what extent are pre-training interventions effective in improving cognitive, skill, and affective learning outcomes that result from training? Are certain interventions more useful in facilitating some types of learning than others? Does the format of the intervention matter? Does the training content and/or the method employed affect their utility?

1.1. Pre-training interventions Pre-training interventions are activities or material introduced before a training or practice session to improve the potential for learning, as well as the efficiency and effectiveness of practice during training. They are thought to enhance learning by appropriately matching components of the training program to trainees' internal learning processes (e.g., Gagné, 1996; Gagné et al., 1992), and effectively focusing trainee attention during learning. Consistent with the tenets of Instructional Design and Social Learning theories (Bandura, 1977, 1982, 1986; Blanchard & Thacker, 2010; Gagné et al., 1992), these interventions are thought to promote learning by orienting trainees to the nature, structure, and/or complexity of the forthcoming training material (e.g., attentional advice and advance organizers), or by providing strategies learners might use to monitor their learning during training and their performance during practice (e.g., meta-cognitive strategies; Cannon-Bowers et al., 1998). Specifically, Instructional Design and Social Learning Theories are widely used as a basis for designing training programs as they articulate how training interventions foster the sort of internal learning processes required for rapid and complete learning (Frandsen, 1961; Gagné et al., 1992; Goldstein & Ford, 2002). Learning processes detailed by these theories (e.g., gaining learner attention and setting expectancies to ensure they attend to the desired stimuli, activating memory by triggering the retrieval of prior knowledge, encourage correct encoding and cognitive organization by providing guidance for learning and retention) are readily influenced by pre-training interventions, though each intervention likely supports some learning processes more than others. Indeed, in their Principles of Instructional Design, Gagné et al. (1992) discuss implications these learning theories have for how training programs should be designed, mentioning strategies similar to pre-training interventions. In the following sections, we review five common pre-training interventions and explain how each supports learning. We summarize the learning processes likely supported by the various pre-training interventions in Table 1.1

1.1.1. Attentional advice Attentional advice refers to a pre-training intervention that provides “information, independent of performance content, about the process or strategy that can be used to achieve an optimal learning outcome during training” (Cannon-Bowers et al., 1998, p. 294). The goal of this type of intervention is to direct attention toward specific aspects of the training or practice curriculum, to prime existing knowledge structures, and to lay a foundation for the development of task-related strategies and general schemas or mental models that can later be applied to similar tasks (Cannon-Bowers et al., 1998; Foster & Macan, 2002; Lung & Dominowski, 1985; Phye, 1991). Attentional advice arguably supports learning processes associated with gaining trainee attention and setting appropriate expectancies, advising trainees of learning objectives, and prompting the activation of relevant existing knowledge.

1 Importantly, it is possible these pre-training interventions operate through the same intervening learning processes. For example, providing a pre-training goal orientation or attentional advice may promote the use of certain meta-cognitive strategies during learning, an outcome that may also be achieved by providing a pre-training meta-cognitive strategy. Unfortunately, in the current study design, we are not able to determine how each pre-training intervention promotes the various learning processes as elaborated in Instructional Design and Social Learning theories, only the extent to which the interventions compare in their effect on various learning outcomes. There is value, however, in knowing whether the pre-training interventions differ in their affects on learning; the potential they differentially affect the same learning processes and/or operate under similar underlying learning mechanisms is an important direction for future research. We thank the editor and an anonymous reviewer for raising this point.

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

263

Research findings regarding the effect of attentional advice on learning and transfer suggest that trainees receiving advice prior to practice perform significantly better on measures of learning both during and after practice, as well as in job performance after training (e.g., Foster & Macan, 2002; Lung & Dominowski, 1985; Phye, 1991; Phye & Sanders, 1994). Further, advice when combined with corrective feedback appears to facilitate superior transfer and the generation of more complete mental models (e.g., Phye, 1989, 1991). Developing a skill, for example, requires knowledge of the steps involved in the skill (verbal knowledge) as well as an understanding as to the hierarchical organization of the steps and their components that are required for task completion (Kraiger et al., 1993). Therefore, initial skill acquisition depends upon the accurate knowledge of the steps involved in performing a skilled task or procedure. Skill compilation requires an understanding as to the hierarchical interrelationships between core concepts involved in the performance of a skilled task, as well as the integration of this knowledge with prior skill-related knowledge. Therefore, when trainees are provided with attentional advice, which serves to direct their attention toward factors in the training environment related to a process/strategy that will be beneficial to learning the skill, as well as suggesting links to prior skillrelated knowledge, it is likely they will “outlearn” trainees not provided with such advice. Further, since attentional advice directs trainee attention to prior knowledge and toward developing general task strategies (rather than performance–contingent strategies), it is likely that trainees provided with attentional advice will also score higher on measures of cognitive learning. Since training knowledge may be more easily integrated into currently held mental models for related information (with the provision of attentional advice), these trainees will likely have more complete and accurate knowledge structures of training content, as well as a better understanding as to the appropriate application of this material. Hypothesis 1. Trainees provided with attentional advice prior to training will have higher average learning scores on measures of cognitive (H1a) and skill-based (H1b) learning than trainees not provided with such advice. Attentional advice is targeted toward learning for subsequent transfer, rather than learning simply for acquisition or memorization (Phye, 1989), and therefore the advice provided may be very general in nature. Logically, however, the specificity of the advice may have implications for its impact on relevant learning outcomes. Although provision of specific advice may aide a trainee in learning within a narrow domain, general advice is likely more easily adapted to novel situations, which likely makes trainees more effective in practice (Cannon-Bowers et al., 1998). Further, more general advice likely requires the trainee to take a more active role in integrating the advice with prior knowledge and training material during training, thus increasing the potential it can be beneficial (Towler & Dipboye, 2001). Hypothesis 2. Trainees provided with general attentional advice prior to training will have higher average learning scores on measures of cognitive (H2a) and skill-based (H2b) learning than trainees provided with specific attentional advice. 1.1.2. Meta-cognitive strategies Meta-cognitive strategies are a pre-training intervention that relies on the concept of meta-cognition, which refers to a “selfregulatory mechanism that helps people to guide their own performance in complex domains” (Cannon-Bowers et al., 1998, p. 296; Cohen, Freeman, & Wolf, 1996), and includes “planning, monitoring, and revising goal appropriate behavior” during practice or in task completion (Ford, Smith, Weissbein, Gully, & Salas, 1998, p. 220). Pre-training meta-cognitive strategy interventions are thought to assist trainees in cognitively monitoring their progress toward meeting learning objectives, and in adjusting their learning strategy accordingly (Cannon-Bowers et al., 1998). Specifically, when trainees make use of appropriate

Table 1 The role of pre-training interventions in supporting learning processes. Pre-training intervention

Supported learning processes

Preparatory information Goal orientation

Gain attention; set expectancies Gain attention; set expectancies Inform of training goals and objectives Gain attention; set expectancies Inform of training goals and objectives Activate memory; stimulate recall of prior knowledge Gain attention; set expectancies Inform of training goals and objectives Activate memory; stimulate recall of prior knowledge Provide guidance for learning via symbolic encoding and cognitive organization; foster assimilation with previously learned material Provide guidance for learning via symbolic coding and cognitive organization; foster assimilation with previously learned material Provide guidance for retention via encouraging symbolic/cognitive rehearsal and providing cues for retrieval and aids for application, generalization, and transfer

Attentional advice

Advance organizers

Meta-cognitive strategies

Note. The learning processes included in this table are associated with the Social Learning and Instructional Design theories (cf. Blanchard & Thacker, 2010; Gagné et al., 1992).

264

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

meta-cognitive strategies, they are expected to learn more effectively because they are able to identify when they are having problems and, consequently, may appropriately modify their learning strategies or behaviors (Ford et al., 1998). Arguably, metacognitive strategy interventions may support learning processes associated with symbolic encoding, cognitive organization, and symbolic rehearsal of training material. Also, these interventions likely support effective behavioral reproduction and practice efforts by aiding the trainee in performing tasks and identifying when and how to apply newly learned material during practice. Research results regarding the effectiveness of meta-cognitive pre-training interventions have been promising (e.g., BerardiColetta, Buyer, Dominowski, & Rellinger, 1995; Coutinho, Wiemer-Hastings, Skowronski, & Britt, 2005; Desoete & Roeyers, 2006; Ford et al., 1998; Narciss, Proske, & Koerndle, 2007; Souvignier & Mokhlesgerami, 2006; Volet, 1991). For example, Berardi-Coletta et al. (1995) found that while trainees forced to solve problems via meta-cognitive processing spent longer in finding solutions, they performed better on problem-solving tasks, made fewer errors, and were better equipped to transfer practiced material to novel situations, than those trainees who were provided hints, explanations, or demonstrations. Further, these learners developed more complete and complex mental models for problem solutions. Volet (1991) found that university students utilizing metacognitive strategies during practice performed better on performance tasks and reported greater satisfaction with training than those in the control group who were not coached to use meta-cognitive tactics during practice. Similarly, Ford et al. (1998) found that trainees utilizing meta-cognitive activities performed significantly higher on measures of knowledge acquisition and skilled performance, and reported elevated task-related self-efficacy following practice. As meta-cognitive strategies prompt trainees to intuit the rationale behind a series of task-related steps, it is likely that these individuals will become less focused on individual steps involved in performing a task, and more focused on the entire process. As such, meta-cognitive strategies should facilitate faster and more fluid task performance. Further, because these trainees are focused on underlying rationale, rather than discreet steps, they are more likely to gain perspective on the suitability of trained knowledge and skills to a variety of applications. And, as the use of meta-cognitive strategies facilitates continuous assessment of understanding and performance, the trainee is likely to correct deficiencies prior to the receipt of negative performance-related feedback. Rather, they are likely to perform better in training, and thus affective outcomes, like self-efficacy, would be positively impacted. Hypothesis 3. Trainees provided with meta-cognitive strategies prior to training will have higher average learning scores on measures of cognitive (H3a), skill-based (H3b), and affective (H3c) learning than trainees not provided with meta-cognitive strategies. According to Berardi-Coletta et al. (1995), the use of meta-cognitive strategies requires a deeper level of processing, wherein training content is learned as a result of gaining knowledge on three levels: (1) knowledge of the self as a learner and problem solver, which results from sustained self-evaluation and self-regulation, (2) knowledge of the task, which results from utilizing knowledge of training content during practice, and (3) knowledge of the strategy, also termed procedural knowledge, which results from practicing the strategy behind the problem solution. Practice is maximally effective when knowledge has been gained on these three levels (Flavell, 1979). In practice, a meta-cognitive strategy may be implemented by (1) instructing trainees to “think aloud” about the process or skill being learned or the interrelation between concepts or (2) reminding trainees to continuously answer the question “Why am I doing this?” Although both strategies are effective in promoting knowledge of the self as a learner as well as knowledge of the task, the why-based strategy effectively shifts attention from the details of the problem to the underlying process used to solve the problem (e.g., Berardi-Coletta et al., 1995), thus also promoting knowledge of the most effective strategies for solving the task/problem. As such, we would expect: Hypothesis 4. Trainees provided with “why-based” meta-cognitive strategies will perform better on measures of learning than trainees provided with “think aloud” strategies. 1.1.3. Advance organizers Advance organizers refer to “a category of activities such as outlines, text, aural descriptions, diagrams and graphic organizers that provide the learner with a structure for information that will be provided in the practice environment” (Cannon-Bowers et al., 1998, p. 298). Organizers include verbal, quantitative, or graphic cues presented to trainees to aide in the integration of new knowledge with existing knowledge (Goldstein & Ford, 2002). An organizer may be presented at the onset of training (advance organizer) to provide a structure for organizing training material, or later in training to clarify lecture concepts (comparative organizer; Cannon-Bowers et al., 1998; Goldstein & Ford, 2002; Langan-Fox, Platania-Phung, & Waycott, 2006; Langan-Fox, Waycott, & Albert, 2000). Such tools are used in conjunction with training for three main reasons: (a) to focus trainee attention on the important components and relationships of the training material, (b) to provide a framework for organizing incoming information, and (c) to facilitate the incorporation of incoming information with current knowledge (Cannon-Bowers et al., 1998; Gagné et al., 1992; Goldstein & Ford, 2002). Arguably, advance organizers support the trainees' internal learning process by identifying relevant training material, stimulating recall of prior relevant knowledge, and encouraging the assimilation of new knowledge within the existing knowledge structure. Both linear and graphic advance organizers have been frequently utilized within the learning context, though their effectiveness in facilitating learning and application of training material has varied (e.g., Langan-Fox et al., 2006, 2000). Linear advance organizers include outlines and text that present information hierarchically or linearly. A potential drawback of linear organizers is an insufficient emphasis placed on concept interrelations (Langan-Fox et al., 2000). Graphic advance organizers

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

265

include structured overviews, concept or knowledge maps, matrices, and conceptual models, and may also present information hierarchically. Research regarding linear and graphic organizers suggests that both are effective in facilitating knowledge organization. Since advance organizers provide trainees with advanced knowledge as to the content of training and the conceptual interrelationships of this material, they likely assist trainees in acquiring verbal knowledge and in developing more expert-like mental models of training material (e.g., Butcher, 2006). Both cognitive outcomes are also crucial to skill development, especially to skill compilation. Required for skill compilation, after all, is knowledge of the underlying rationale for steps/processes, as well as an accurate understanding of the conceptual interrelationships of the declarative knowledge required to perform the skill. Hypothesis 5. Trainees provided with advance organizers will perform better on indicators of cognitive (H5a) and skill-based (H5b) learning than trainees not provided with advance organizers. 1.1.4. Goal orientation As a pre-training intervention, goal orientation refers to a goal (mastery or performance) set within the learning environment to focus trainee cognition and behavior (Cannon-Bowers et al., 1998; DeShon & Gillespie, 2005; Kozlowski & Bell, 2006).2 Mastery and performance goals are thought to differentially impact a trainee's cognitive, affective, and motivational processes during learning and practice. Arguably, goal orientation supports learning processes associated with setting appropriate learning/training expectations, adoption of training objectives, and interpreting the goals and structure of the learning environment. Trainees with a mastery orientation are dedicated to increasing their competence at a task and are oriented toward self-improvement (Kozlowski et al., 2001). Therefore, these trainees will focus their attention on the task, devoting effort to learning for learning's sake (Fisher & Ford, 1998; Seijts & Latham, 2005). In contrast, trainees with a performance orientation are focused on performance of a task and are motivated to prove their ability to others. These trainees will focus on performing well on learning criteria and will therefore devote less time and effort to developing task proficiency and more effort to ego management (Fisher & Ford, 1998). Research suggests mastery-oriented trainees acquire trained knowledge faster, perform trained skills more proficiently, and have higher post-training self-efficacy, than their performance-oriented counterparts (e.g., Kozlowski et al., 2001). Importantly, research also suggests setting a specific, attainable goal (regardless of whether the goal is to learn training material or simply perform well on tests of training content) improves task performance (e.g., Mento, Steel, & Karren, 1987). Therefore, regardless of the orientation provided (mastery or performance), trainees who receive a pre-training goal, are likely to outperform those who do not. Further, if that goal fosters a mastery orientation, trainees are likely to outperform (on measures of cognitive and skill outcomes) those who receive a performance-oriented goal. Hypothesis 6. Trainees provided with a pre-training goal orientation (whether mastery- or performance-oriented) will perform better on indicators of cognitive (H6a), skill-based (H6b), and affective (H6c) learning than trainees not provided with a pretraining goal. Hypothesis 7. Trainees provided with a mastery goal orientation will perform better on indicators of cognitive (H7a), skill-based (H7b), and affective (H7c) learning than trainees provided with a performance goal orientation. 1.1.5. Preparatory information Preparatory information aids trainees by “setting the learner's expectations about the events and the consequences of actions that are likely to occur in the learning environment” prior to entering practice/learning (Cannon-Bowers et al., 1998, p. 305). Most research related to the use of preparatory information has focused on preparing trainees for performing under stressful conditions and in potentially distressing environments. Research suggests preparatory information may benefit complex decision-making in high-stress situations (e.g., Inzana, Driskell, Salas, & Johnston, 1996; Johnson & Cannon-Bowers, 1996) by building trainee selfefficacy by reducing shock associated with unexpected stress and by providing strategies to overcome it (Cannon-Bowers et al., 1998). By preparing trainees for the events that will take place during training, learners may be able to access schemas of similar events in advance. Such schema priming facilitates the efficient encoding of training material via assimilation and accommodation processes, promoting more efficient cognitive and skill-based learning than can be accomplished by creating new schemas. Further, setting accurate expectations about the nature of potentially unpleasant events in training promotes trainee self-efficacy and satisfaction with the training process (Cannon-Bowers et al., 1998). Hypothesis 8. Trainees provided with preparatory information will perform better on indicators of cognitive (H8a), skill-based (H8b), and affective (H8c) learning than trainees not provided with preparatory information. 1.2. Potential moderators of learning In addition to the format of the pre-training intervention, several training design factors may moderate the effectiveness of pretraining interventions in promoting learning. Specifically, in their meta-analysis assessing the effectiveness of organizational 2 Although goal orientation was originally conceptualized as a bipolar dispositional trait variable, research suggests it may be susceptible to situational or contextual pressures as well (Kozlowski et al., 2001). Therefore, goal orientation may be viewed as either (a) a trait or dispositional variable that influences learning processes and outcomes, or (b) the focus of an intervention meant to influence the direction of motivation during learning and practice.

266

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

training, Arthur et al. (2003) found the training methods used during training and the nature of the material trained were important predictors of training outcomes. Further, Wexley and Latham (2002) argued that training methods may be differentially effective depending on the nature of the training material they are paired with. When a pre-training intervention is employed in conjunction with different training methods, the resulting impact on the level and quality of learning may be reduced or enhanced. Similarly, as was the case in Arthur et al. (2003; where training methods interacted with training material), pre-training interventions may interact with the content of training to affect level of learning. As such, we will explore the extent to which the (1) training methods employed (e.g., lecture, simulation, case studies; e.g., Gagné et al., 1992; Goldstein & Ford, 2002), and (2) the content of the training (i.e., the skill or task to be trained; e.g., self-awareness programs, human relations programs, problemsolving or decision-making programs, job-related tasks; Burke & Day, 1986; Gagné et al., 1992) moderate the role of pre-training interventions in learning. As very little, if any, primary research has specifically assessed these effects, we do not make specific hypotheses regarding the nature of these moderated relationships. Rather, contingent upon the availability of this data in the literature, the potential that these variables interact with the five pre-training interventions will be examined through exploratory moderator analyses.3 2. Method 2.1. Database One hundred and fifty-nine independent studies reported in 128 manuscripts (total N = 13,684) examining the impact on learning of one or more pre-training interventions were included in this meta-analysis (43 studies examined attentional advice, 24 studies explored meta-cognitive strategies, 65 studies examined organizers, 17 studies utilized goal orientation, and 7 dealt with preparatory information). To ensure a comprehensive search, these studies were located by (1) conducting computerized searches of the PsycInfo, ABI Inform, and ERIC databases using relevant key words and phrases4, (2) snowballing the references cited in published reviews (e.g., Arthur et al., 2003; Cannon-Bowers et al., 1998), and (3) requesting relevant studies from recent annual conferences (e.g., Society for Industrial and Organizational Psychology, Academy of Management). Studies were included only if they involved the administration of a pre-training intervention to adult trainees5 as part of a training program, measured at least one type of learning outcome (related to cognitive, skill, or affective outcomes), and provided sufficient information to compute relevant effect sizes. When authors reported separate effect sizes for different sub-groups (e.g., males and females) or samples, those effect sizes were examined separately as appropriate to allow the potential for more focused moderator analyses. For example, in four manuscripts, authors examined two different pre-training interventions in the same study, but tested them using independent experimental groups. As their effects were included in different meta-analyses, they did not compromise the independence assumption. In cases where multiple measures of the same construct were provided, an average correlation was computed to avoid “double counting” of studies. Due to the lengthy list of articles in this database, a comprehensive list of studies included in the meta-analysis is available from the first author. 2.2. Coding procedures The first author undertook an independent effort to code studies that met the criteria for inclusion in this meta-analysis. A random subset of the studies was coded by a graduate-level research assistant so that coder reliability could be determined. Intercoder agreement was very high (97%), likely due to the objective nature of the data coded. Data coded included the study sample size, the type and format of the implemented pre-training intervention, measures of learning outcomes, reliability (if available) for criterion measures, and data related to potential moderators of the practice–learning relationship. Potential moderators coded included the (1) training method employed and (2) training content/material covered during training. 2.2.1. Coding of pre-training interventions The coding of pre-training interventions was typically straightforward. Nonetheless, a few decision rules were developed in an effort to ensure high construct validity. For example, while conceptually distinct, meta-cognitive strategies and attentional advice can often appear similar in practice. Both interventions focus attention toward learning during the training process. However, attentional advice is meant to orient and focus trainee attention toward the learning objectives of training, and is geared toward helping the trainee to identify and attend to important training content (Cannon-Bowers et al., 1998). Advice might be specific or general in nature, but the focus is still on helping the trainee attend to the overall learning objectives of the training program.

3 In addition to training method and content, it is also possible sample type (e.g., educational versus industrial) and research design (e.g., lab versus field study) may operate as moderators. There were not enough studies to explore sample type or research design as moderators in this meta-analysis nested within the combination of different pre-training interventions by learning outcomes. However, all studies included in our database focused on adult trainees, providing the basis for generalizability of our findings to industrial/field samples. 4 For example, attentional advice, strategy advice, general strategies, schemata training, pre-training questions, meta-cognitive strategies, self-regulatory strategies, advance organizers, orienting activities, outlines, graphic organizer, knowledge map, goal orientation, training objectives, mastery goals, performance goals, preparatory information, and realistic previews. 5 Studies were omitted if the sample was composed of school children (e.g., samples of middle or high school students, etc.) or non-traditional populations (e.g., participants with learning disabilities, brain damage, etc.), or when they dealt with unrelated constructs.

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

267

Meta-cognitive strategies, on the other hand, are meant to assist the trainee in monitoring their level and quality of learning during training and practice by focusing the trainee's attention on underlying processes (Cannon-Bowers et al., 1998). Meta-cognitive strategies orient the trainee toward these elements by, for example, encouraging the trainee to “think aloud” during training or practice, or by encouraging the trainee to focus on “why” processes, concepts, or components are interrelated. To be considered a pre-training intervention, meta-cognitive strategies must be provided to the trainee by the educator or learning environment, rather than be self-developed or simply a learning tendency. Studies that assessed the role of meta-cognitive tendencies in learning (which would be indicative of an individual difference variable rather than a pre-training intervention) were excluded from analysis. Organizers can be provided to trainees at any point in the training process (before, during, or after). However, to be considered a pre-training intervention, the organizer must be attended to prior to the commencement of training. In other words, to be included in this meta-analysis, the primary study must have utilized an “advance” organizer. Advance organizers may take on a variety of formats (graphs, outlines, text, videos, oral explanations, etc.) Of the studies included in this meta-analysis, the vast majority utilized either a graphic organizer or a textual organizer. Goal orientation, as a pre-training intervention, must be provided as an explicit goal or as a goal imposed by or within the learning environment prior to the commencement of a training program or session. These goals may orient the trainee towards mastering training content (mastery goals) or towards performing well on post-learning assessments (performance goals). The primary studies have provided data that facilitates an examination of the impact of goal orientation on learning by (1) comparing the provision of a goal orientation prior to training (regardless of whether it is a mastery or performance orientation) versus no goal orientation, (2) comparing the provision of a mastery orientation with no specific goal orientation, (3) comparing the provision of a performance orientation with no specific orientation, and (4) comparing the provision of a performance orientation versus a mastery goal orientation. A positive effect between performance versus mastery goal orientation and a learning outcome would indicate that mastery orientation is superior to performance orientation in promoting that type of learning. With respect to preparatory information, studies were coded based upon whether the preparatory information provided trainees with strategies for “coping” with unexpected emotions or incidents within the learning environment, or whether it is intended to simply foster realistic expectations for what might occur during training (e.g., negative reactions to stimuli, etc.).

2.2.2. Coding of learning outcomes Learning outcomes were coded using the Kraiger et al. (1993) taxonomy as a guide. According to this taxonomy, cognitive learning may be assessed at any of three stages of learning: verbal knowledge, knowledge organization, and cognitive strategies. Verbal knowledge may include declarative, factual, procedural, strategic, and tacit knowledge (e.g., Anderson, 1982; Greeno, 1980; Kraiger et al., 1993; Wagner, 1978); in the primary studies, verbal knowledge was typically assessed using multiple-choice, free recall, or true/false tests which examine the extent to which declarative, factual, procedural or strategic knowledge has been attained. Knowledge organization, which is concerned with ascertaining the accuracy of the trainee's cognitive organization of declarative knowledge, was typically assessed using multiple choice, essays, free recall, card sort tactics, cognitive task analysis, and other structural assessment approaches. Cognitive strategies represent strategic knowledge of how and when to use trained material (Gagné et al., 1992). Cognitive strategies were typically assessed utilizing self-reports of this information. Skill-based learning is also assessed at any of three stages of learning: skill acquisition, skill compilation, and skill automaticity. Initial skill acquisition is measured by the development of procedure-related declarative knowledge (Day, Arthur, & Gettman, 2001), typically using multiple-choice, true/false, and free recall tests of procedure-related declarative knowledge. Compilation involves integrating specific skill-related steps into a coherent whole. Trainee skill compilation was generally evaluated by assessing the trainee's ability to perform discrete skill-related behaviors under conditions of maximal performance, or by having trainees demonstrate they are able to generalize skill-related behaviors to novel problems or contexts. Skill automaticity, which is characterized by fluid, effortless, error-free, and invariant performance of skills, was rarely assessed or reported. However, the few studies that did report data relevant to trainee skill automaticity had evaluated the trainee's ability to perform two tasks concurrently (one requiring the use of the trained skill and another task), or to perform a task requiring the trained skill while distracted. Finally, the third category of learning outcomes, affective learning, includes all those learning outcomes that are neither primarily cognitive nor primarily skill-based, but still indicate that learning has occurred (i.e., attitude change, motivational disposition, and self-efficacy; Kraiger et al., 1993). Affective learning outcomes were in all cases assessed using self-reports. Attitudinal outcomes may include any of a variety of attitudes. However, the most frequently assessed attitude was the trainee's assessment of the training program. Assessment of self-efficacy was always measured following learning, and focused primarily on the trainee's self-efficacy or confidence in using trained material. Disposition generally involved an evaluation of the trainee's intention to use or transfer trained material.

2.2.3. Coding of potential moderators The method by which training was delivered and the content of training were coded in an effort to examine whether pretraining interventions were better used in conjunction with certain training methods and/or paired with certain training content. Three categories of training methods were coded, including (a) traditional classroom training, (b) self-directed or distance learning, or (c) simulation, such as with the use of role-plays or virtual reality.

268

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

2.2.4. Training content Training content was coded as (a) intellectual, (b) interpersonal, (c) task-related, or (d) attitude; however, intellectual and task-related training content were the most frequently used in the primary studies. Intellectual training content included that which was taught with the intent of enhancing a general or specific knowledge, but which was not specifically intended to improve task (job) performance. Studies wherein training material focused on teaching or improving performance of a task were coded as task-related training content (e.g., administrative assistants learning to use of a specific computer program, nurses learning to draw blood or take a temperature, individuals learning the protocol specific to the job or training module, or persons learning a knowledge or skill relevant to a specific operation, such as that which may be required to fly an airplane or drive a car). Information which lays the foundation for task performance, but which in and of itself does not sufficiently prepare a trainee to perform the task, would be coded as intellectual content (e.g., an anatomy lesson would lay the knowledge foundation for a surgeon to perform an operation, but would not sufficiently prepare the surgeon to perform this task). Interpersonal training content might include training programs aimed at improving a supervisor's skill with respect to providing feedback, team-building exercises, conflict resolution tactics, marriage/family therapy, etc. Training content coded as “attitude” might include material covered in diversity awareness/appreciation programs, and those programs aimed at improving attitudes toward the organization, job satisfaction, attitudes toward conservation of energy, etc. 2.3. Analysis The meta-analytic methods outlined by Hunter and Schmidt (2004) were employed to analyze this data. Effect sizes were compiled and transformed into a common measure of effect, d, using formulas provided in Hunter and Schmidt (2004). The effect size d represents the difference between means independent of sample size. Essentially, d represents the population treatment effect, without reference to sample size. Sample sizes in the experimental and control groups were approximately the same in all studies. In meta-analysis, effect sizes are typically weighted by sample size to provide a sample-size weighted estimate of the observed effect size. The standard deviation of this effect size (which is composed of true variation as well as variation due to artifacts like sampling and measurement error) across the multiple studies was also computed to provide a sample-size weighted estimate of variability. Next, sample-size weighted mean observed effect sizes and standard deviations were corrected for unreliability in measures. Since reliability information was only sporadically available, reliability corrections were accomplished using artifact distribution meta-analysis (Hunter & Schmidt, 2004). Artifact distribution meta-analysis uses a weighted mean reliability estimate (drawn from those studies that did report sufficient reliability information) to correct for unreliability in the measures. In this study, sufficient data was available to correct only for unreliability within criterion (learning outcome) measures. Meta-analyses were conducted for all possible combinations of pre-training interventions and learning outcomes. Contingent upon the availability of data, a similar approach was taken for moderator analyses. Information provided in results tables of these meta-analyses includes the number of effect sizes compiled (k), the total sample size across all estimates (N), the sample-size weighted mean observed effect size (d), the sample-size weighted standard deviation (SDd), the reliability-corrected d-value (δ), the standard deviation of the reliability-corrected d-value (SDδ), the percent of observed variance attributable to sampling error (% SEV), the percent variance attributable to all artifacts (%ART), and the 80% credibility interval (80% CV) around the reliabilitycorrected d-value.

Table 2 Summary of reliability-corrected effect sizes for pre-training interventions and learning outcomes. Cognitive learning Avg. Advice

Meta-Cognitive

Organizer

Goal Orientation

Preparatory info

Overall General Specific Overall Think Aloud Why? Overall Graphic Textual Overall Perform Mastery Perf versus Mast Overall

.67* .71* .66* .61* .62 .60* .54* .53* .56* .71* – .72* – –

VK .79* .64* .83* .64* .56 .74* .68* .73* .62* .76 – .87 – –

Skill-based learning

Affective learning

KO

CS

Avg.

Acq

Comp

Auto

.53* .69* .50* .63* .95* .39* .52* .49* .59* .52* – .49* – –

.74* .73* .64* .34* −.10* .52* .23 .33 – – – – – –

.80* .88* .65* .51* .57* .40* .71* .86* .47* .71* .60* .89* – .48*

.79* – .48* .35* .36* .34* .69* .81* .46* .66* .46* 1.03* – –

.85* .88* .81* .65* .76* .46* .54 .73* .15 .64* .70* .59* – –

– – – – – – 1.07* 1.07* – .90* – – – –

Avg. .59* .37 .69* .40 .31 .48* .30 .29 – .85* 1.16* .76* .47* .45*

Att .82* – .69* .84 .04 – .33 .31 – .92* 1.34* – .74* .35*

SE .37 .37 – .26* .15 .33* – – – .88* 1.23* – .14* .56*

Disp – – – .52* – – – – – .92* 1.47* .61* .46* .41*

Note. Abbreviations used in this table are as follows: Perf versus Mast = Performance versus Mastery goal orientation — wherein the study explicitly compared these two goal conditions to one another rather than to a control, Avg. = average learning outcome, VK = verbal knowledge, KO = knowledge organization, CS = cognitive strategies, Acq = skill acquisition, Comp = skill compilation, Auto = skill automaticity, Att = attitude toward training, SE = self-efficacy, Disp = disposition toward using training content, GO = goal orientation. * = credibility interval of reliability-corrected effect size does not include zero.

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

269

Three types of hypotheses were tested in this paper. First, to test whether an intervention had a significant effect we checked if the 80% credibility intervals around the corrected mean included zero. If zero was not included, we inferred that the intervention had a significant/generalizable effect. The second type of hypotheses checked whether the intervention was more effective for one group of studies versus another (moderator analyses). Here we computed credibility intervals for the different sub-groups. Credibility intervals provide an estimate of the variability of reliability-corrected effects across studies. Wide credibility intervals signal the presence of a moderator (Hunter & Schmidt, 2004). According to Hunter and Schmidt (2004) we take the full set of dvalues and form sub-groups based on the moderator variables. Sub-group meta-analyses are conducted — a mean and its associated credibility interval are formed in each sub-group of effect sizes. Credibility intervals that do not overlap for sub-groups of d-values based on the moderator variables support the presence of a moderator. The third set of hypotheses tested whether the “average effect size” in the sub-groups of d-values based on the moderator variable differed significantly. For this, we computed the confidence interval around the mean [as δ ± 1.28 SDδ/sqrt(k)] and concluded that the mean effect sizes differed if the confidence intervals did not overlap. Although the confidence intervals overlapped in most cases, when the effects were substantially different, we discussed the implications of these differences. Finally, we recognize the potential the File-Drawer Effect (wherein non-significant findings are less likely to be published) threatens the validity of findings in meta-analysis (Rosenthal, 1979). As such, we also compute a File-Drawer k (FDk) for each meta-analysis using formulas provided in Hunter and Schmidt (2004). Specifically, we estimate the number of studies averaging null results which would be required to reduce the observed reliability-corrected d-value to a less significant finding (we used .05). In most cases, dozens, if not hundreds, of studies would be required to reverse our substantive findings. Similarly, we checked for outliers in our data-set by computing z-values for reported effect sizes within each type of pre-training intervention. We found no outliers within our database. 3. Results Table 2 summarizes the reliability-corrected effect sizes obtained from the meta-analyses examining the relationships between each pre-training intervention and learning outcome. Effects marked with an asterisk are considered to be “significant” values as their credibility intervals do not include zero (Hunter & Schmidt, 2004). Overall, medium to large effects6 are found for the role of pre-training interventions in influencing each type of learning and facilitating learning at each stage. Attentional advice and goal orientation appear to result in the most consistent learning gains. 3.1. Pre-training interventions 3.1.1. Attentional advice The detailed meta-analytic results for the role of attentional advice (including both typical formats, general and specific advice) in cognitive, skill, and affective learning are provided in Table 3. Hypothesis 1 predicted attentional advice would promote higher average learning scores for cognitive (H1a) and skill-based (H1b) learning. Results support this hypothesis. Medium to large effects are found for the role of attentional advice in promoting each type of learning, with the largest effects found for skill-based learning (average δ = .67 for cognitive learning, .80 for skill-based learning, and .59 for affective learning). Hypothesis 2 predicted general advice would yield higher cognitive and skill-based learning than specific advice. Hypothesis 2 was partially supported. Although no difference was found between general and specific advice for cognitive learning, consistent with H2b, higher learning averages resulted for general attentional advice with skill-based learning (δ = .88 for general advice versus .65 for specific advice). Results also suggest stronger effects for specific advice with affective learning outcomes (δ = .69 for specific advice versus .37 for general advice). 3.1.2. Meta-cognitive strategies The detailed meta-analytic results for the role of meta-cognitive strategies (including both typical approaches, “why?” and “think aloud”) in cognitive, skill, and affective learning are provided in Table 4. Hypothesis 3 predicted meta-cognitive strategies would promote cognitive (H3a), skill-based (H3b) and affective (H3c) learning. Results provide support for H3a and H3b, and partial support for H3c. Specifically, meta-cognitive strategies appear to enhance all levels of cognitive and skill-based learning (effects for cognitive learning range from δ = .34 for cognitive strategies to δ = .64 for verbal knowledge; effects for skill-based learning range from δ = .35 for skill acquisition to δ = .65 for skill compilation). With regards to affective learning, meta-cognitive strategies promote consistent gains for self-efficacy (δ = .26) and disposition (δ = .52), but are more variable for attitudes (δ = .84, but the credibility interval includes zero). Hypothesis 4 predicted “why-based” meta-cognitive strategies would promote higher learning gains than “think aloud” strategies. Results suggest this may be true with cognitive learning, as why-based strategies yielded more consistent positive effects for all levels of cognitive learning (i.e., although the think aloud δ-value for knowledge organization exceeded that of why strategies, the think aloud δ-values for average and verbal knowledge were not significant and the δ-value for cognitive strategies was negative; this suggests why-based strategies might be relied upon to provide more

6 For effect sizes, estimates of δ that are at least .20, .50, or .80 can be considered small, medium, or large effects, respectively (Cohen, 1992). If the credibility interval for a δ estimate does not include zero, Cohen's guidelines can be used to give a feel for the relative impact of various pre-training interventions on learning outcomes.

270

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

Table 3 Reliability-corrected effect sizes for the impact of attentional advice on learning outcomes. Meta-analysis

k

Cognitive learning — average Advice — overall 38 Advice — general 9 Advice — specific 29

N

d

SDd

δ

SDδ

3394 602 2792

.61 .63 .61

.26 0 .30

.67* .71* .66*

0

Cognitive learning — verbal knowledge Advice — overall 35 3099 Advice — general 9 602 Advice — specific 26 2497

.73 .58 .77

.52 0 .58

.79* .64* .83*

0

Cognitive learning — knowledge organization Advice — overall 32 2998 Advice — general 8 525 Advice — specific 24 2473

.49 .61 .47

.26 .20 .27

.53* .69* .50*

Cognitive learning — cognitive strategies Advice — overall 4 392 Advice — general 2 158 Advice — specific 2 234

.66 .73 .61

.34 0 .44

.74* .73* .64*

0

Skill-based learning — average Advice — overall 6 Advice — general 2 Advice — specific 4

475 244 231

.74 .88 .60

0 0 0

.80* .88* .65*

0 0 0

Skill-based learning — skill Advice — overall Advice — general Advice — specific

acquisition 5 400 1 169 4 231

.74 1.08 .48

.25 – 0

.79* – .48*

– 0

Skill-based learning — skill Advice — overall Advice — general Advice — specific

compilation 6 475 2 244 4 231

.79 .88 .81

.10 0 .24

.85* .88* .81*

0

.02 .02

– – –

– – –

– – –

.54 .37 .67

.31 .47 0

.59* .37 .69*

0

.76 .92 .69

0 – 0

.82* – .69*

0 – 0

.37 .37

.47 .47 –

.37* .37* –

Skill-based learning — automaticity Advice — overall 1 Advice — general – Advice — specific 1 Affective learning — average Advice — overall 4 Advice — general 2 Advice — specific 2

31 – 31

–

280 129 151

Affective learning — attitudes toward training Advice — overall 3 222 Advice — general 1 71 Advice — specific 2 151 Affective learning — post-training self-efficacy Advice — overall 2 129 Advice — general 2 129 Advice — specific – –

–

80% CV

%SEV

%ARTV

FDk

.31/1.03 .71/.71 .25/1.08

41.58 100 33.19

42.80 100 34.13

425 104 336

.63

.07/1.51 .64/.64 .03/1.63

15.53 100 12.00

16.08 100 12.37

476 95 374

.28 .22 .29

.17/.89 .41/.97 .13/.86

39.37 62.60 36.13

40.15 64.29 36.76

282 90 202

.37

.26/1.21 .73/.73 .04/1.24

27.95 100 15.61

29.53 100 15.61

49 27 22

.80/.80 .88/.88 .65/.65

100 100 100

100 100 100

83 33 44

.45/1.13 .48/.48

47.22 – 100

48.73 – 100

69 21 34

.72/.97 .88/.88 .50/1.12

83.92 100 57.64

86.94 100 57.64

89 33 61

– – –

– – –

– – –

.16/1.02 −.23/.98 .69/.69

38.61 22.58 100

38.94 22.58 100

100 – 100

100 – 100

.28 .32

.56

.47

.26 –

.10 .24

.34 .47

.82/.82 – .69/.69

.47 .47 –

−.23/.98 −.23/.98 –

22.58 22.58 –

22.58 22.58 –

– – –

39 13 25

43 – 26

13 13 –

Note. k = the number of effect sizes compiled, N = the total sample size across all estimates, d = the sample-size weighted mean observed effect size, SDd = the sample-size weighted standard deviation after removing sampling error variance, δ = the reliability-corrected d-value, SDδ = the standard deviation of the reliability-corrected d-value, %SEV = the percent of observed variance attributable to sampling error, %ART the percent variance attributable to all artifacts, and 80% CV refers to the 80% credibility interval around the reliability-corrected d-value. FDk indicates the number of missing studies averaging null results required to reduce the mean observed effect to .05. * = credibility interval of reliability-corrected effect size does not include zero.

consistent positive learning gains across levels of cognitive learning). However, think aloud strategies appear to be more useful for promoting skill-based (δ = .57 for think aloud versus .40 for why strategies).

3.1.3. Advance organizers The detailed meta-analytic results for this category of pre-training intervention (including both typical formats, graphic and textual) are provided in Table 5. Hypothesis 5 predicted advance organizers would promote cognitive (H5a) and skill-based (H5b)

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

271

Table 4 Reliability-corrected effect sizes for the impact of meta-cognitive strategies on learning outcomes. Meta-analysis

k

Cognitive learning — average MetaCog — overall 20 MetaCog — think 10 MetaCog — why? 10

N

d

SDd

δ

SDδ

80% CV

%SEV

%ARTV

FDk

1762 930 832

.55 .56 .55

.34 .44 .17

.61* .62 .60*

.37 .48 .18

.13/1.08 0/1.23 .36/.83

29.57 19.25 64.08

30.36 19.96 64.93

200 102 100

Cognitive learning — verbal knowledge MetaCog — overall 18 1557 MetaCog — think 9 789 MetaCog — why? 9 768

.58 .52 .65

.42 .54 .20

.64* .56 .74*

.45 .59 .21

.06/1.22 −.19/1.31 .46/1.01

22.37 14.21 56.88

23.07 14.45 59.97

191 85 108

Cognitive learning — knowledge organization MetaCog — overall 14 1181 MetaCog — think 7 509 MetaCog — why? 7 672

.58 .86 .36

.43 .43 .27

.63* .95* .39*

.47 .48 .30

.03/1.23 .35/1.56 .01/.77

21.76 25.22 36.57

22.07 26.02 36.63

149 113 43

.21

.34* −.10* .52*

.23 0 0

.05/.64 −.10/−.10 .52/.52

55.78 100 100

56.10 100 100

26 1 25

.17

.51* .57* .40*

0 0 .17

.51/.51 .57/.57 .18/.62

100 100 58.37

100 100 58.41

63 50 14

.11

.35* .36* .34*

0 0 .12

.35/.35 .36/.36 .19/.48

100 100 75.45

100 100 75.45

35 24 11

.65* .76* .46*

.12 0 .22

.49/.81 .76/.76 .18/.75

76.38 100 46.59

76.45 100 46.64

70 55 16

– – –

– – –

17 17

Cognitive learning — cognitive strategies MetaCog — overall 5 378 MetaCog — think 2 112 MetaCog — why? 3 266 Skill-based learning — average MetaCog — overall 7 MetaCog — think 5 MetaCog — why? 2

.31 −.08 .47

0 0

654 445 209

.50 .55 .39

0 0

Skill-based learning — skill MetaCog — overall MetaCog — think MetaCog — why?

acquisition 6 513 4 304 2 209

.34 .35 .33

0 0

Skill-based learning — skill MetaCog — overall MetaCog — think MetaCog — why?

compilation 6 554 4 345 2 209

.63 .74 .45

Skill-based learning — skill MetaCog — overall MetaCog — think MetaCog — why?

automaticity 1 15 1 15 – –

Affective learning — average MetaCog — overall 7 MetaCog — think 4 MetaCog — why? 3

544 252 292

.88 .88 –

.38 .29 .45

.12 0 .22

– – –

– – –

– – –

.35 .51

.40 .31 .48*

.37 .53 0

−.07/.87 −.38/.99 .48/.48

31.03 21.12 100

31.07 21.13 100

.84 .04

1.03 1.20 –

−.48/2.15 −1.49/1.57

11.41 12.24 –

11.41 12.24 –

.12/.40 −.12/.42 .33/.33

79.64 48.87 100

79.64 48.87 100

100 – –

100 – –

0

Affective learning — attitudes toward training MetaCog — overall 3 123 MetaCog — think 2 53 MetaCog — why? 1 70

.77 .04 1.32

.94 1.10 –

Affective learning — post-training self-efficacy MetaCog — overall 5 491 MetaCog — think 2 199 MetaCog — why? 3 292

.25 .15 .33

.10 .21 0

Affective learning — disposition MetaCog — overall 2 MetaCog — think 1 MetaCog — why? 1

.49 .56 .42

0 – –

287 141 146

– – –

–

– –

.26* .15 .33*

.11 .21 0

.52*

0 – –

–

.52/.52 – –

–

46 19 24

42 – –

20 4 17

18 – –

See note to Table 3.

learning outcomes. Results support this hypothesis (average δ = .54 for cognitive learning and .71 for skill-based learning). Further, although we made no specific hypothesis regarding the best format of advance organizers, stronger results were found for graphic rather than textual organizers across levels of skill-based learning outcomes (average δ = .86 for graphic organizers versus .47 for textual organizers).

272

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

Table 5 Reliability-corrected effect sizes for the impact of advance organizers on learning outcomes. Meta-analysis

k

Cognitive learning — average Organizer — overall 49 Organizer — graph 29 Organizer — text 20

N

d

SDd

δ

SDδ

80% CV

%SEV

%ARTV

FDk

4884 3154 1730

.49 .48 .51

.31 .32 .31

.54* .53* .56*

.35 .35 .34

.10/.98 .09/.98 .13/.99

30.61 28.44 34.55

31.74 29.23 35.99

431 250 184

knowledge 47 3943 25 2137 22 1806

.60 .66 .54

.34 .32 .35

.68* .73* .62*

.37 .35 .38

.21/1.15 .29/1.17 .13/1.11

31.62 34.03 30.57

34.92 34.97 35.06

517 305 216

Cognitive learning — knowledge organization Organizer — overall 32 3496 Organizer — graph 19 2559 Organizer — text 13 937

.47 .44 .54

.33 .35 .25

.52* .49 .59*

.37 .39 .28

.05/.98 .00/.99 .24/.95

26.52 20.46 48.79

27.29 21.09 50.05

270 149 127

Cognitive learning — cognitive strategies Organizer — overall 5 269 Organizer — graph 4 189 Organizer — text 1 80

.18 .25 0

.22 .26

.23 .33

.28 .34 –

−.13/.59 −.11/.77 –

62.71 56.66 –

63.52 57.95 –

13 16 –

.19

.71* .86* .47*

.12 0 .20

.56/.87 .86/.86 .22/.73

88.29 100 75.99

89.49 100 76.17

220 163 48

.16

.69* .81* .46*

0 0 .17

.69/.69 .81/.81 .24/.69

100 100 80.01

100 100 80.84

204 154 38

.44 .39 .31

.54 .73* .15

.48 .44 .33

−.08/1.15 .18/1.28 −.28/.58

30.23 35.23 46.57

30.41 35.42 46.59

53 48 4

Cognitive learning — verbal Organizer — overall Organizer — graph Organizer — text

Skill-based learning — average Organizer — overall 18 Organizer — graph 11 Organizer — text 6

773 475 238

.66 .79 .45

Skill-based learning — skill acquisition Organizer — overall 17 Organizer — graph 11 Organizer — text 5

735 475 200

.65 .75 .43

Skill-based learning — skill compilation Organizer — overall 6 Organizer — graph 4 Organizer — text 2

308 210 98

.49 .65 .14

Skill-based learning — skill automaticity Organizer — overall 2 108 Organizer — graph 2 108 Organizer — text – –

1.01 1.01 –

Affective learning — average Organizer — overall 5 Organizer — graph 4 Organizer — text –

.26 .24 –

301 238 –

–

–

.12 0

0 0

0 0 –

1.07* 1.07* –

.24 .29 –

.30 .29 –

0 0 –

1.07/1.07 1.07/1.07 –

100 100 –

100 100 –

.27 .34 –

−.05/.64 −.15/.72 –

54.64 45.89 –

54.95 46.02 –

−.05/.70 −.16/.77 –

52.45 42.72 –

52.72 42.85 –

–

– – –

– – –

– – –

– – –

Affective learning — attitudes toward training Organizer — overall 5 301 Organizer — graph 4 241 Organizer — text – –

.28 .26 –

–

–

.29 .36 –

Affective learning — post-training self-efficacy Organizer — overall 1 77 Organizer — graph 1 77 Organizer — text – –

.44 .44 –

– – –

– – –

– – –

.25 .31

.33 .31

38 38 –

21 15 –

23 17

See note to Table 3.

3.1.4. Goal orientation The detailed meta-analytic results for this category of pre-training intervention (including specific analyses for mastery goals, performance goals, and studies that compared performance goals with mastery goals) are provided in Table 6. Hypothesis 6 predicted the provision of a pre-training goal orientation would promote cognitive (H6a), skill-based (H6b), and affective (H6c) learning. Results support this hypothesis. Goal orientation appears to promote each type of learning, especially as compared with conditions where no overt goal is provided (average δ = .71 for cognitive learning, .71 for skill-based learning, and .85 for affective learning). Hypothesis 7 predicted mastery goals would promote stronger learning gains than performance goals. Results suggest this is true for both skill-based (average δ = .89 for mastery goals versus .60 for performance goals) and affective learning (we

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

273

were unable to test this hypothesis with cognitive learning). With affective learning, although results for performance goals were stronger than mastery goals when each orientation was compared to a no goal condition (δ = 1.16 for performance goals versus .76 for mastery goals), when studies specifically compared mastery with performance goals, stronger effects were found for mastery goals (δ = .47). 3.1.5. Preparatory information The results of meta-analyses associated with preparatory information are presented in Table 7. Hypothesis 8 predicted preparatory information would promote cognitive, skill, and affective learning. Results support this hypothesis for skill-based and affective learning (δ = .48 and .45, respectively, for average skill-based and affective learning); we were unable to test the hypothesis for cognitive learning. 3.2. Moderator analyses for training method and content Given research suggesting training program factors may differentially impact training effectiveness (e.g., training methods used and content trained; e.g., Arthur et al., 2003), we explored whether these factors moderated the efficacy of pre-training interventions examined in this study. Contingent upon data availability, separate meta-analyses were conducted for each moderator (type and category) for each pre-training intervention's role in enhancing average scores on cognitive, skill, and affective learning measures (Hunter & Schmidt, 2004; Viswesvaran & Sanchez, 1998). 3.2.1. Training method The results of the meta-analyses examining the training method moderator of the intervention/learning relationship are presented in Table 8. Moderated relationships were found for meta-cognitive strategies and advance organizers. Specifically, stronger effects for skill-based learning were found for meta-cognitive strategies employed with traditional training methods than with simulations (δ = .63 for traditional methods versus .36 for simulations). Stronger effects for cognitive learning were found for advance organizers combined with self-directed training methods (δ = .78) than with either traditional or simulation approaches (δ = .44 and .31, respectively). Preliminary results obtained from the small-k meta-analyses would suggest future research might explore whether training method also moderates the effect of advance organizers on skill-based and affective learning. 3.2.2. Training content The results of the meta-analyses examining the training content moderator are presented in Table 9. Moderated relationships were found for meta-cognitive strategies and preparatory information. Specifically, stronger effects for skill-based learning were found when meta-cognitive strategies were employed in the training of intellectual training content than for task-related content (δ = .70 and .38, respectively). Similarly, stronger effects for affective learning were found for preparatory information when they were employed in training programs focused on training/changing attitudes than task-related KSAs (δ = .32 for task-related KSAs versus .92 for attitudes). Preliminary results obtained from small-k meta-analyses would suggest future research might investigate the potential that training content also moderates the relationship between (1) attentional advice and skill-based learning and (2) preparatory information and affective learning. 3.3. Learning outcomes 3.3.1. Cognitive learning In general, cognitive learning appears to be most strongly affected by the use of attentional advice and goal orientation. Further, larger effect sizes for the first stage of cognitive learning, verbal knowledge, were found for specific attentional advice (δ = .83), why-based meta-cognitive strategies (δ = .74), and graphic advance organizers (δ = .73. The second stage of cognitive learning, knowledge organization, was most strongly impacted by general attentional advice (δ = .69) and “think aloud” meta-cognitive strategies (δ = .95). The highest level of cognitive learning, development of cognitive strategies, appears to be enhanced most by the provision of why-based meta-cognitive strategies (δ = .52) and attentional advice (δ = .74). 3.3.2. Skill-based learning Skill-based learning was aided by the use of advance organizers (especially graphic organizers; δ = .86), attentional advice (δ = .80), and by goal orientation (especially mastery goals; δ = .89). The first stage of skill-based learning, skill acquisition, was most influenced when mastery goals were provided and when graphic advance organizers were employed (δ = 1.03 and .81, respectively). Skill compilation, the next level of skill-based learning, was best fostered by think aloud meta-cognitive strategies (δ = .76), attentional advice (δ = .85), and pre-training goals (δ = .64). Less data was available for skill automaticity, but large effects were found for graphic organizers (δ = 1.07) and pre-training goals (δ = .90). 3.3.3. Affective learning Overall, affective learning was most strongly affected by the provision of a pre-training goal orientation and attentional advice (δ = .85 and .59, respectively). Attitudes toward training (δ = 1.34), post-training self-efficacy (δ = 1.23), and disposition towards future use of training material (δ = 1.47) appear to be most influenced when a performance goal orientation is used. Attentional advice, mastery goals, and preparatory information are also promising.

274

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

Table 6 Reliability-corrected effect sizes for the impact of goal orientation on learning outcomes. Meta-analysis

k

Cognitive learning — average Goal — overall 4 Goal — perform 1 Goal — mastery 2 Goal — P versus M 1

N

d

SDd

δ

192 51 90 51

.61 .80 .62 .41

.10 – .28 –

.71* – .72* –

knowledge 4 192 1 51 2 90 1 51

.72 .80 .82 .46

.66 – .98 –

.47

SDδ

80% CV

.11 –

.57/.86 –

.32

.31/1.13 –

.76 – .87 –

.70 – 1.04 –

−.13/1.66 – −.46/2.21 –

0 – 0 –

.52* – .49* –

0 – 0 –

.20

– – – –

– – – –

– – – –

759 372 346 51

.66 .56 .83 .20

.38 .22 .46 –

.71* .60* .89* –

Skill-based learning — skill acquisition Goal — overall 9 487 Goal — perform 4 225 Goal — mastery 4 221 Goal — P versus M 1 41

.66 .46 1.01 −.18

.46 0 .49 –

.66* .46* 1.03* –

Skill-based learning — skill compilation Goal — overall 8 562 Goal — perform 4 304 Goal — mastery 3 217 Goal — P versus M 1 41

.59 .64 .54 .44

.17 .30 0 –

.64* .70* .59* –

0 –

Skill-based learning — skill automaticity Goal — overall 2 148 Goal — perform 1 106 Goal — mastery 1 42 Goal — P versus M – –

.80 1.12 0 –

.44 – – –

.90* – – –

– – –

Affective learning — average Goal — overall 11 Goal — perform 5 Goal — mastery 3 Goal — P versus M 3

.77 1.04 .68 .42

.29 .22 .13 0

.85* 1.16* .76* .47*

0

Affective learning — attitudes toward training Goal — overall 6 348 Goal — perform 3 152 Goal — mastery 1 40 Goal — P versus M 2 156

.82 1.19 .06 .65

.27 0 – 0

.92* 1.34* – .74*

0 – 0

Affective learning — post-training self-efficacy Goal — overall 7 548 Goal — perform 4 277 Goal — mastery 1 125 Goal — P versus M 2 146

.77 1.11 .78 .12

.45 .30 – 0

.88* 1.23* – .14*

Affective learning — disposition Goal — overall 7 Goal — perform 3

.85 1.34

.47 .31

.92* 1.47*

Cognitive learning — knowledge organization Goal — overall 3 141 Goal — perform – – Goal — mastery 2 90 Goal — P versus M 1 51 Cognitive learning — cognitive strategies Goal — overall 1 51 Goal — perform – – Goal — mastery – – Goal — P versus M 1 51 Skill-based learning — average Goal — overall 12 Goal — perform 6 Goal — mastery 5 Goal — P versus M 1

719 317 205 197

400 174

– .42 .56

.20 – –

FDk

91.11 – 56.08 –

45 15 23 –

–

–

100 – 100 –

100 – 100 –

– – – –

– – – –

– – – –

– –

32.83 59.72 24.55 –

33.12 60.08 24.92 –

146 61 78 2

.07/1.26 .46/.46 .39/1.66

27.65 100 25.93 –

27.65 100 25.93 –

110 33 77 3

.40/.88 .27/1.12 .59/.59

67.84 38.47 100 –

68.01 38.59 100 –

86 47 29 8

23.72 – – –

23.72 – – –

30 21 – –

.49/.49

.41 .23 .48

17.65

%ARTV

17.65 – 9.46 –

.52/.52

– 9.46

.19/1.23 .30/.90 .28/1.51 –

.47 0 .50 –

56.08 –

–

–

89.44 –

–

Cognitive learning — verbal Goal — overall Goal — perform Goal — mastery Goal — P versus M

%SEV

–

.19 .33 –

.49

.27/1.53 – – –

54 – 31 7

25 – 15 –

3

10

.32 .24 .14

.45/1.26 .86/1.46 .58/.94 .47/.47

45.70 61.83 80.17 100

46.12 63.04 80.82 100

158 99 38 22

.30

.54/1.30 1.34/1.34 – .74/.74

52.85 100 – 100

53.53 100 – 100

92 68 1 24

.51 .33

.22/1.53 .81/1.65

21.96 43.54

22.89 45.54 – 100

101 85 15 3

26.43 48.22

112 77

– 0

– .14/.14

.52 .34

.27/1.58 1.03/1.90

– 100

26.35 47.90

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

275

Table 6 (continued) Meta-analysis

k

N

Affective learning — disposition Goal — mastery 2 Goal — P versus M 2

d

80 146

δ

SDd .56 .42

.19 0

SDδ .61* .46*

.20 0

80% CV .35/.87 .46/.46

%SEV

%ARTV

FDk

75.86 100

75.93 100

20 15

See note to Table 3.

4. Discussion Billions of dollars are spent annually on the design and implementation of a wide variety of training programs aimed at fostering immediate and long-term learning and transfer (Colquitt et al., 2000; Gagné et al., 1992; Galvin, 2001; Goldstein & Ford, 2002). Properly preparing trainees for forthcoming training may be one of the easiest and most effective ways a trainer can maximize the potential benefits of training and optimize the time spent in learning and practice (Baldwin & Ford, 1988; CannonBowers et al., 1998; Colquitt et al., 2000; Colquitt & Simmering, 1998; Frandsen, 1961; Gagné et al., 1992; Goldstein & Ford, 2002). Our research provides strong support for the idea that pre-training preparation enhances learning; specifically, each pre-training intervention we examined yielded positive outcomes for cognitive, skill-based, and affective learning. It is likely the success of these interventions is attributable to their effective support of trainees' internal learning processes at critical stages during learning (e.g., Gagné, 1996; Gagné et al., 1992). Specifically, each intervention taps into learning by focusing trainee attention on appropriate training stimuli, highlighting learning objectives and relationships between training material and existing knowledge, facilitating effective semantic encoding, eliciting the generation of responses at appropriate intervals, and/or facilitating feedback about progress toward learning via practice and response opportunities (cf. Bandura, 1986; Blanchard & Thacker, 2010; Gagné, 1996; Gagné et al., 1992). 4.1. Attentional advice The goal of attentional advice is to facilitate learning by directing the trainee's attention toward specific aspects of the training or practice curriculum, priming existing knowledge structures, and laying a foundation for developing task-related strategies for subsequent application and transfer (Lung & Dominowski, 1985; Phye, 1991). Since attentional advice fosters the assimilation of training material with the trainee's current knowledge, it was thought the provision of attentional advice would benefit cognitive learning. Results support this hypothesis. Accurate knowledge organization better prepares trainees for developing effective (cognitive) strategies for the application of new knowledge to novel situations/contexts. Thus, attentional advice lays a foundation for both the learning and transfer of cognitive knowledge. Large effects were also found for the role of attentional advice in promoting both skill acquisition and compilation. 4.1.1. Specific versus general advice? The impact of attentional advice on skill-based learning appears to be maximized when general, rather than specific, advice is offered. Given the number of individual difference variables and the variety of learning strategies adopted by individual trainees, it may be that general advice is more easily integrated into the trainee's learning tendencies and strategies (e.g., Ackerman, 1987). Alternatively, the provision of general rather than specific advice may require the trainee to take a more active role in deciphering and integrating the advice during learning, thus increasing the potential that the advice is attended to (e.g., Towler & Dipboye, 2001). Importantly, specific and general advices were comparable in their effects on cognitive learning outcomes. In deciding which to use during training, it may be important to consider the likely similarity between the training context and the transfer context. Specifically, identical elements theory would suggest that trainees are better prepared to retain and make use of training material

Table 7 Reliability-corrected effect sizes for the impact of preparatory information on learning outcomes. Meta-analysis

k

N

d

SDd

δ

SDδ

80% CV

%SEV

%ARTV

FDk

Cognitive learning average Skill-based learning average Affective learning average Attitudes toward training Post-training self-efficacy Disposition

1 2 6 4 5 3

179 184 657 549 408 484

.32 .40 .41 .33 .53 .36

– 0

– .48* .45* .35* .56* .41*

– 0 .24 0 .41 .28

– .48/.48 .14/.75 .35/.35 .04/1.08 .04/.77

– 100 43.54 100 26.14 29.24

– 100 43.89 100 26.19 30.20

5 14 43 22 48 19

See note to Table 3.

.22 0 .39 .25

276

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

Table 8 Training method as a moderator of the effect of pre-training interventions on learning. Cognitive learning

Advice

Meta-cognitive

Organizer

Goal orientation

Preparatory Info

Traditional Self-direct Simulation Traditional Self-direct Simulation Traditional Self-direct Simulation Traditional Self-direct Simulation Traditional Self-direct Simulation

Skill-based learning

k

N

δ

27 3 8 17 – 3 33 13 3 2 – 2 – – 1

2532 147 715 1496 – 266 3268 1463 153 102 – 90 – – 179

.70* .61* .55* .58* – .78* .44* .78* .31* .64* – .78* – – .32

SDδ .27 0 .36 .37 – .32 .32 .26 0 0 – 0 – – –

Affective learning

k

N

δ

SDδ

k

N

δ

2 1 – 4 – 3 2 1 2 4 – 8 1 – 1

111 169 – 368

.68* .91 – .63* – .36* .20* .68 .63* .47* – .88* .52 – .40

0 – – 0 – 0 0 – 0 .23 – .42 – – –

3 – 1 5 – 2 2 1 2 4 – 7 2 – 3

229 – 51 383 – 161 110 80 111 342 – 377 278 – 327

.54* – .62 .31 – .60* .54* −.28 .42* .77* – .94* .29* – .54*

286 90 186 122 313 – 446 91 – 92

SDδ .38 – – .33 – .37 0 – 0 0 – .44 0 – .27

See note to Table 3.

when it is as similar as possible to what will be encountered later in the transfer context (Noe, 2010; Thorndike & Woodworth, 1901). 4.2. Meta-cognitive strategies Focusing trainee attention to the underlying aspects of a problem and on interrelations between core concepts, as are the goals of pre-training meta-cognitive strategy interventions, was thought to yield superior performance on measures of both cognitive and skill-based learning; our results provide support for this proposition. Indeed, the provision of a meta-cognitive strategy prior to training benefited trainee cognitive and skill-based learning at all levels. Meta-cognitive strategies also appear to positively impact affective learning, particularly post-training self-efficacy. Trainees that monitor their progress in training by using metacognitive strategies may be better aware that they have learned during training, improving post-learning self-efficacy. 4.2.1. “Think aloud” versus “why” strategies? It is interesting to note that different formats of the meta-cognitive strategies were effective for each learning type. While a “think aloud” approach was most beneficial to skill-based learning, it was less consistently beneficial to cognitive learning. For cognitive learning, it appears that a “why?” approach is superior. This finding makes sense when you consider that cognitive learning requires both the development of knowledge structures that accurately represent the hierarchical interrelationships

Table 9 Training content as a moderator of the effect of pre-training interventions on learning. Cognitive learning

Advice

Meta-Cognitive

Organizer

Goal orientation

Preparatory info

See note to Table 3.

Intellectual Task KSA Attitude Intellectual Task KSA Attitude Intellectual Task KSA Attitude Intellectual Task KSA Attitude Intellectual Task KSA Attitude

Skill-based learning

k

N

δ

32 5 1 17 5 – 41 8 – 2 2 – – 1 –

2998 356 169 1492 529 – 4126 758 – 102 90 – – 179 –

.67* .62 .91 .59* .46* – .57* .40* – .64* .61* – – .32 –

SDδ .23 .55 – .39 .31 – .36 .20 – 0 .21 – – – –

k

N

3 1 – 3 4 – 10 8 – 1 9 – – 2 –

159 128 – 268 386 – 467 306 – 87 731 – – 183 –

.70* .04 – .70* .38* – .68* .77* – .35 .82* – – .33* –

Affective learning SDδ

k

N

δ

0 – – 0 0 –

3 1 – 5 2 – 3 2 – 1 8 – – 3 3

229 51 – 383 161 – 190 111 – 51 747 – – 520 137

.54* .62 – .30 .45* – .25 .42* – .58 .71* – – .32* .92*

.20 0 – – .34 – – .13 –

SDδ .38 – – .32 0 – .43 0 – – .40 – – 0 .10

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

277

Table 10 Suggested pre-training interventions by targeted learning outcomes. Targeted learning outcome

Pre-training intervention

Cognitive learning

Attentional advice (general) Meta-cognitive strategies Goal orientation (mastery) Attentional advice Meta-cognitive strategy (why?) Attentional advice (general) Meta-cognitive strategy (think aloud) Attentional advice (general) Meta-cognitive strategy (why?) Goal orientation (mastery) a Advance organizer (graphic) Attentional advice b Goal orientation (mastery) Advance organizer (graphic) Attentional advice Goal orientation Attentional advice Meta-cognitive strategy (think) Goal orientation Advance organizer (graphic) Goal orientation Attentional advice Goal orientation Attentional advice Goal orientation Preparatory information Goal orientation Meta-cognitive strategy preparatory information

Verbal knowledge Knowledge organization Cognitive strategies Skill-based learning

Skill acquisition

Skill compilation

Skill automaticity Affective learning Attitudes Self-efficacy Disposition

a b

Better when paired with task-related training content. Better when paired with intellectual training content.

between core concepts, and the development of strategies for when and how trained knowledge should be applied (Kraiger et al., 1993). Hence, it would seem that a meta-cognitive strategy which prompts trainees to continuously consider why the concepts are interrelated would better facilitate this form of learning. Results also suggest the relationship between provision of a meta-cognitive strategy and skill-based learning may be moderated by training method and/or training content. Specifically, trainees tended to perform better on skill-based assessments of learning when a meta-cognitive strategy was paired with (1) traditional training content rather than simulations and (2) intellectual content rather than task-based content. Importantly, in this case, the two moderators were confounded such that the majority of studies using traditional training methods trained intellectual content and the majority of studies using simulations trained taskrelated content, so it is as yet unclear whether training method, training content, or both moderates this relationship. 4.3. Advance organizers Consistent with the proposition that advance organizers provide trainees with a cognitive structure for forthcoming training content and facilitate the integration of training material with existing knowledge structures (Hannafin & Hughes, 1986; Herron, 1994; Luiten et al., 1980), our results suggest cognitive and skill-based learning benefit from the provision of an advance organizer. Importantly, graphic advance organizers appear to be superior to textual organizers in promoting skill-based learning. In many cases, graphic organizers provide a sequential or linear visual depiction of the task-related steps to be trained, potentially laying the groundwork for the development of more fluid and error-free task performance. We also found advance organizers were particularly effective in promoting cognitive learning when paired with self-directed training methods as compared with traditional and simulation methods, suggesting understanding the content and interrelationships among forthcoming training material is even more important when a trainee is directing his/her own learning efforts. Advance organizers stimulate recall of prior relevant knowledge and facilitate the encoding of training material into existing knowledge structures. Results suggest self-directed training methods benefit from interventions that help trainees to integrate new with existing knowledge. 4.4. Goal orientation Although individuals may enter the training context with a tendency towards a specific goal orientation, pre-training goal interventions establish a training-specific orientation (Cannon-Bowers et al., 1998). Given past research confirming the

278

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

importance of goal setting in learning (e.g., Mento et al., 1987), it was predicted that trainees provided with a specific goal (mastery- or performance-oriented) prior to training would outlearn those not provided with a goal on all learning measures. Indeed, on average, provision of a goal, regardless of its orientation, yielded gains in cognitive, skill, and affective learning. 4.4.1. Mastery versus performance goals The data suggests perception of mastery goals fosters skill learning gains, while perception of a performance goal fosters affective learning. Importantly, when studies specifically compared mastery and performance goals in affective learning, mastery goals consistently yielded better learning performance. Therefore, it is probably preferable to encourage the adoption of mastery goals since achieving higher levels of learning will likely result in better long-term retention and performance in the transfer domain. 4.5. Preparatory information It was predicted the provision of preparatory information would benefit cognitive, skill-based, and affective learning, as it likely minimizes the effects of stress and shock on learning and performance during training by alerting trainees about what to expect during training and strategies for how to deal with unexpected or negative stimuli improves reactions to the training program and its content (Inzana et al., 1996). Although insufficient data was available to examine cognitive learning, positive effects were found for skill-based learning, as well as indicators of affective learning, including self-efficacy, attitudes toward training, and intention to use training material in the transfer context. Importantly, moderator analyses suggest preparatory information yields stronger affective learning gains when the training content is focused on changing/improving attitudes than when it is focused on taskbased content. 4.6. Implications for training design The most important conclusion to be drawn from this research is that pre-training interventions promote cognitive, skill, and affective learning at each and every level of learning. Remarkably, these benefits are recognized regardless of the training method employed or the material that is trained. In fact, the primary studies included in this meta-analysis reported learning that resulted from a wide variety of training systems (e.g., work-related training programs, such as typing, proofreading, computer programming, diversity and sexual harassment seminars, team-building programs, and even police operations/ military combat training, and general education programs wherein a variety of subjects were covered), further supporting their flexibility and adaptability for use in a range of training initiatives. That said, our results suggest the effectiveness of pre-training interventions varies according to the learning outcome of interest, reinforcing the need for trainers to conduct thorough needs assessments prior to designing training programs so potential benefits of pre-training interventions can be maximized. Although attentional advice and goal orientation resulted in the most consistent learning gains across types and levels of learning, our results suggest specific interventions and formats may be more or less beneficial depending on the desired learning outcome. Another important implication of these results for training design is that specific learning outcomes can actually be targeted, and hence maximized, by employing specific pre-training interventions. For example, if a trainer is most interested in maximizing skill acquisition then he or she may consider using a graphic organizer or mastery goals. However, if the development of efficient cognitive strategies for applying training material is the outcome of greatest interest, then the educator would be welladvised to employ attentional advice. Further, if trainers are interested in promoting favorable trainee reactions to training (since many training programs are evaluated, at least in part, by “smile sheets”, wherein trainees rate how much they liked the program), they may be well-advised to use attentional advice or pre-training goals, as these maximize learner attitudes while also yielding strong benefits to cognitive and skill-based learning. While attitudes toward training do not necessarily predict transfer, research suggests when training programs are known to be enjoyable, future trainees will be more motivated to attend. Desire to attend a training program is an important pre-requisite to learning motivation (Colquitt et al., 2000; Noe, 1986). In Table 10, we summarize suggestions regarding which pre-training interventions to use depending on the type and level of learning outcome desired. 7 4.7. Study limitations One limitation of this research relates to the relatively small number of primary studies located relevant to a few of the reported meta-analyses. Although analyses of the more general relationships (i.e., between pre-training interventions and learning averages) incorporated a relatively large number of studies, the more focused analyses (e.g., between specific formats of pretraining interventions and specific learning outcomes, particularly higher-level learning outcomes, or in certain moderator analyses) were at times based upon a limited number of studies. For the sake of completeness, we report all possible metaanalyses, even those of only a few studies. Although we are unaware of concrete guidelines regarding the minimum number of

7 Importantly, although learning is an important pre-requisite to retention and transfer of training material (Arthur et al., 2003; Colquitt et al., 2000; Kraiger et al., 1993), it does not guarantee improved job performance. Future research is needed to test the assumption that the enhanced learning that results from the provision of pre-training interventions actually translates into improved retention and transfer.

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

279

studies required to conduct meta-analyses, we recognize the small number of studies available for some of our meta-analyses heightens concerns regarding second-order sampling error (Hunter & Schmidt, 2004; Spector & Levine, 1987). We were careful not to interpret nor base our substantive conclusions on the results of small-k meta-analyses, and advise readers to use caution when interpreting the results of small-k meta-analyses as future research is needed to confirm our results. Importantly, although the k was smaller, the actual sample sizes were in the hundreds and often in the thousands, signifying some generalizability of the conclusions. Furthermore, by integrating findings across decades of research and over multiple research domains, this study provides a comprehensive investigation of the role of pre-training interventions in the enrichment of learning from training. Thus, while conclusions regarding certain intervention formats or specific learning outcomes would benefit from further investigation, this study was successful in providing a snapshot of what we know about the utility of pre-training interventions, yielding some concrete implications and best practices for trainers and other educators, and informing research as to issues and topics that will most profit from future exploration. A second potential limitation concerns the lack of reliability data available in many of the studies included in this meta-analysis. All measures are affected by measurement error and thus the correlates reported here may underestimate true relationships. In an attempt to address this potential concern, artifact distribution meta-analysis was employed. This technique utilizes the reported reliability estimates to make reliability corrections to the reported correlations using a weighted reliability estimate. Third, the effect sizes compiled in this meta-analysis were culled from experimental designs wherein the effect on learning of a pre-training intervention administered to an experimental group was compared with the level of learning achieved in a control group. One potential concern associated with experimental designs is the extent to which the controls were adequately applied. We were not able to code for the adequacy of controls applied in the primary studies as relevant information was inconsistently reported. However, all studies included in our meta-analytic database were subjected to the peer-review process prior to their publication, which should have the result of weeding out poorer quality studies. That said, the inadequacy of controls is likely to be random (in their effect on d-values; e.g., whereas the Hawthorne effect may attenuate effect sizes, demoralization may inflate them) across studies and meta-analytic cummulation is more likely to provide a better estimate. 4.8. Questions and implications for research in training design The results of this project yield several important questions for future research. First, what is the best way in which to design and administer pre-training interventions? For example, we know that graphic advance organizers, as opposed to textual organizers, are most beneficial to skill-based learning. However, is it preferable to organize core concepts linearly or hierarchically? Similarly, we now know that “why?” meta-cognitive strategies are in some cases superior to “think aloud” approaches. But, how should the “why?” questions be phrased? (For example, is it better to ask “why is this concept related to the next?” or “why am I performing this particular step/task?”) Also, should “why?” be supplemented with “how?” and “what?” questions? (For example, should the learner also consider “how does this action impact the next step?” and “what are the consequences of this action to the next?”). It would seem that the difficulty of “why?” questions would also have implications for learning at different levels. Further, perhaps as learning progresses, different types of meta-cognitive strategies should be utilized. For example, in looking at cognitive learning outcomes, we note that why-based strategies promoted learning at initial knowledge acquisition and later during development of cognitive strategies, but during knowledge organization, think aloud strategies yielded significantly stronger gains. Perhaps during initial stages of knowledge acquisition and later in the development of strategic knowledge regarding how and when to use new facts/concepts, it is important to reinforce conceptual interrelationships (e.g., using a why-based strategy), but when attempting to cement and semantically organize this knowledge, it helps to think aloud about the concepts and interrelationships. The potential that different forms of meta-cognition and associated metacognitive strategies are needed as learning progresses is an important avenue for future research. Second, future research might also devise standard methods by which to create and implement these interventions. Such standardization would not only benefit practitioners who wish to utilize these approaches, but also facilitate a greater degree of construct validity in subsequent research. Specifically, the pre-training interventions used in the primary studies included in our meta-analytic database ranged from quite basic adaptations of pre-training interventions to much more elaborate interventions. Unfortunately, the authors of these primary studies tended to provide relatively little information that might be used to determine the quality of the interventions and their appropriateness given the content of the training programs. Logically, intervention quality and appropriateness have implications for the potential they may enhance trainee learning during training. Such variability in the interventions used in the primary studies likely explains some of the variance we found in our reported effect sizes. Although our results suggest even simple adaptations of pre-training interventions are likely to afford improvements in trainee learning, trainers should not be misled into thinking that any haphazard attempt will result in substantial learning gains. For example, when deciding whether to incorporate pre-training interventions into a training program and how to design them, trainers must consider the number and type of trainees to be included in the training initiative (e.g., trainee complexity, age, learning tendencies, and experience with training material), the resources they have available to devote to intervention design (e.g., time, experience, knowledge, and funds), match to training content, and desired learning processes and outcomes. The importance of being systematic in the design of pre-training interventions further underscores the need for trainers to have conducted a complete and competent needs assessment prior to undertaking any aspect of training design. Third, this study focused on the role of pre-training interventions in facilitating learning. Learning is only one outcome of interest in evaluating training programs; retention and transfer of training material are also important. What effect do these interventions have on other outcomes of interest? Similarly, there is an assumption that enhancing learning will promote gains at

280

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

other levels of evaluation, but it is important to consider there can often be a gap between learning and retention and transfer. The potential pre-training interventions also promote retention and transfer should be the focus of future research. Fourth, a relatively small number of studies were located for certain interventions, formats, and potential moderators. Future research may explore the validity of conclusions drawn here on the basis of small sample sizes. For example, few studies investigated the role of pre-training interventions in promoting skill automaticity. Skill automaticity, the highest level of skillbased learning, is the most valuable level of learning to influence, as it reflects skill proficiency which most likely results in improved performance in the transfer domain. In particular, given we found positive effects for attentional advice and metacognitive strategies for lower levels of skill-based learning, future research might examine whether these interventions are also effective in promoting skill automaticity. It may be a more longitudinal research design is needed to pick up this sort of skill development. Similarly, the majority of studies included in our database used traditional training methods and focused on intellectual training content. With increasing organizational use of computer-based and hands on training as well as an increased tendency for organizational training initiatives to delve into other domains (e.g., diversity awareness and attitude change programs), more research is needed to confirm our findings with other training methods and content domains. We found very few studies to have investigated the role of preparatory information in cognitive and skill-based learning. However, Cannon-Bowers et al. (1998) argued that preparatory information might be particularly useful for improving cognition and skill in low confidence trainees or those susceptible to stress. Future research should investigate the potential preparatory information is useful in promoting cognitive and skill-based learning outcomes, and whether it may offer additional value to low self-efficacy trainees. Additional attention is also needed to explore the comparative effect of performance versus mastery goals. Primary studies found positive effects for both types of goals when compared to a no-goal condition, and sometimes the effects for performance goals were larger than for mastery goals. However, in studies that specifically compared mastery with performance goals, stronger effects tended to be found with mastery goals. Additional primary research is needed to tease apart the effects of various pretraining goal orientations on different types and levels of learning. Last, some pre-training interventions might be combined to yield additional learning gains. For instance, the provision of attentional advice regarding what to expect and focus on in the training curriculum followed by a meta-cognitive strategy for how to process the information during learning and practice or a pre-training goal regarding mastering certain aspects of the training material might be particularly useful combinations of interventions. Fifth, past research has highlighted the importance of attending to the role of individual differences in training design (e.g., Ford et al., 1998; Mathieu, Tannenbaum, & Salas, 1992). Are there individual differences that limit the effectiveness of certain pretraining interventions in learning? For example, cognitive ability, self-efficacy, learning orientation and motivation, and learner anxiety have been found to affect learner performance in training programs (Colquitt et al., 2000); future research might explore the extent to which they moderate the pre-training intervention/learning relationships identified here. Similarly, do learning tendencies (e.g., learning styles identified by Kolb, 1999; Felder & Silverman, 1988, etc.) interact with certain interventions to differentially affect learning (e.g., might graphic advance organizers be more useful to visual than verbal learners)? Do metacognitive tendencies interact with certain meta-cognitive strategies? Are individuals with learning disabilities more or less likely to benefit by the provision of a pre-training intervention? Exploration of such issues will serve to enhance our understanding of the utility of pre-training interventions. Finally, an interesting idea for future research is to explore the role of pre-training interventions in error management training programs where trainees are taught to use errors as an opportunity to learn.8 Both conventional wisdom and recent research findings suggest trainees learn more from training programs when they have the opportunity to learn from mistakes (Heimbeck, Frese, Sonnentag, & Keith, 2003; Keith & Frese, 2005, 2008). Error management training promotes the use of meta-cognition before, during, and after training to plan, monitor, and evaluate the use of training content (Keith & Frese, 2005; Noe, 2010). Although these trainees may make more mistakes and take longer to finish training, they likely engage in a deeper level of cognitive processing than the average trainee, thus promoting better memory and recall of training content. Logically, trainers may be able to enhance the benefits of error management training by incorporating pre-training interventions which support these learning processes. For example, pre-training meta-cognitive strategies might be used to encourage the use of appropriate types of meta-cognition during error management training (Keith & Frese, 2005); similarly, mastery goals may be emphasized over performance goals to encourage trainees to devote the necessary energy to learning from errors (Heimbeck et al., 2003).

4.9. Conclusion Now, more than ever, trainers are required to demonstrate the value-added products of their training efforts. As the goals of training programs generally focus on facilitating a desired change in performance within the transfer context, it is crucial that trainers maximize the potential that learning will occur and appropriately assess the extent to which desired levels of learning have taken place. We compiled the extant literature relevant to common pre-training interventions and evaluated their impact on learning. Our results reveal these flexible and easily implemented interventions are consistently successful in promoting and enhancing important learning outcomes.

8

We thank an anonymous reviewer for this suggestion.

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

281

References Ackerman, P. L. (1987). Individual differences in skill learning: An integration of psychometric and information processing perspectives. Psychological Bulletin, 102, 3−27. Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review, 89, 369−406. Arthur, W., Jr., Bennett, W., Jr., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. The Journal of Applied Psychology, 88, 234−245. Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41, 63−105. Bandura, A. (1977). Social learning theory. Upper Saddle River, NJ: Prentice Hall. Bandura, A. (1982). Self-efficacy mechanisms in human behavior. The American Psychologist, 37, 122−147. Bandura, A. (1986). Social foundations of thought and action. Englewood Cliffs, NJ: Prentice Hall. Berardi-Coletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger, E. R. (1995). Metacognition and problem-solving: A process-oriented approach. Journal of Experimental Psychology. Learning, Memory, and Cognition, 21, 205−223. Blanchard, P. N., & Thacker, J. W. (2010). Effective training: Systems, strategies, and practices, 4th Ed Upper Saddle River, NJ: Pearson Prentice Hall. Burke, M. J., & Day, R. R. (1986). A cumulative study of the effectiveness of managerial training. The Journal of Applied Psychology, 71, 232−245. Butcher, K. R. (2006). Learning from text with diagrams: Promoting mental model development and inference generation. Journal of Education & Psychology, 98, 182−197. Cannon-Bowers, J. A., Rhodenizer, L., Salas, E., & Bowers, C. (1998). A framework for understanding pre-practice conditions and their impact on learning. Personnel Psychology, 51, 291−320. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155−159. Cohen, M. S., Freeman, J. T., & Wolf, S. (1996). Metarecognition in time-stressed decision-making: Recognizing, critiquing, and correcting. Human Factors, 38, 206−219. Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. The Journal of Applied Psychology, 85, 678−707. Colquitt, J. A., & Simmering, M. S. (1998). Conscientiousness, goal orientation, and motivation to learn during the learning process: A longitudinal study. The Journal of Applied Psychology, 83, 654−665. Coutinho, S., Wiemer-Hastings, K., Skowronski, J. J., & Britt, M. A. (2005). Metacognition, need for cognition, and use of explanations during ongoing learning and problem-solving. Learning and Individual Differences, 15, 321−337. Day, E. A., Arthur, W., Jr., & Gettman, D. (2001). Knowledge structures and the acquisition of complex skill. The Journal of Applied Psychology, 86, 1022−1033. DeShon, R. P., & Gillespie, J. Z. (2005). A motivated action theory account of goal orientation. The Journal of Applied Psychology, 90, 1096−1127. Desoete, A., & Roeyers, H. (2006). Metacognitive macroevaluations in mathematical problem-solving. Learning and Instruction, 16, 12−25. Felder, R. M., & Silverman, L. K. (1988). Learning and teaching styles in engineering education. Journal of Engineering Education, 78, 674−681. Fisher, S. L., & Ford, J. K. (1998). Differential effects of learner effort and goal orientation on two learning outcomes. Personnel Psychology, 51, 397−420. Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. The American Psychologist, 34, 906−911. Ford, J. K., Smith, E. M., Weissbein, D. A., Gully, S. M., & Salas, E. (1998). Relationships of goal orientation, metacognitive activity, and practice strategies with learning outcomes and transfer. The Journal of Applied Psychology, 83, 218−233. Foster, J., & Macan, T. H. (2002). Attentional advice: Effects on immediate, delayed and transfer task performance. Human Performance, 15, 367−380. Frandsen, A. N. (1961). Educational psychology: The principles of learning. New York: McGraw-Hill. Gagné, R. M. (1996). Learning processes and instruction. Training Research Journal, 1, 17−18. Gagné, R. M., Briggs, L. J., & Wager, W. W. (1992). Principles of instructional design, 4th Ed. Fort Worth, TX: Harcourt Brace Jovanovich. Galvin, T. (2001). Industry Report 2001.Training Available on-line: http://209.11.43.229/training/images/pdf/2001_industry_report.pdf Goldstein, I. L., & Ford, J. K. (2002). Training in organizations, 4th Ed. Belmont, CA: Wadsworth. Greeno, J. G. (1980). Some examples of cognitive task analysis with instructional implications. In R. E. Snow, P. A. Federico, & W. E. Montague (Eds.), Aptitude, learning, and instruction, Vol. 2. (pp. 1−21)Hillsdale, NJ: Erlbaum. Hannafin, M. J., & Hughes, C. W. (1986). A framework for incorporating orienting activities in computer-based interactive video. Instructional Science, 15, 239−255. Heimbeck, D., Frese, M., Sonnentag, S., & Keith, N. (2003). Integrating errors into the training process: The function of error management instructions and the role of goal orientation. Personnel Psychology, 56, 333−361. Herron, C. (1994). An investigation of the effectiveness of using an advance organizer to introduce video in the foreign language classroom. The Modern Language Journal, 78, 190−198. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings, 2nd Ed. Thousand Oaks, CA: Sage Publications. Inzana, C. M., Driskell, J. E., Salas, E., & Johnston, J. H. (1996). Effects of preparatory information on enhancing performance under stress. The Journal of Applied Psychology, 81, 429−435. Johnson, J. H., & Cannon-Bowers, J. A. (1996). Training for stress exposure. In J. E. Driskell & E. Salas (Eds.), Stress and human performance (pp. 223−256). Hillsdale, NJ: Lawrence Erlbaum. Keith, N., & Frese, M. (2005). Self-regulation in error management training: Emotion control and metacognition as mediators of performance effects. The Journal of Applied Psychology, 90, 677−691. Keith, N., & Frese, M. (2008). Effectiveness of error management training: A meta-analysis. The Journal of Applied Psychology, 93, 59−69. Kolb, D. (1999). Learning style inventory. Boston, MA: Hay/McBer Training Resources Group. Kozlowski, S. W. J., & Bell, B. S. (2006). Disentangling achievement orientation and goal setting: Effects on self-regulatory processes. The Journal of Applied Psychology, 91, 900−916. Kozlowski, S. W. J., Gully, S. M., Brown, K. G., Salas, E., Smith, E. M., & Nason, E. R. (2001). Effects of training goals and goal orientation traits on multidimensional training outcomes and performance adaptability. Organizational Behavior and Human Decision Processes, 85, 1−31. Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. The Journal of Applied Psychology, 78, 311−328. Langan-Fox, J., Platania-Phung, C., & Waycott, J. (2006). Effects of advance organizers, mental models, and abilities on task and recall performance using a mobile phone network. Applied Cognitive Psychology, 20, 1143−1165. Langan-Fox, J., Waycott, J. L., & Albert, K. (2000). Linear and graphic advance organizers: Properties and processing. International Journal of Cognitive Ergonomics, 4, 19−34. Luiten, J., Ames, W., & Ackerson, G. (1980). A meta-analysis of the effects of advance organizers on learning and retention. American Educational Research Journal, 17, 211−218. Lung, C. T., & Dominowski, R. L. (1985). Effects of strategy instructions and practice on nine-dot problem solving. Journal of Experimental Psychology. Learning, Memory, and Cognition, 11, 804−811. Mathieu, J. E., Tannenbaum, S. I., & Salas, E. (1992). Influences of individual and situational characteristics on measures of training effectiveness. Academy of Management Journal, 35, 828−847. Mento, A. J., Steel, R. P., & Karren, R. J. (1987). A meta-analytic study of the effects of goal setting on task performance: 1966–1984. Organizational Behavior and Human Decision Processes, 39, 52−83. Narciss, S., Proske, A., & Koerndle, H. (2007). Promoting self-regulated learning in web-based learning environments. Computers in Human Behavior, 23, 1126−1144. Noe, R. A. (1986). Traniees' attributes and attitudes: Neglected influences on training effectiveness. Academy of Management Review, 11, 736−749.

282

J. Mesmer-Magnus, C. Viswesvaran / Human Resource Management Review 20 (2010) 261–282

Noe, R. A. (2010). Employee training and development (5e). New York: McGraw-Hill Irwin. Phye, G. D. (1989). Schemata training and transfer of an intellectual skill. Journal of Education & Psychology, 81, 347−352. Phye, G. D. (1991). Advice and feedback during cognitive training: Effects at acquisition and delayed transfer. Contemporary Educational Psychology, 16, 87−94. Phye, G. D., & Sanders, C. E. (1994). Advice and feedback: Elements of practice for problem solving. Contemporary Educational Psychology, 19, 286−301. Rosenthal, R. (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638−641. Seijts, G. H., & Latham, G. P. (2005). Learning versus performance goals: When should each be used? Academy of Management Executive, 19, 124−131. Souvignier, E., & Mokhlesgerami, J. (2006). Using self-regulation as a framework for implementing strategy instruction to foster reading comprehension. Learning and Instruction, 16, 57−71. Spector, P. E., & Levine, E. L. (1987). Meta-analysis for integrating study outcomes: A Monte Carlo study of its susceptibility to type I and type II errors. The Journal of Applied Psychology, 72, 3−9. Thorndike, E. L., & Woodworth, R. S. (1901). The influence of improvement in one mental function upon the efficiency of other functions: Functions involving attention, observation, and discrimination. Psychological Review, 8, 553−564. Towler, A. J., & Dipboye, R. L. (2001). Effect of trainer expressiveness, organization, and trainee goal orientation on training outcomes. The Journal of Applied Psychology, 86, 664−673. Viswesvaran, C., & Sanchez, J. I. (1998). Moderator search in meta-analysis: A review and cautionary note on existing approaches. Educational and Psychological Measurement, 58, 77−87. Volet, S. E. (1991). Modeling and coaching of relevant metacognitive strategies for enhancing university students' learning. Learning and Instruction, 1, 319−336. Wagner, R. K. (1978). Tacit knowledge in everyday intelligent behavior. Journal of Personality and Social Psychology, 52, 1236−1241. Wexley, K., & Latham, G. P. (2002). Developing and training human resources in organizations (3e). Upper Saddle River, NJ: Prentice Hall.