Psychological Reports, 2011, 108, 2, 420-436. © Psychological Reports 2011

EVALUATION OF A SOCIAL SKILLS PROGRAM BASED ON SOCIAL LEARNING THEORY, IMPLEMENTED IN A SCHOOL SETTING1 BETH A. SHERIDAN

DOUGLAS A. MacDONALD

Saybrook University

University of Detroit Mercy

MARK DONLON AND BETH KUHN

KATIE McGOVERN

Glengarda Child and Family Services

Saybrook University

HARRIS FRIEDMAN University of Florida Summary.—Using a sample of 647 Canadian children in kindergarten to Grade 3 (325 boys, 322 girls), the present study evaluated the perceived effectiveness of Skillstreaming (McGinnis & Goldstein, 2003), a widely known social skills program implemented to target the development of four skill sets, i.e., listening, following directions, problem-solving, and knowing when to tell. Results indicated significant postprogram improvements in all skills as well as in ratings of overall prosociality obtained from both classroom teachers and mental health staff, with medium to large effect sizes obtained from teachers’ and mental health professionals’ ratings, respectively. Additional analyses yielded significant but weak moderator effects of grade and preprogram prosocial functioning for teacher ratings but no consistent moderator effects for children’s sex or school location (i.e., urban versus rural) regardless of rater.

In response to aggressive behavior observed in school-aged children and youth (Swearer, Grills, Haye, & Tam Cary, 2004), interventions and programs aimed at developing prosocial behavior have been garnering greater attention from, and use by, mental health professionals and educators. Concurrent to their rise in popularity, and in keeping with the growing emphasis on evidence-based practice, such programs have become the focus of research aimed at establishing their effectiveness (Schneider, 1992; Sugai, Horner, Dulap, Hieneman, Lewis, Nelson, et al., 2000). Overall, the emerging literature suggests that many of these programs have potential for ameliorating the behavior and skills of children (e.g., Schneider & Byrne, 1985; Sasso, Melloy, & Kavale, 1990; Beelmann, Pfingsten, & Lösel, 1994; Stage & Quiroz, 1997; McMahon & Wells, 1998; Barrera, Biglan, Taylor, Gunn, Smolkowski, Black, et al., 2002; Wilson, Lipsey, & Derzon, 2003). Critical examination of many of the existing studies, however, reveals problems with conceptualization and/or limited generalization of treatment effects (Ogilvy, 1994; Harris, 1998; Anastas, 2000). By implicaAddress correspondence concerning this article to Beth A. Sheridan, Ph.D., 5 Windmill Lane, Atkinson, NH 03811 or e-mail ([email protected]). 1

DOI 10.2466/10.11.17.PR0.108.2.420-436

ISSN 0033-2941

SKILLSTREAMING EFFECTIVENESS

421

tion, there is a need for investigations that evaluate programs in an ecologically valid manner to provide a better sense of the robustness of program outcomes in real-world settings. The purpose of the present study was to examine the effectiveness of one widely utilized prosocial skills program called Skillstreaming (Goldstein & McGinnis, 1997; McGinnis & Goldstein, 1997, 2003) in a context where children’s behavioral difficulties are commonly observed and often cause the greatest problems—the school setting. Despite its empirical soundness, current research on Skillstreaming is lacking in both quality and informativeness. For example, although McGinnis and Goldstein (1997) mention that the program has been cited or investigated in over 100 studies, virtually all of these studies use small samples (i.e., less than 150; Goldstein, Blake, Cohen, & Walsh, 1971; Gutride, Goldstein, & Hunter, 1973; Sasso, et al., 1990), which is likely to affect the generalizability of findings to the broader population. Furthermore, the findings from these studies have yielded inconsistent results (e.g., Baran, 1988; Sasso, et al., 1990; Dana, 1998) and virtually no attention has been given to assessing differential program outcomes as a function of several potentially important confounding variables including children’s sex, grade level, location of school (e.g., rural versus urban), and pre-program prosocial functioning. As such, many questions exist regarding whether or not Skillstreaming is effective and, if so, for which populations of students. Skillstreaming Skillstreaming is a social skills program based on social learning theory, in which dysfunctional behavior is viewed as the product of social and problem-solving skills deficits in a variety of areas (Goldstein & McGinnis, 1997; McGinnis & Goldstein, 1997, 2003). Under the assumption that such skills can be acquired, maintained, and generalized to multiple settings and situations, the Skillstreaming program consists of curricula tailored to meet the needs of children in three developmental ranges, i.e., early childhood, elementary school-age, and adolescence (Goldstein & McGinnis, 1997; McGinnis & Goldstein, 1997, 2003). In each curriculum, the program covers a broad range of observable social skills in teachable units. Identified skill deficiencies are ameliorated through instruction, modeling, role-playing, feedback, and encouragement to generalize the skill beyond the learning environment. To illustrate more fully, the Skillstreaming in Early Childhood (McGinnis & Goldstein, 2003) curriculum consists of 40 skill areas that are organized into six groups: beginning social skills, school-related skills, friendshipmaking skills, dealing with feelings, alternatives to aggression, and dealing with stress. The elementary school curriculum (Goldstein & McGinnis,

422

B. A. SHERIDAN, ET AL.

1997) comprises 60 skills in five skill categories. Likewise, the adolescent program (Goldstein & McGinnis, 1997) has five skill categories made up of 50 skill lessons. All skills within each curriculum are ideally presented in a small group format, by two instructors familiar with the Skillstreaming philosophy and curriculum. However, the curriculum can be delivered to larger groups (e.g., an entire classroom) if needed. The program entails identifying students who have behavioral difficulties or appear to have a limited repertoire of social skills. Various evaluators (e.g., psychologist, teacher, parent, guidance counselor, social worker) make observational notes and complete a checklist assessing the child’s behavior in the identified skill sets. Children are asked to complete a similar checklist in which they rate their own behavioral performance. Children with comparable skill deficits are individually informed about the topic of the group and invited to participate. The group begins with identifying the target skill to be developed and having the students collectively generate a list of group rules to aid in developing cohesiveness and mutual respect in the sessions. Following the introduction of the skill topic, nine steps are implemented in the Skillstreaming lesson. First, the group is led in a discussion identifying and defining what the skill is in operational terms. One way this is accomplished is by providing the students with skill cards outlining the skill steps, or these can be written on an easel or chalkboard. Then, the instructor models two examples of the skills. Students are asked to share a related incident they have struggled with in the recent past. Specific students are invited or volunteer to participate in a role-play activity. One student describes a situation in which the skill may be useful and is designated as the primary character. This student chooses other group members to play other relevant characters, while remaining students are observers. The main character is asked to follow and verbalize the behavioral steps while enacting the scene. Co-actors, who are asked to stay in character while observing students, may be asked to watch for one particular sequence of the skill set. Role-play activities are repeated so that each group member has an opportunity to participate and practice each skill. Following each role-play, the co-actor and observers provide feedback to the main character. Here, students are asked to point out how well the specific behavioral steps were demonstrated and encourage the main character to use the skill outside the group setting. The main character is then asked to respond to the feedback. This provides group leaders with the opportunity to observe how well the student assesses his or her own performance following feedback. Specific portions of a role-play can be repeated and practiced as needed. Homework is assigned after each session, to encourage the students to practice the skills beyond the group setting. These homework assign-

SKILLSTREAMING EFFECTIVENESS

423

ments are initially simple (Level 1), providing a reasonable assurance of success, and gradually shift to more demanding (Level 2) and complex (Level 3) tasks. Homework completion is charted individually by the student and the group. Students are rewarded for successful mastery of a skill. Establishing parental involvement is essential prior to teaching the identified skills and assigning homework. The final step is the selection of another student to be the main role-play character. The previous eight steps are repeated until all group participants are proficient at the designated skill. Follow-up checklists are completed after all students have reached mastery of target skills. These data allow educators, administrators, and parents to make informed decisions regarding future curriculum initiatives. While there are a variety of social skills programs available, Skillstreaming is a particularly promising program because it appears to embody the five features of effective programs identified in the literature (i.e., collaborative, targets early childhood, flexibility across settings and populations, generalized effects beyond setting, and integrated evaluation of effectiveness; Yoshikawa, 1995; Kazdin, 1996; Frick, 1998; Connor, 2002; Sugai, Horner, Sailor, Dunlap, Eber, Lewis, et al., 2005). Skillstreaming is collaborative in that the instructional format encourages enlisting the support and involvement of parents (e.g., through instructional workshops) and teachers (McGinnis & Goldstein, 1997). It is designed for the elementary school population, including children in the pivotal years prior to age eight (Ziegler, Taussig, & Black, 1992; Kazdin, 1996; Frick, 1998; McMahon & Wells, 1998). The program is also flexible; portions of the program have been administered with some success to meet the needs of at-risk students (Baran, 1988; Sasso, et al., 1990; Dana, 1998). Likewise, the curriculum has been tailored so as to facilitate its delivery by a wide range of professionals in addition to mental health professionals, including teachers (FarmerDougan, Viechtbauer, & French, 1999), nurses and attendants (Goldstein, et al., 1971), and prison inmates (Hayman & Weiss-Cassady, 1981). Also, the program content can be modified to accommodate the specific needs of a child, a classroom, a school, or a community. Last, data collection is an integral part of the Skillstreaming program, providing baseline and follow-up data from multiple sources (i.e., parent, teacher). Research Expectations The principal aim of the present investigation was to examine the Skillstreaming program in terms of its potential to facilitate prosocial skills development. More specifically, the intent of this study was to evaluate the effectiveness of Skillstreaming as implemented with children in a school-based classroom setting.

424

B. A. SHERIDAN, ET AL.

It was generally expected that children who participated in the Skillstreaming program would show statistically significant gains in ratings of social skills functioning. Further, it was expected that such gains would be seen regardless of sex, grade (kindergarten through Grade 3), school location (urban vs rural), and preprogram functioning (low vs adequate). In terms of effect size, available literature on the effectiveness of skillstreaming programs generally suggests a medium effect size for applications of social skills programs; in a meta-analysis, Gansle (2005) reported a mean effect size of 0.34 for social skills programs; Lindberg, Canning, and Gregory (2001) reported a mean effect size of 0.43. It was anticipated that this application of the Skillstreaming program would produce comparable effect sizes. Method Program Content and Implementation The Skillstreaming program as investigated in this study is one of a range of behavioral support services offered to publicly funded elementary schools by a community-based mental health facility located in Southwestern Ontario, Canada. The program was delivered to several intact classes ranging in size from about 8 to 35 students over a period of six weeks throughout the course of the 2002–2003, 2003–2004, and 2004–2005 academic years. The explicit intent of the mental health facility was to provide a service to early school-aged children so as to reduce the likelihood of subsequent problems with behavior at school and home. Consistent with this, the program was completed by children in kindergarten through Grade 3 in the classroom environment. The program curriculum utilized was taken from McGinnis and Goldstein’s Skillstreaming in Early Childhood (2003). This version of Skillstreaming provides 40 possible skills to choose as the focus of instruction. Based upon the mental health agency’s experience with the schools and the types of behavioral concerns most commonly reported, as well as its own initial efforts to evaluate the quality of its service, the agency elected to include four core skills each time it implemented the program and permitted schools to select up to an additional three skills, if desired. The four skills were Listening, Following directions, Problem-solving, and Knowing when to tell. Though all of these skills were delivered in a form highly consistent with McGinnis and Goldstein, the latter skill (i.e., Knowing when to tell) was supplemented with language taken from Coloroso (2002) to help better clarify whether students are tattling on a classmate in order to get them “in” trouble or telling on a friend in order to get their friend “out” of trouble; some circumstances may include both elements. Program delivery was directed and implemented by a team of men-

SKILLSTREAMING EFFECTIVENESS

425

tal health practitioners (one social worker and two child and youth workers) in collaboration with classroom teachers who were trained in the program. During the second week of the program, a presentation of the Skillstreaming program was given to parents with efforts to encourage the parents to use the program with their children at home. Participants In each classroom, up to 10 children were selected as cases for preand postprogram evaluations completed by both the teacher and the mental health staff. At the request of the mental health staff before the start of the program, the teacher identified four children, two boys and two girls, who appeared to demonstrate low prosocial behavior. The remaining six children (three boys and three girls) were selected at random by the team. This sampling strategy generated a sample of 647 children consisting of 325 boys and 322 girls. The mean age of the sample was 6.4 yr. (SD = 1.1, range = 4–10). The sample included children in kindergarten (n = 132), Grade 1 (n = 245), Grade 2 (n = 181), and Grade 3 (n = 89). In terms of school location, 435 children attended schools in an urban environment, while 212 attended schools in rural areas. Finally, 254 participants were children identified by teachers at preprogram as having low prosociality, i.e., as rarely or never engaging in socially appropriate behavior with peers and/ or adults, and 393 participants were children selected quasi-randomly from classes receiving the Skillstreaming program. Measures and Data Collection Teacher ratings.—Teachers completed pre- and postratings of each of the selected children in terms of the frequency with which they were seen to engage in behavior consistent with the four core skills while at school. As well, the teachers provided a rating of the frequency of each child’s overall prosocial behavior. All ratings were based upon a visual analog response scale of 0 to 100 (0: Never, 50: Sometimes, 100: Always). Mental health staff ratings.—Each selected child met individually with a mental health staff member at pre- and postprogram to complete a brief structured evaluation of their prosocial skills. In particular, for each of the four core skills, children were presented with a scenario relevant to the skill involving a fictional boy named Opie, asking how Opie should respond. The scenarios were taken from McGinnis and Goldstein (2003) and were the same at pre- and postprogram. Team members rated each response in terms of its perceived prosociality using a 100-point visual analog scale (0: Not at all, 50: A fair amount, 100: A great deal). In addition, using the same response scale, the team members also gave a rating reflecting their perception of each child’s overall prosocial functioning. Last, at postprogram, an additional scenario developed by the research-

426

B. A. SHERIDAN, ET AL.

ers was included in the evaluation (completed only by the mental health staff) that was designed to get information about the extent to which children could generalize beyond the school environment the understanding and use of prosocial skills learned through the program. Unlike the other scenarios derived from McGinnis and Goldstein, this one placed the child in the role of the main character and asked the child what he or she would do in that situation (“You are playing outside at home and some children in your neighborhood start to tease you and call you names. What would you do?”). Analyses While the data for this study lend themselves to analysis via hierarchical modeling (e.g., children are nested within grades, within schools, within school locations), analyses were completed using analyses of variance (ANOVAs) due to the relative straightforwardness of interpretation. Also, since the main purpose of this study was to simply evaluate program outcomes—analyses for moderating effects of only secondary import— the use of ANOVA was pragmatic. Ten repeated-measures ANOVAs were completed, five for teachers’ and five for mental health professionals’ ratings, to test for pre- and postprogram differences in all four specified skills and for overall prosocial functioning. To evaluate potential moderator variables, 16 mixed-model split-plot ANOVAs were done for each of the four skills and overall prosociality for both teachers and mental health staff separately for a total of 32 ANOVAs. In these analyses, pre- and postratings for each skill and overall prosociality served as the within-subject variables while child sex (male vs female), school location (urban vs rural), grade (kindergarten through Grade 3), and preprogram ratings of low prosocial functioning by teachers (low vs adequate) were used as separate between-subjects variables. Following the completion of the ANOVAs, to examine whether or not education and maturation may account for program outcomes, postprogram ratings on all skills for lower grades were compared to preprogram ratings (e.g., postprogram kindergarten ratings were compared to preprogram ratings for Grades 1 to 3; Grade 1 postprogram ratings were compared to preprogram ratings for Grades 2 and 3; Grade 2 postprogram ratings were compared to preprogram ratings for Grade 3). Results Descriptive statistics for all pre- and postprogram ratings from teachers and mental health staff were calculated for the entire sample and separate repeated-measures ANOVAs were conducted (Table 1). For all analyses, statistically significant results were obtained, indicating that ratings of children’s prosocial skills across all four skill areas and overall prosocial

427

SKILLSTREAMING EFFECTIVENESS TABLE 1 Repeated-measures ANOVAs For Teacher and Mental Health Staff Ratings For the Entire Sample (N = 647) Variable

Preprogram

Postprogram

M

M

SD

72.52 74.91 72.07 74.07 74.68

21.34 21.14 22.42 22.26 20.23

8.98 9.11 14.01 14.87 10.93

80.89 84.47 81.15 80.83 84.23

20.48 14.76 19.72 23.48 12.83

44.58 44.23 41.36 44.91 38.76

SD

Teacher ratings Skill 1 63.54 24.57 Skill 2 65.80 24.43 Skill 3 58.07 26.00 Skill 4 59.21 26.12 Overall 63.75 22.70 Mental health staff ratings Skill 1 36.30 26.49 Skill 2 40.22 25.74 Skill 3 39.79 25.91 Skill 4 35.92 26.02 Overall 45.47 23.29

Difference

F1,646

p

η2

144.72 155.21 328.06 332.73 259.90