This article was downloaded by: [Michelle B. Stein] On: 03 September 2013, At: 06:18 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Personality Assessment Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hjpa20

SCORS–G Stimulus Characteristics of Select Thematic Apperception Test Cards a

b

c

a

Michelle B. Stein , Jenelle Slavin-Mulford , Caleb J. Siefert , Samuel Justin Sinclair , a

a

d

Megan Renna , Johanna Malone , Iruma Bello & Mark A. Blais

a

a

Psychological Evaluation and Research Laboratory (PEaRL) , Massachusetts General Hospital and Harvard Medical School b

Department of Psychology , Augusta State University

c

Department of Psychology , University of Michigan–Dearborn

d

Department of Psychiatry, New York University , Langone Medical Center and School of Medicine Published online: 27 Aug 2013.

To cite this article: Journal of Personality Assessment (2013): SCORS–G Stimulus Characteristics of Select Thematic Apperception Test Cards, Journal of Personality Assessment, DOI: 10.1080/00223891.2013.823440 To link to this article: http://dx.doi.org/10.1080/00223891.2013.823440

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions

Journal of Personality Assessment, 1–11, 2013 C Taylor & Francis Group, LLC Copyright  ISSN: 0022-3891 print / 1532-7752 online DOI: 10.1080/00223891.2013.823440

SCORS–G Stimulus Characteristics of Select Thematic Apperception Test Cards MICHELLE B. STEIN,1 JENELLE SLAVIN-MULFORD,2 CALEB J. SIEFERT,3 SAMUEL JUSTIN SINCLAIR,1 MEGAN RENNA,1 JOHANNA MALONE,1 IRUMA BELLO,4 AND MARK A. BLAIS1 1

Downloaded by [Michelle B. Stein] at 06:18 03 September 2013

Psychological Evaluation and Research Laboratory (PEaRL), Massachusetts General Hospital and Harvard Medical School 2 Department of Psychology, Augusta State University 3 Department of Psychology, University of Michigan–Dearborn 4 Department of Psychiatry, New York University, Langone Medical Center and School of Medicine There has been surprisingly little research into the stimulus properties of the Thematic Apperception Test Cards (TAT; Murray, 1943). This study used the Social Cognition and Object Relations Scale–Global Rating Method (SCORS–G; Stein, Hilsenroth, Slavin-Mulford, & Pinsker, 2011; Westen, 1995) to explore the stimulus properties of select TAT cards in a clinical sample. The SCORS–G is a theoretically based and empirically validated measure of object relations that has widely been used in TAT research. A sample of 80 patients referred for psychological assessment at a large Northeastern hospital were administered the TAT (Cards 1, 2, 3BM, 4, 13MF, 12M, and 14) as part of their assessment battery. Trained raters scored the narratives using the SCORS–G. The SCORS–G ratings were analyzed to determine the nature and degree of object representation “pull” both across and within the TAT cards. The results showed that Cards 3BM and 13MF exhibited the greatest card pull for negative pathological object representations, and Card 2 displayed the highest card pull for adaptive and mature object representations. Both clinical and research related implications are discussed.

The Thematic Apperception Test (TAT; Murray, 1943) is a freeresponse and implicit measure of personality. Murray described it as “a method of revealing some of the dominant drives, emotions, sentiments, complexes, and conflicts of personality” (p. 3). In considering what accounts for individual differences Aronow, Weiss, and Reznikoff (2001) posited that there are three contributors to the TAT response: card stimulus, testing environment, and the patient’s inner world. It is typically left up to the examiner to differentiate what aspects of an individual’s responses can be attributed to each of these sources. To date, the majority of the TAT literature has focused on understanding how the patient’s inner world and experience becomes reflected in TAT responses. There is a relative dearth of research examining the role of the testing environment and the stimulus pull of individual TAT cards. This study begins to address this gap in the literature by analyzing the stimulus properties of a select group of commonly employed TAT cards. Stimulus pull can be defined as the “tendency of the stimulus to evoke or predispose certain perceptual and/or affective responses in the subject (Peterson & Schilling, 1983, p. 273). “It denotes a reality which does not come from within the subject . . . but corresponds more closely to the actual picture perception, a more or less objective report of what’s out there” (Peterson & Schilling, p. 266). To date, the literature on stimulus pull is limited and empirical studies are “embarrassingly lacking” (Peterson & Schilling, 1983; Zubin, Eron, & Schumer, 1965, p. 60). Further, a significant portion of the existent research on stimulus pull is contained in doctoral dissertations and book

chapters. As a result, clinicians and researchers might have difficulty accessing this valuable information. There are a number of potential reasons for the present state of TAT research. For example, such research is burdensome in terms of obtaining and rating a suitable number of protocols. Further, TAT rating scales are not widely known nor routinely employed in clinical practice. And there is no single widely accepted scoring system for the TAT like the Comprehensive System for the Rorschach (Exner, 1995). Jenkins (2008) reported that the most common measures used to rate TAT narratives include the Social Cognition and Object Relations Scale (SCORS; Westen, 1995) and the Defense Mechanism Manual (DMM; Cramer, 1991). However to our knowledge, neither of these measures has been employed to examine stimulus characteristics of individual cards. Although generally underexamined, there have been some studies examining stimulus pull within and across individual cards. The TAT cards used in this study (1, 2, 3BM, 4, 13MF, 12M, and 14) are the focus of the literature review, even though additional cards might have been used in previous research. Table 1 provides a description of each card and Table 2 provides a summary of prior research findings. Early empirical research on TAT stimulus pull focused on rating the frequency of themes and emotions evoked by the cards in both patient and nonpatient samples (Eron, 1948, 1950, 1953; Rosenzweig & Fleming, 1949). There was also research into the perceptual clarity associated with the cards (e.g., the extent to which people were viewing the characters, objects, and social perception of characters similarly; Murstein, 1972; Rosenzweig & Fleming, 1949). Eron (1950) developed normative data as well as created TAT rating scales for emotional tone and checklist for themes and interpretations levels within and across cards. Eron’s work on developing normative data was groundbreaking. It allowed researchers and clinicians to determine the extent to which patient narratives deviated from those of a particular diagnostic group

Received July 28, 2012; Revised February 25, 2013. Address correspondence to Michelle B. Stein, Psychological Evaluation and Research Laboratory (PEaRL), Massachusetts General Hospital and Harvard Medical School, One Bowdoin Square, 7th Floor, Boston, MA 02114; Email: [email protected]

1

STEIN ET AL.

2 TABLE 1.—Thematic Apperception Test card descriptions. TAT Card 1 2 3BM 4 13MF 12M

Downloaded by [Michelle B. Stein] at 06:18 03 September 2013

14

Description (Murray, 1943) A young boy is contemplating a violin which rests on a table in front of him. Country Scene; in the foreground is a young woman with books in her hand, in the background a man is working in the fields and an older woman is looking on. On the floor against a couch is the huddled form of a boy with his head bowed on his right arm. Beside him on the floor is a revolver. A woman is clutching the shoulders of a man whose face and body are averted as if he were trying to pull away from her. A young man is standing with downcast head buried in his arm. Behind him is the figure of a woman lying in bed. A young man is lying on a couch with his eyes closed. Leaning over him is the gaunt form of an elderly man, his hand stretched out above the face of the reclining figure. The silhouette of a man (or woman) against a bright window. The rest of the picture is totally black.

and from stimulus properties of a card such as themes evoked, emotional tone, perceptual accuracy, and verbal productivity and word count. Other researchers followed suit and began using specific rating scales to examine unique aspects of stimulus pull. For example, Alvarado (1994) asked students to make comparisons of the relative similarity of cards over multiple trials and employed a more experimental or social psychological method for examining stimulus pull. Using a visual triad task (e.g., subjects are asked to make comparisons of the relative similarity of items in each domain), 21 TAT cards (1, 2, 3BM, 4, 5, 6BM, 6GF, 7GF, 8BM, 8GF, 9BM, 9GF, 10, 12BG, 12F, 12M, 13MF, 17BM, 18BM, 18GF, and 20) were presented to subjects over multiple trials. Participants were asked to rate similarity between cards as well as intensity of emotional response evoked by each card. Consensus analysis was used to determine whether responses of subjects were idiosyncratic or whether a common response existed within the group. Using this method, Alvarado found that Cards 1 and 2 were rated as being similar to each other; likewise, 13MF and 4 were rated as similar and 3BM and 12M were ranked as similar. Interestingly, the results also indicated that the emotional tone of the card partially mediated the similarity effect. Alvarado interpreted these findings as suggesting that participants rated cards as being more or less similar based on the degree to which they “pulled” for common aspects of object representations such as affect tone, emotional investment, and self-esteem and identity. Ehrenreich (1990) examined stimulus pull for Cards 1, 2, 3BM, 12M, and 13 MF using narrative content gathered from a nonpatient sample. He assessed drive expression (Pine, 1960), defensive patterns (as measured across four tiers ranging from mature to extremely pathological), dependency (e.g., rated as dependent, interdependent, independent/autonomous, or noninteractive, p. 16), and locus of control. He found that narratives for Cards 3BM and 13MF contained the lowest level of defenses, whereas narratives for Card 2 tended to contain the highest, most adaptive level defenses. With the exception of locus of control, narratives for all cards differed significantly from one another, suggesting that individual cards “elicited specific themes and were differentially arousing” (p. 15) across measures of drive expression, defense, and dependency.

More recently, Seedwaldt (2006) conducted a study focusing on stimulus characteristics across and within TAT cards (1, 2, 3BM, 4, 5, and 13MF) using a well-validated method for conceptualizing interpersonal themes, the Core Conflictual Relationship Theme (CCRT; Luborsky & Crits-Christoph, 1998). The CCRT was originally created to be used with Luborsky’s Relationship Anecdotes Paradigm interview or from transcripts in supportive-expressive therapy. In this study, the CCRT was adapted for the TAT. Notably, aspects of Murray’s scoring system (needs and press) informed the CCRT standard categories. The CCRT was developed to identify a “central relationship pattern, script, or schema that each person follows in conducting relationships” (Luborksy & Crits-Christoph, 1998, p. 3). The CCRT uses three components—the Wish (W), Response from Other (RO), and the Response of Self (RS)—to identify repetitive relationship patterns expressed in narrative data (a more detailed description of the CCRT method can be found in Luborsky & Crits-Christoph, 1998). In Seedwaldt’s study, Cards 1, 2, 4, and 13MF exhibited strong and moderate stimulus pull for W, RO, and RS. In contrast, Card 3BM had high stimulus pull for content pertaining to the RO and RS, but weak card pull for W. Overall, this study highlights that these selected TAT cards vary in their degree of stimulus pull and elicit different interpersonal themes. An important strength of this study is the use of an empirically based scale for coding TAT narratives, the CCRT method, that has both research and clinical relevance. Aronow et al. (2001), Bellak and Abrams (1997), and Holt (1978) provided comprehensive reviews of the research on stimulus pull. They also provided insights from their clinical experience on the common interpersonal and emotional themes seen across cards, some of which were described earlier. These are predominantly summarized in Table 2. To expand this line of research, we used the Social Cognition and Object Relations Scale–Global Rating Method (SCORS–G; Stein, Hilsenroth, Slavin-Mulford, & Pinsker, 2011; Westen, 1995) to examine the stimulus pull of the TAT cards listed in Table 1 as they relate to the emotional and cognitive aspects of interpersonal perception and functioning. These are the TAT cards, which we administer in our clinic and have data for. The SCORS–G is a widely used empirically based measure for coding TAT narrative data in a reliable and valid manner (see Stein et al., 2011, for details). One advantage of the SCORS–G is that it rates multiple domains of psychological functioning. Westen et al. (1991) stated that the TAT is a “particularly good test for assessing object relations because the stimulus is unambiguously social and the subjects are likely to provide enough detail in describing characters and relationships as to provide considerable access to cognitive and affective motivation patterns related to interpersonal functioning” (p. 56). For these reasons, the SCORS–G appears to be an ideal measure in examining the stimulus properties of selected TAT cards. In summation, there is limited research on stimulus pull, with the majority of studies on this topic being conducted several years ago before the most common coding systems for TAT narratives were developed. Previous studies have also tended to limit examination of pull to the global level and most have utilized nonclinical and college samples. This study builds on previous research and pursues four goals. First, using a clinical sample, we examined seven TAT cards for pull using an expertrater coding system (i.e., the SCORS–G). This system allows

STIMULUS CHARACTERISTICS AND SCORS–G

3

TABLE 2.—Summary of research on stimulus pull. Emotional Tone Card 1

1. Neutral to moderately sad1, 2 2. Most frequent outcome is happy1

Card 2

1. Moderately happy1, 8 2. Most frequent outcome is happy1, 8

Downloaded by [Michelle B. Stein] at 06:18 03 September 2013

Card 3BM

Card 4

1. 2. 3. 4. 5. 6. 1. 2. 3. 4. 5.

6. 7. 1. Moderately to 1. very sad1, 2, 8, 9 2. 3. 4. 5. 6. 7. 8. 9. 10. 1. Neutral to mildly 1. negative1 2. Outcome is moderately 2. happy1 3.

Card 13MF 1. Sad1, 8 2. Guilt and remorse1, 3, 4, 5

Card 12M

1. Moderately sad1

Card 14

1. Positive affect1, 5

CCRT Wish7

Frequently Occurring Themes

1. 2. 3. 4. 5. 6. 7. 1. 2. 3. 4. 5. 6. 7. 1. 2. 3. 4. 5. 6. 7.

Aspiration3, 13 Parental pressure/conflict3, 4, 5, 6 Belongingness1 Achievement5, 6, 13 Intermediate levels of defense14 External locus of control14 Ambivalence between personal ambition and families ties/role conflict3, 5, 9 Aspiration/occupational concern/achievement1, 4, 5, 6, 10, 11 Economic pressure1, 4 Parental pressure1, 4, 6 Conception of family relationships and attitudes as giving, supportive versus barren and depriving13 High level defenses14 Internal locus of control14 Depressive themes3, 5, 9, 13 Suicide1, 3, 5, 9 Loss of relationship/grief 3, 5, 9 Failure1, 5 Punishment/parental pressure1, 5 Behavior disorder1 Coping resources5 Guilt13 Low-level defenses14 External locus of control14 Male–female roles/sexual attitudes and conflicts such as a love triangle or infidelity3, 5, 9, 13 Pressure/departure/nurturance to and from partner1, 4 Impulsivity, aggression, and dangerousness2, 4, 5 Sex1, 3, 5, 6, 9, 13 Aggression1, 3, 5, 6, 9, 10 Hostility10, 11 Suicide1, 5 Death/illness1, 4, 5, 6, 12 Low-level defenses14 External locus of control14 Hypnotism1 Religion1 Illness/death1 Emotional attitudes toward father5 Ideas regarding therapy/therapist5, 9, 13 Intermediate levels of defense14 External locus of control14 Happy reminiscence1, 4, 5 Aspiration/hope/ determination1, 4, 5, 13 Dreaming, looking, seeking inspiration, and relaxing1, 5 Contemplation9 Occupational concerns4 Curiorsity1 Frustrations/worries13

CCRT Response of Other7

CCRT Response of Self 7

1. To achieve and help others 1. Controlling 2. To feel good and comfortable 2. Rejecting and opposing

1. Disappointed and depressed 2. Self-controlled and self-confident

1. To achieve and help others

1. Rejecting and opposing 2. Strong

1. Self-controlled and self-confident 2. Disappointed and depressed

1. To be distant and avoid conflicts

1. Rejecting and opposing 2. Helpful

1. Disappointed and depressed 2. Helpless

Female protagonist 1. To be loved and understood 2. To be close and accepting Male protagonist 3. To be distant and avoid conflicts 4. To achieve and help others 1. To be loved and understood 2. To be distant and avoid conflicts 3. To be close and accepting

Female protagonist 1. Rejecting and opposing 2. Upset Male protagonist 3. Rejecting and opposing

Female protagonist 1. Unreceptive Male protagonist 2. Self-controlled and self-confident 3. Disappointed and depressed

1. Rejecting and opposing 2. Upset

1. Disappointed and depressed 2. Anxious and ashamed

N/A

N/A

N/A

N/A

N/A

N/A

Note. CCRT = Core Conflictual Relationship Theme (Luborsky & Crits-Christoph, 1998). 1 Eron (1950). 2Goldfried and Zax (1965). 3Bellak & Abrams (1997). 4Eron (1953). 5Holt (1978). 6Rosenzweig & Fleming (1949). 7Seedwaldt (2006). 8Alvarado (1994). 9Aronow, Weiss, & Reznikoff (2001). 10Campus (1976). 11Murstein (1965). 12Eron (1948). 13Rapaport, Gill, & Schafer (1968). 14Ehrenreich (1990).

us to examine evidence for pull at the global level (i.e., several dimensions of object relations influenced) and the specific level (i.e., one or two dimensions influenced). Second, if evidence for pull was found, we wanted to determine which cards were more neutral, which had higher levels of global pull, and which had higher levels of specific pull. Third, we compared and contrasted our results to previous findings for all cards containing

some level of pull. Finally, we examined how word count was related to SCORS–G variables across specific cards. The overarching purpose of this study was to determine and highlight any evidence for card pull on the SCORS–G. In so doing, and by contextualizing our findings within the preexisting literature, we hoped to outline various needs for future research that would have implications for SCORS–G scoring and interpretation.

STEIN ET AL.

4

METHOD

Downloaded by [Michelle B. Stein] at 06:18 03 September 2013

Participants Participants were 80 outpatients (48 male) referred for psychological assessments in the outpatient psychiatry department of a large Northeastern academic medical center. All participants were assessed between September 1, 2008 and July 1, 2010. On average, the sample was 38 years old (SD = 15.0) with 14.5 years of education (SD = 3.0). The most common referral diagnoses were depressive disorder (47%), anxiety disorder (18%), and bipolar disorder (17%). However, these are outpatients who have a complex clinical picture, often have multiple diagnoses, and are being referred for a diagnostic clarification. Approximately 41% of participants had history of psychiatric hospitalization. Our sample was primarily White (91%). Procedure Outpatients received a battery of standard clinical instruments including the TAT. After the TAT cards were administered, they were transcribed (written verbatim as patient was creating narrative), deidentified, and scored by two independent raters using the SCORS–G. The SCORS–G was used to rate all TAT narratives from each patient. Raters (M. Stein and J. Slavin-Mulford) are expert raters who have used this scale for at least 5 years and previously completed manualized training on the SCORS–G (Hilsenroth, Stein, & Pinsker, 2007; Westen, 1995). Good (> .60) to excellent (> .74) reliability was attained by both raters in previous research (Eudell-Simmons, Stein, DeFife, & Hilsenroth, 2005; Pinsker, Stein, & Hilsenroth, 2007; Slavin, Stein, Pinsker-Aspen, & Hilsenroth, 2007; Stein, Hilsenroth, PinskerAspen, & Primavera, 2009; Stein, Pinsker-Aspen, & Hilsenroth, 2007; Stein, Slavin-Mulford, Sinclair, Siefert, & Blais, 2012). Each rater independently rated all 544 TAT narratives employed in this study. It is important to note that although the “raw” TAT narratives were reviewed as part of the clinical assessment, assessing clinicians did not have access to the SCORS–G ratings. Also, the SCORS–G raters were blind to the clinical assessment results. Measures Thematic Apperception Test. The standard TAT protocol used at the clinic consists of Cards 1, 2, 3BM, 4, 13MF, 12M and 14, routinely given in this order. However the test battery is applied flexibly and some subjects were given fewer than seven cards (due to symptom acuity, fatigue, or both). All participants were administered Cards 1, 2, 3BM, and 4. Ninety-five percent of participants were administered Cards 13MF and 14. Card 12M was administered to 92.5% of participants. Clinicians in the clinic are trained to employ standard TAT instructions: Now I’m going to show you some cards with pictures on them. The pictures are of people in various situations and what I want you to do is make up a story around the picture. Like all stories, yours should have a beginning, middle, and ending. Tell me what led up to the picture, how it turns out, and what the people feel and think. Here’s the first one. Make up a story around this picture.

Social Cognition and Object Relations Scale–Global Rating Method. This scale consists of eight variables that are scored on a 7-point anchored scale where lower scores (e.g., 1, 2 or 3) indicate more pathological responses and higher scores

(e.g., 5, 6 or 7) indicate healthy responses. The first variable is complexity of representations of people (COM), which evaluates internal states and how well the patient is able to see internal states in the self and other when reporting narratives. It also assesses the patient’s relational boundaries and ability to integrate both positive and negative aspects of self and others. Affective quality of representations (AFF) examines a patient’s expectations of others within a relationship and the description of significant relationships in the past. It assesses emotional tone of the narrative. Emotional investment in relationships (EIR) assesses a patient’s ability for intimacy and emotional sharing. Emotional investment and values in moral standards (EIM) assesses the extent to which the patient uses abstract thought in relation to morality and compassion for others. Understanding of social causality (SC) assesses the extent to which the patient understands human behavior. Experience and management of aggressive impulses (AGG) assesses the patient’s ability to tolerate and manage aggression appropriately. Self-esteem (SE) assesses the patient’s self-concept, and identity and coherence of self (ICS) assesses a patient’s level of fragmentation and integration. More thorough descriptions of the eight SCORS variables, global rating method, and various training examples are provided in the manuals developed by Stein et al. (2011) as well as Westen (1995).

RESULTS SCORS–G Reliability Intraclass correlation coefficients (ICCs) were used to calculate SCORS–G ratings of TAT narratives. We calculated ICC (2) and Spearman–Brown corrected two-way random effects model (2, 1). Shrout and Fleiss (1979) reported the magnitude for interpreting ICC values in which poor is < .40, fair = .40–.59, good = .60–.74, and excellent >.74. As Table 3 shows, ICC (2) fell in the good to excellent range and ICC (2, 1) fell in the excellent ranges for all SCORS–G variables. The coefficient alpha for each SCORS–G variable was also calculated and these fell in the excellent range. Alpha differs from ICC (2, 1) simply by virtue of the fact that it is based on the Pearson correlation TABLE 3.—Interrater reliability of the Social Cognition and Object Relations Scale–Global Rating Method (SCORS–G) variables for Thematic Apperception Test (TAT) narratives. SCORS Variables COM AFF EIR EIM SC AGG SE ICS M

M

SD

ICC(2)

ICC(2,1)

Coefficient Alpha

3.35 3.30 2.99 3.62 3.17 3.53 3.74 4.43

.55 .49 .51 .38 .64 .43 .30 .41

.65 .79 .75 .73 .67 .83 .67 .68 .72

.79 .88 .86 .85 .80 .90 .80 .81 .84

.79 .88 .86 85 .80 .90 .80 .81 .84

Note. N = 80; 544 TAT narratives. ICC = intraclass correlation coefficient; (2) = Model 2, two-way random effect; (2,1) = Model 2, two raters, Spearman–Brown correction for two-way random effect; COM = complexity of representations; AFF = affective quality of representations; EIR = emotional investment in relationships; EIM = emotional investment in values and moral standards; SC = understanding of social causality; AGG = experience and management of aggressive impulses; SE = self-esteem; ICS = identity and coherence of self.

STIMULUS CHARACTERISTICS AND SCORS–G

5

TABLE 4.—Estimated variance components and percentage variance explained from two-facet random crossed models ( p × r × c). COM

Participant ( p) Rater (r) Card (c) Pr Pc Rc prc, e

AFF

df

MS

σ2

%

df

72 1 6 72 432 6 432

3.576 1.358 40.393 0.270 1.387 0.854 0.241

.154 .001 .267 .004 .573 .009 .241

12% 0% 21% 0% 46% 1% 19%

71 1 6 71 426 6 426

df

MS

σ2

%

df

MS

71 1 6 71 426 6 426

5.794 6.349 3.106 0.567 0.552 0.206 0.212

.349 .011 .018 .051 .170 0 .212

43% 1% 2% 6% 21% 0% 26%

70 1 6 70 420 6 420

2.669 0.122 13.765 0.220 1.108 0.119 0.127

Downloaded by [Michelle B. Stein] at 06:18 03 September 2013

SC

Participant ( p) Rater (r) Card (c) Pr Pc Rc prc, e

MS

EIR σ2

EIM

%

df

MS

σ2

%

df

36% 0% 4% 5% 23% 1% 30%

71 1 6 71 426 6 426

3.916 0.778 28.847 0.377 1.766 0.875 0.313

.149 0 .184 .009 .727 .008 .313

11% 0% 13% 1% 52% 1% 23%

71 1 6 71 426 6 426

σ2

%

df

MS

σ2

%

df

.105 0 .089 .013 .491 0 .127

13% 0% 11% 2% 59% 0% 15%

71 1 6 71 426 6 426

1.258 0.025 8.337 0.262 0.696 0.194 0.152

.033 .00 .053 .015 .272 .000 .152

6% 0% 10% 3% 52% 0% 29%

71 1 6 71 426 6 426

4.085 .241 2.446 .003 5.120 .029 0.407 .030 0.507 .154 0.613 .006 0.198 .198 AGG

SE

σ2

%

.069 0 .071 .013 .389 .002 .179

10% 0% 10% 2% 54% 0% 25%

MS

σ2

%

2.488 1.750 21.941 0.464 1.047 0.928 0.255

.088 .001 .140 .030 .396 .009 .255

10% 0% 15% 3% 43% 1% 28%

MS 2.020 0.254 11.333 0.274 0.956 0.356 0.179 ICS

Note. MS = mean squares; σ 2 = estimated variance component; % = percentage of variance accounted for; COM = complexity of representations; AFF = affective quality of representations; EIR = emotional investment in relationships; EIM = emotional investment in values and moral standards; SC = understanding of social causality; AGG = experience and management of aggressive impulses; SE = self-esteem; ICS = identity and coherence of self.

rather than the ICC. The SCORS–G ratings of TAT narratives were highly reliable and showed strong internal consistency. To examine potential sources of variance, we estimated variance components (σ 2) with techniques from generalizability theory (Crick & Brennan, 1982; Shavelson & Webb, 1991). Given that all participants were rated by all raters, and that raters were not viewed as systematically different from the universe of raters trained to rate the SCORS–G, variance component estimates were calculated using a random two-facet model in which participants ( p) were crossed with raters (r) crossed with cards (c). This approach estimates variance components for the universe level (i.e., the participant level), as well as for rater and for card. Estimated variance components are also calculated for the interactions between participant and card (σ 2pc ), participant and 2 rater (σ pr ), and rater and card (rc). Finally, an estimated variance component is calculated for the residual (σ 2prc,e ). As can be seen in Table 4, raters were highly consistent with percentage variance attributable to raters being 1% or less across SCORS–G dimensions. Raters were also consistent across cards and participants. The interaction between raters and card accounted for very little variance (all σ 2rc accounting for 1% or less of total variance), and the interaction between rater and participant 2 was also relatively small (σ pr ranging from 0% to 6% across SCORS–G dimensions). Although raters tended to rate consistently across both cards and participants, estimated variance components for that card level and Card × Participant interaction were consistently larger. For COM, EIR, and SE, variance component estimates were larger for card than for the participant level (i.e., universe level), indicating that participants’ narratives differed somewhat from card to card. Further, the percentage of variance attributable to the Card × Participant interaction considerably exceeded the amount accounted for at the participant level for all SCORS–G dimensions except AFF and SC. This suggests that participants’ relative standing on the SCORS–G dimensions, with the exception of AFF and SC dimensions, varied considerably as a

function of card. Said differently, a participant who told a story high in COM to Card 1 might not have necessarily told a high COM story to other cards. In general, these findings suggest that individual reactions to the cards introduce the most variability for several of the SCORS–G dimensions. However, these results in and of themselves do not indicate that respective cards pull for particular types of stories. To determine if cards pull for particular types of narratives, the mean response level for a dimension on each respective card should be compared to the overall mean for that dimension (see Table 5).

SCORS–G Variable Means The means and standard deviations based on raw scores for the eight SCORS–G dimensions across all TAT cards are presented in Table 5. Mean word count and standard deviations for individual cards are presented in Table 6. Stimulus Characteristics of Individual TAT Cards Raw SCORS–G dimension scores are presented in the uppermost portion of Table 5. To assess for individual card characteristics, each participant’s response to each card was then transformed into z scores (M = 0, SD = 1). The advantage of z scores is that one can easily see if a card is consistently scoring above the mean (i.e., a positive z score) or below the mean (i.e., a negative z score) and can easily determine the magnitude of the difference from the mean. There are two ways of calculating z scores (across and within participant z scores) and both are presented next. Significant results on either approach are suggestive of card “pull.” Across participant z scores. We calculated across participant z scores by determining the mean and standard deviation for each SCORS–G dimension across each participant’s response for all cards. For example, for the COM scale, the mean and standard deviation for the sample was calculated using each participant’s COM score for each individual card. Thus, each of

STEIN ET AL.

6

Downloaded by [Michelle B. Stein] at 06:18 03 September 2013

TABLE 5.—Average raw and standardized SCORS–G dimension scores by card.

Card

COM

AFF

1 2 3BM 4 13MF 12M 14

3.62 3.55 3.14 3.42 3.21 3.35 3.18

3.52 3.72 2.73 3.35 2.47 3.32 3.89

Raw Scores Averages Across Participants EIR EIM SC

AGG

2.90 3.64 3.41 3.62 3.35 3.87 3.32 3.79 2.54 3.58 3.10 3.46 3.23 3.54 3.27 3.37 2.75 3.06 3.06 2.96 3.66 3.66 3.29 3.56 2.51 3.85 3.04 3.82 Average z Scores (Across Participants) by SCORS–G Dimension

SE

ICS

3.90 3.99 3.42 3.75 3.44 3.81 3.89

4.69 4.88 3.84 4.54 4.05 4.47 4.57

Card

COM

AFF

EIR

EIM

SC

AGG

SE

ICS

M

SD

1 2 3BM 4 13MF 12M 14 F p ŋ2

0.41 0.28 −0.23 0.10 −0.19 0.03a −0.23 9.62