JOURNAL OF SPORT PSYCHOLOGY,1985,7,65-74

Methodological Considerations in Youth Sport Motivation Research: A Comparison of Open-Ended and Paired Comparison Approaches Leonard M. Wankel and Philip S.J. Kreisel The University of Alberta A review of open-ended and structured approaches to assessing youth sport motivation reveals that each has certain strengths and limitations. Hence, the method of choice for a particular study will depend upon the particular problem addressed. In order to examine the comparability of results obtained from different methodologies, a study was conducted to compare the results pertaining to factors underlying sport enjoyment obtained from open-ended and Thurstonian paired comparison inventories. Although a number of similarities appeared in the results across the two methodologies (e.g., personal accomplishment, excitement of the sport, and just doing the skills were important, while getting rewards and pleasing others were relatively unimportant), there were also some differences reflecting the particular methodologies. A consideration of previous research employing a broader variety of methodological approaches (openended, ranking, Likert scaling, Thwstone paired comparisons) further indicates that although the particular methodology utilized does affect the results obtained, certain important motivational factors tend to surface regardless of the methodology employed.

In recent years, there has been a marked increase in interest in studying factors related to involvement in youth sports. Most of that research has been conducted within the last 5 years (Gould & Horn, 1984). Further, Gould (1983) reports that a survey of sport practitioners and sport psychologists involved with youth sports research indicated that "determining why young athletes participate in youth sports" and "understanding why young athletes stop participating" were ranked as two of the most important psychological issues requiring further study. The general recognition of the importance of the motivation area, together with the demonstrated interest of more researchers to get actively involved in this area of inquiry, bodes well for the future development of this aspect of sport psychology. For significant progress to occur, however, due consideration This research was financially supported by Fimess and Amateur Sport CanadaGrant 256-002-1. A previous report on this research was presented at the Canadian Society for Psycho-Motor Learning and Sport Psychology Conference, Edmonton, Alberta, October 1982 and is published in the proceedings: Wankel, L.M., & Wilberg, R.B. (Eds.) (1983) Psychology of Sport and Motor Behavior: Research and Pracrice. Edmonton, the University of Alberta, Faculty of Physical Education and Recreation. Requests for reprints should be sent to the first author at the Dept. of Recreation and Leisure Studies, University of Alberta, Edmonton, Mta., Canada T6G 2H9. 65

66

WANKEL AND KREISEL

must be given to important methodological considerations. One of the most crucial questions facing the researcher who embarks on research pertaining to youth sport motivation is, what type of instruments should be employed in the research? There is no easy answer to this question as each approach has its own advantages and disadvantages. Therefore, the specific requirements of the intended study must be considered together with the strengths and weaknesses of the different approaches in order to select the most appropriate technique. In this paper, consideration is given to two approaches that might be employed in investigating youth sport motivation: an open-ended approach and a paired comparison approach. After reviewing the major advantages and disadvantages of each, a study will be presented to compare the results obtained from the two techniques. This paper is delimited to a consideration of factors that have a positive influence on youth sport participation. Although reasons for dropping out of sport or aspects disliked about the sport experience are also an integral part of the youth sport participation motivation literature (e.g., Gould & Horn, 1984; Passer, 1982), they are beyond the scope of this paper. Further, within the positive factors facilitating involvement, primary emphasis will be placed on those aspects related to fun or enjoyment. This reflects a basic assumption that minor sport programs are recreational programs that participants engage in voluntarily because of the enjoyment or satisfaction they provide.

Approaches The Open-Ended Approach This approach entails asking respondents to state their opinions in response to specific stimulus questions. This may take the form of asking subjects, "What do you enjoy most about participating in sport?" or "What do you enjoy about participating in sport?" While the first question produces data that are more readily compared across groups and studies, since all respondents simply state their most enjoyed feature it leaves many important areas of enjoyment unexplored. The latter approach presents a problem in that all responses appear together and no differentiation is given to the relative importance of various sources of enjoyment. A composite approach asks for the most important factor and subsequently for other aspects that are enjoyed. This distinguishes between most enjoyed aspects and other less enjoyed aspects but does not scale the importance of these other responses. The major advantage of using an open-ended questioning approach is that it gives the respondents the freedom to answer the questions in their own way. No prepared set of responses are provided that might limit or influence the subjects' responses. This flexibility of the open-ended questions can lead to a completeness of information which may be lacking in more structured approaches. In reference to this point Oppenheim (1966) states, "We obtain his ideas in his own language, expressed spontaneously, and this spontaneity is often extremely worthwhile as basis for new hypothesis" (p. 41). This freedom and spontaneity, however, does not come without a cost. Frequently, open-ended questions on a questionnaire generate general and rather vague stereotyped responses. As Iso-Ahola (1980) points out, these stereotypical culturally acceptable explanations for behavior may conceal more important underlying reasons for an individual's actions. In the sport motivation research, examples of such culturally acceptable explanations which are commonly given for involvement are "because it's fun,"

YOUTH SPORT MOTIVATION RESEARCH

67

and "to get in shape." While these undoubtedly are important general categories, they have limited interpretive value without subsequent investigation that can break down the meanings underlying these general categories. If specific items are reported in response to an open-ended question, a different kind of problem is encountered. The range of specific responses can be virtually limitless, and in order to interpret these responses some classification or categorization must be carried out. This is seldom a straightforward process. Vague and ambiguous responses are difficult to categorize objectively. Thus, there is often a lack of comparability of results from one study to another when different levels of reduction have been used to categorize the responses. This problem is apparent when reviewing the youth sport motivation research. Whereas Wankel (1982) utilized over a dozen categories to classify freely given reasons for sport involvement in his study, Robertson (1981) and Fry, McClements, and Sefton (1981) in their studies collapsed the specific responses into four quite generalized categories. Further, these general categories were quite different in the two studies. In summary, the open-ended approach is well suited to exploring the parameters of a new area of study. It has major limitations, however, when it comes to comparing the results from one group to another or especially when comparing results from one study to another.

The Paired Comparison Approach The paired comparison approach to data collection is a type of closed question format. As with all closed techniques, the paired comparison approach, by requiring individuals to respond to a set of fixed items, ensures that the results of different individuals and/or groups can be compared and that all respondents have considered the same total content before responding (Oppenheim, 1966). However, this presupposes that the respondents understand the items and share the same general meaning for them. Hence, it is important that preliminary research establish the appropriateness of the wording for the participants in the study and that the built-in closed items reflect important aspects of the area to the respondents as determined through prior openended exploratory research. A major advantage of the paired comparison technique over other closed techniques, such as ranking and Likert scaling procedures, is the precision of measurement that it provides. It yields an interval level of measurement, whereas the other techniques are only at the ordinal level.' The paired comparison technique, by requiring respondents to choose between items presented two at a time, provides a distribution of scores that, when converted to standardized z-scores, allows a discrete scaling of the importance of various items to a particular group. Further, between-group comparisons can be made in terms of the size of intends between the same items for different groups. As there is no absolute zero point on these scales, however, individual scores cannot be directly compared across two different groups. A second advantage of the paired comparison method is that because it only requires choosing between two items, it can effectively be used with young subjects. A disadvantage of this approach is the excessive time required to administer it. As the number

'Data from Lirt-type scales are frequently analyzed as if they are at an interval level of measurement, although they are only at an ordinal level as they do not provide a true neutral point on a scale between two extremevalues (e.g., strongly disagree-strongly agree) (Oppenheim, 1966).

68

WANKEL AND KREISEL

of separate items to be compared (N) increases, the number of comparison pairs (X) in). Therefore, this technique is not practical when there creases markedly (X = are many separate aspects to be investigated within the particular area of study. Under these conditions, a modified approach may be used wherein the list of items is divided into two separate lists with some overlapping items and the two lists are administered to two separate samples. The paired comparison approach is most valuable when there is a desire to scale a limited number of items within a domain, however, and to compare the relative importance of the items for different groups.

Nq

Purpose of the Study Because different methodologies have distinct advantages and disadvantages, a question arises as to how the particular methodology used influences the results of a given study. Or more generally, to what extent do the differences in results reported from different studies merely reflect methodological differences? This study was designed to provide some insight into these issues by investigating whether an open-ended question approach would yield similar results to a structured paired comparison approach to assessing enjoyment in children's sport.

Methods Subjects The subjects for the study were boys between the ages of 7 and 14 who participated in an organized baseball, soccer, or hockey program during the 1981-82 s e a ~ o n The .~ sample was drawn by randomly selecting 20 schools from the population of elementary and junior high schools in the public school system of a large Canadian city. A representative sample of participants from the three sports was selected from the designated schools. In some schools a true random sample of all the participants was drawn by the researchers, but in others this was not possible because the principal and teachers selected certain classes for involvement to minimize the disruption of regular school activities. However, in no case did the selection procedures introduce any known source of bias into the sample. The sample selected included 310 soccer participants, 338 hockey participants, and 176baseball participants. The vast majority of the boys participated in community-sponsored programs, with only a small minority involved in organized school-sponsored sport programs.

The Instrument The enjoyment items that generated the data reported in this paper were part of a larger questionnaire designed to investigate various factors related to participation in

Separate analysis were not performed for different age and sport groups in the current study because of a small number of respondents in certain categories. Previous analyses of the pairedcomparison data, however, indicated that there was considerable stability across both age level and different sports in the relative importance of the different enjoyment factors.

YOUTH SPORT MOTIVATION RESEARCH

69

youth sports. The introductory open-ended question section of the questionnaire included two questions pertaining to enjoyment: "What do you like best about playing (name of sport)?" "Is there anything else that you like about (name of sport)? If so, explain." The second section of the questionnaire was the Minor Sport Enjoyment Inventory (MSEI), a 10-item inventory for investigating participants' sport enjoyment. The 10 items comprising this inventory were: "doing the skills of the game," "comparing one's skills against others," "achieving a sense of personal accomplishment," "the excitement of the game," "improving the skills of the game," "winning the game," "being with friends," "being on a team,'' "pleasing others," and "receiving rewards. " The 10 items were incorporated into a Thurstonian paired comparison inventory which required that a respondent select the preferred enjoyment factor from each of 45 possible pairs. The original 10 items for the structured inventory came from two major sources: a review of extant literature on youth sport motivation and the general motivation literature and open-ended interviewing of 50 youth sport participants. A panel of five researchers then reviewed the generated list of specific items and formulated from it a list of more general and more distinct items. Further interviews were then conducted with youth sport participants to test the clarity and inclusiveness of those items and to select the 10 most important items (i.e., those mentioned most frequently by the sport participants). Pilot testing was then camed out to assess the suitability of the wording for the targeted age groups. On the basis of a preliminary study (Wankel & Pabich, 1982), one item from the original inventory (have a uniform and good equipment) was deleted and another (improve skills and feel a sense of accomplishment) was broken into two items (improve ability at the game; feel good when have played well) for the final inventory.

Data Collection and Analysis The data were collected from October through December 1981. All data collection took place in a school classroom under the supervision of a trained researcher during regular school hours, and the results of the questionnaires were then coded and prepared for computer analyses. The two researchers independently categorized the separate codes for the open-ended enjoyment questions into 10 comparable categories to the 10 structured inventory items. The separate categorizations were then compared and synthesized to provide a coding key for the responses to the open-ended questions, which would allow direct comparison of the open-ended question responses to the paired comparison inventory results. The paired comparison responses were analyzed using the University of Alberta Division of Educational Research Services (DERS) program SCALQ1. This program produced proportional matrices, z-value matrices, and scale values (standardized scores) for the 10 items for each separate sport group (Edwards, 1957). The scale values were then ranked in ascending order of importance. The ranks and standard scores from the structured inventory were then compared to the ranks and frequency responses for the openended responses.

Results and Discussion The results for the open-ended and structured inventory questions pertaining to enjoyment are presented in Table 1. For the open-ended questions, the percentage of overall responses falliig within each of the 10 categories is presented for (1) most liked, (2) also

-

Table 1 A Comparison of Open-Ended and Structured Question Results Pertaining to Sport Enjoyment

a Like most

Enjoyment factor

oh

Open-ended questions b c Combined Also like (a and b)

Rank

%

Rank

oh

Rank

Soccer (N = 310) Scale value

Rank

Structured scale Hockey Baseball (N = 338) (N = 174) Scale value

Rank

Scale value

Rank

Combined (N = 822) Rank

Comparing skills against others Excitement of the game Personal accomplishment Improving one's skills Doing the skills Being on a team Being with friends Winning the game Getting rewards Pleasing others Fun Exercise Number of responses

758

596

1354

Note: As respondents could indicate more than one enjoyment factor in response to the open-ended questions, the percentage data refers to the percentage of total responses to a question that falls within a given category.

YOUTH SPORT MOTIVATION RESEARCH

71

liked, and (3) either most liked or also liked. The percentage responses for the two items "fun" and "exercise" are also included in the table, although they were not included in the ranking as no comparable items were included in the structured inventory. Fun was not included as an item in the structured inventory as the inventory was designed to investigate factors underlying fun. Exercise was not included as preliminary research indicated that although exercise and fitness were important reasons for participating in sport and physical activity, they were not identified as being one of the most important factors for fun or enjoyment, and hence were not given priority for inclusion in the structured inventory. With the exception of fun and exercise, the responses to the open-ended questions were quite comprehensively covered by the 10 structured categories. This indicates that the review of literature and preliminary interviewing which led to the selection of the 10 items for the structured inventory were quite successful in identifying important facets affecting sport enjoyment. Examination of the three sets of ranks for the open-ended data (Table 1) indicates that the results were very consistent across the three separate approaches. The one exception to this was beiig with friends, which was frequently indicated to be an also liked aspect of sport but was not frequently indicated to be the most liked factor. As a result of this general consistency in the results of the different types of open-ended questions, only the combined question data for the open-ended responses were used in making comparisons to the structured questionnaire data. In making these between-methodology comparisons, both similarities and differences are evident. In the structured inventory five items emerged as being of considerable importance to enjoyment. In the open-ended data three of those same factors, doing the skills, personal accomplishment, and excitement of the game, gained considerable support. The other two items, comparing skills and improving skills, received considerably less support. Comparing skills ranked 6.5 out of 10, while improving skills was of little importance and ranked ninth out of 10 factors. Conversely, the items being with friends and winning were given greater relative importance on the open-ended question responses than they were on the structured inventory. Being on a team was of moderate importance regardless of the methodology, whereas getting rewards and pleasing others ranked low in importance as a reason for enjoying sport regardless of whether an open-ended or structured methodology was used. Although the reasons for the differences are not readily apparent, some plausible interpretations are offered. First, with respect to the lack of importance of improving one's skills at the game, it would appear that this factor was viewed as being closely related to personal accomplishment and was not readily distinguished from it by respondents in their volunteered reasons. Similarly, it would appear that respondents didn't clearly differentiate between excitement of the game and just doing the skills of the sport, and the judges categorized most of the volunteered reasons as enjoyment due to doing the skills, while excitement of the sport received little emphasis. The greater relative emphasis on being with friends and winning the game in the open-ended responses than in the structured inventory data may reflect a prevalence of these reasons in everyday language. As Iso-Ahola (1980, p. 247) indicates, an individual's freely given reasons for doing something are influenced by commonly reported reasons for people doing similar activities. It would appear that a structured question approach would help to overcome these limited response sets. Thus, a paired comparison approach might give a more complete indication of an individual's real reason: for enjoying an activity.

72

WANKEL AND KREISEL

Alternatively, it might be argued that the paired-choice format is more susceptible to social desirability as it presents two items in pairs which may lead to the most so&illy desirable response being overrepresented. Although the reduced emphasis on winning in the structured results is consistent with this interpretation (i.e., winning in youth sports is generally not considered a priority, at least in thwry), there are strong arguments against it. The method of presenting the paired choices followed the approach o f ~ a r t e (lb79), r which is specifically designed to legitimize both choices and to reduce social desirability effects. Second, being with friends, a highly socially accepted response, received less support on the structured inventory-contradictory to what would be expected if the structured format was more susceptible to social desirability effects. Although little previous research has specifically addressed the issue of what participants enjoy about sport, a number of studies have addressed reasons for participating in sport. This broader approach includes instrumental factors (e.g., to get fit) beyond immediate enjoyment. Although the use of different measuring instruments and different categories in the various studies precludes any precise comparison of these results, it is informative to consider general patterns in the reported results. Improvement of skills has consistently been found to be an important reason for involvement in youth sports. This has been true in studies using an open-ended format (Fry et al., 1981), a ranking procedure (Watson, 1975), Likert-type inventories (Gill, Gross, & Huddleston, 1983; Gould, Feltz, Weiss, & Petlichkoff, 1982; Sapp & Haubenstricker, 1978), and the paired wmparison approach in the current study. Excitement and personal accomplishment, items found to be of high importance on the paired comparison inventory and of moderate importance on the current open-ended questions, have consistently been reported to be of moderate or high importance in studies of youth sport motivation regardless of the particular methods employed. In studies of young Australian sport participants, Robertson (1981), using an open-ended approach, and Watson (1975), using a ranking procedure, both found personal accomplishment to be moderately important. Similarly, studies utilizing a Likert approach have indicated that the desire to do something one was good at was an important reason for participating (Gill et al., 1983; Gould et al., 1982). Excitement of the sport, which scored high on the structured data and moderately high on the open-ended data in this study, has been reported to be of moderately high importance in studies employing a ranking procedure (Watson, 1975) and a Likert scaling approach (Gill et al., 1983; Gould et al., 1982; Sapp & Haubenstricker, 1978). Similarly, doing the skills, which was found to be highly important to enjoyment in the open-ended aspect of this study and moderately important in the paired comparison aspect, has been consistently found to be an important reason for youth sport participation. This has been the case for both open-ended (Fry et al., 1981;Robertson, 1981) and ranking approaches (Watson, 1975). Pleasing others and getting rewards have consistently swred low across openended approaches (Robertson, 1981), ranking (Watson, 1975), and Likert approaches (Gill et al., 1983; Gould et al., 1982). Winning the game, which scored low on both measures in this study, has been reported to be either low (Fry et al., 1981; Gould et al., 1982; Watson, 1975) or moderate in importance (Gill et al., 1983; Robertson, 1981; Wood, 1981). Being on a team was of moderate importance to enjoyment in the structured aspect of this study. Similarly, it was found to be a moderately important reason for participating in youth sports in the Likert studies of Gill et al. (1983) and Gould et al. (1982). The remaining two factors examined in this study, testing ability against other players and being with friends, have yielded less consistent results across different

YOUTH SPORT MOTIVATION RESEARCH

73

methodologies. Testing ability against other players was of high importance in the structured aspect of the current study, while a related item, to compete, also scored high in the Gill et al. (1983) study and moderately high in the Gould et al. (1982) study. On the other hand, the open-ended question results of the current study, as well as those previously reported by Fry et al. (1981), indicate competition to be of relatively little importance. Similarly, Watson (1975), using a ranking procedure, found competition to be scored low as a reason for participating in youth sports. The social factor being with friends has varied considerably across studies and in no consistent manner with the measurement approach utilized. In conclusion, it would appear that the particular methodology employed does have some effect on the results obtained in a given study. Hence, caution must be exercised in interpreting the results from any one investigation. On the other hand, a number of consistencies or general trends across measurement approaches do substantiate the interpretation that there are meaningful factors underlying the enjoyment of sport. In terms of the particular methodologies employed in this study, the paired comparison approach has a real advantage in scaling the relative importance of different factors. It goes beyond just ranking the importance of factors to give specific information concerning the magnitude of the importance of the various factors.

References Edwards, A.L. (1957). Techniques of a#itude scale construction. New York: Appleton-Century-Crofts. Fry, A.P., McClements, D., & Sefton, P. (1981, September). A report on participation in the Saskatchewan Hockey Association. Report presented to Sask. Sport. Gi,D.L., Gross, J.B., & Huddleston, S. (1983). Participation motivation in youth sports. Znternational Journal of Sport Psychology, 14, 1-14. Gould, D. (1983). Future directions in youth sports participation motivation research. In L. Wankel & R. Wiberg (Eds.), Psychology of sport and motor behavior: Research and practice (pp. 1-18). Edmonton: University of Alberta, Faculty of Physical Education and Recreation. Gould, D., Feltz, D., Weiss, M., & Petlichkoff, L. (1982). Participation motives in competitive youth swimmers. In T. Orlick, J.T. Partington, & J.H. Salmela (Eds.), Mental training for coaches and athletes @p. 57-59). Ottawa: Coaching Association of Canada. Gould, D., & Horn, T.S. (1984). Participation motivation in young athletes. In J.M. Silva & R.S. Weinberg (Eds.), Psychological foundations of sport @p. 359-370). Champaign, IL: Human Kinetics. Harter, S.P. (1979). Perceived competence scalefor children (Manual: Fonn 0).Denver: University of Denver. JmAhola, S.E. (1980). lhe socialpsychologyof leisure rmd recreation. Dubuque, IA: Wm. C. Brown. Oppenheim, A.N. (1966). Questionnaire design and attitude measurement. London: Henemann. Passer, M. (1982). Children in sport: Participation motives and psychological stress. Quest, 33, 231-344. Robertson, I. (1981). Children's perceived sarisfa&'on and stresses in sport. Paper presented at ACHPER 13th National Biennial Conference, Australia. Sapp, M., & Haubenstricker, J. (1978, April). Motivationfor joining and reasonsfor not continuing in youth programs in Michigan. Paper presented at the AAHPER National Conference, Kansas City. Wankel, L.M. (1982). Factors a$ecting sport participation: Enjoyment and dissati$action in minor sport. Report submitted to Fitness and Amateur Sport Canada (Project No. 265-003-02). Ottawa, Ontario. Wankel, L.M., & Kreisel, P.S.J. (1983). Factors underlying enjoyment and lack of enjoyment of minor sport: Sport and age group comparisons. In L.M. Wankel & R.B. Wilbert (Eds),

-

74

WANKEL AND KREISEL

Psychology ofsport and motor behavior: Research d p m c t i c e @p. 1943). Edmonton: University of Alberta. Wankel, L.M., & Pabich, P. (1982). The minor sport experience: Factors contributing to or detracting from enjoyment. In T. Orlick, J. Partington, & J. Salmela @ds.), Mental training for coaches and athletes @p. 70-71). Ottawa: Coaching Association of Canada. Watson, G.G. (1975). i%e meaning ofparental influences and intrinsic reward in children's sport: i%e case of little athletics. Paper presented at Conference on Sport, Society and Personality. La Trobe University, Bundoora, Victoria. Wood, N.L. (1981). Incentive motivation in sport: A theoretical analysis and the development of a measuring instrument. Unpublished doctoral dissertation, University of Alberta. Manuscript submitted, April 18, 1984 Revision received: September 5 , 1984