The Contributions of Mixing Methods to Survey Research Danish Survey Research Society February 2015, Copenhagen Jennifer C. Greene

Overview •  A conversation •  Introductions •  What is survey research? –  Strengths, challenges, limitations?

•  Mixing methods – key concepts •  Mixed methods survey research, examples •  Ongoing discussion

Introductions •  JCG •  Survey researchers

What is survey research? The relatively accurate assessment (or estimate) of the frequency, incidence, magnitude, and dispersion of selfreported characteristics, attitudes, behaviors, and other human practices among a designated population or set of populations. •  Additions, corrections?

What does survey research do well? •  Character of survey research –  “Proven” methodology –  Strong potential for representative samples –  Ability to develop good items and psychometrically strong scales for measuring higher-order constructs ! highly reliable inferences with some degree of validity

•  Character of survey research (continued) –  Ability to inter-relate demographics with attitudes and behaviors, with some complexity and nuance (multi-level modeling) –  Ever-increasing methodological sophistication (e.g., HLM, data analytics)

•  What else?

•  Contributions of survey research –  Defensible inferences related to multiple constructs –  Detailed snapshots of groups of people –  Complex patterns of association among various clusters of variables –  Relational patterns across discipline and fields of study

•  What else?

What are key challenges of survey research? •  Defer to the experts

What are key limitations of survey research? •  Limited conceptualization and definition of especially complex constructs •  Insufficient attention to context •  View the complexity of the social world from just one angle, just one perspective •  Correlational not explanatory •  What else?

What can mixed methods thinking contribute to survey research? •  •  •  • 

MM definition MM purposes MM designs Examples

“Definition” of MM •  The intentional, and connected or linked, use of more than one social science tradition, methodology, and/or method in service of better understanding –  Tradition = philosophical paradigms and assumptions, logics of justification, privileged questions, ways of knowing •  Examples: postpositivism, interpretivism, constructivism, feminisms, critical social science

–  Methodology = inquiry logic, including questions, design, sampling, method choice, analysis, quality criteria, and defensible forms of writing •  Examples: experimentation, survey research, ethnography, case study, narrative inquiry

–  Method = a technique or tool for data gathering •  Examples: Ask ~ questionnaire, interview, assessment Watch ~ observation Find traces ~ unobtrusive measures

–  Can also mix theories or conceptual lenses

•  The intentional, and connected or linked, use of more than one social science tradition, methodology, and/or method in service of better understanding •  A study is a mixed methods study when there is some connection or linkage among the various methods and data sets at one or more stages of inquiry. •  Statement of vision

Purposes for Mixing, various forms of “better understanding” •  Example: Study of the character and (in)stability of the educational pathways of K-12 children when their families/ parents are homeless ! Toward revised educational policies for children in temporarily-homeless families

• 

Triangulation is the use of different methods to generate findings that (hopefully) converge in their assessment of the same phenomenon, toward increased validity and defensibility of inquiry inferences.

• 

EX: Mix methods to better understand phenomenon of school attendance patterns. Mix analysis of school attendance records with family (parent and child) interviews

•  Complementarity is the use of different methods to assess overlapping phenomena or multiple facets of the same phenomenon, whereby the results from one method are used to enhance, augment, clarify the results of the other, toward more comprehensive understanding. •  EX: Mix methodologies to better understand the role or place of schooling amidst a homelessness crisis. Mix surveys of parents, children, and homeless service staff with mini-case studies of selected families

•  Development is the sequential use of different methods to assess the same phenomenon, where the results of the first method are used to inform the development of the second. •  EX: To address questions regarding how schooling patterns differ for families that are homeless for different reasons, use results of surveys or case records to purposefully select diverse sample of families for further interviews. (Could be a mix of methods or methodologies.)

•  Expansion is the use of different methods to assess different phenomena in order to expand the breadth and scope of a study, again toward more comprehensive understanding. •  EX: An analysis of school data on student academic and behavioral performance is added to this otherwise-experiential study of the place of schooling within the phenomenon of homelessness (method mix)

•  Initiation is the intentional use of different methods to seek paradox or fresh conceptual insight for a given phenomenon or a set of phenomena, or serves as a placeholder for the legitimization of dissonance in results.

(Initiation, continued) •  EX: Mix ‘paradigms’ (and methodologies and methods) to possibly invoke fresh insights into the educative dimensions of the homeless experience itself for children and youth. Mix constructivist narrative elicitation of stories with a structured assessment of possible knowledge and skills gained during the homeless episode.

MM purposes ! MM design Key design dimensions (so far) 1. 

Assess the same or different phenomena with the different methods 2.  Methods of equal status, or one dominant and the other less-dominant 3.  Sequential or concurrent implementation 4.  Mix at the end or mix throughout the study •  Component designs •  Integrated designs

Design

What phenomena?

Status of methods?

Sequence of implem’tion?

Linking task?

COMPONENT DESIGNS * Triangulation

Same

Equal

Concurrent

Comparing results

* Expansion

Different

Variable

Variable

Connecting inferences

INTEGRATED DESIGNS * Development

Same

Preferably equal

Sequential

Representing data in new form

* Complementarity, Initiation

Same

Preferably equal

Concurrent

Jointly analyzing data and generating integrated inferences

* Complementarity, Initiation

Same

One method primary

Concurrent

Jointly analyzing data and generating integrated inferences

•  Comments, questions, challenges??

Example 1 Ungar, M., & Liebenberg, L. (2011). Assessing resilience across cultures using mixed methods: Construction of the Child and Youth Resilience Measure. Journal of Mixed Methods Research, 5(2), 126-149.

Study purpose •  Focus – develop a structured, crossculturally respectful measure of resilience in children and youth •  Resilience viewed as not only an individual characteristic, but also as qualities of the environment •  Aim – use measure to study internal and external assets most influential in positive developmental pathways

Study design •  Mixed methods approach and rationale –  Legitimize multiple standpoints –  Honor tensions between homogeneity and heterogeneity, between context and generality, between standardization and localization –  Iterative, collaborative, involving international team and local researchers

Study implementation and selected results •  Initial development of resilience measure included: –  Research team ideas (and theories) –  Focus groups with youth and adults in each participating country –  Expert view ! local views ! integrated view

•  Pilot testing of structured instrument combined with interviews with youth = conversation about what it means to be “doing well despite adversity” •  Analysis of pilot and interview data –  Bronfenbrenner’s framework not useful –  Developed 7 aspects of resilience from interviews ! scales in quantitative measure

•  Analyzed scales further ! four portraits of resilience, reflecting four groupings of youth –  Minority world boys and girls –  Majority world girls –  Majority world boys in contexts with high social cohesion –  Majority world boys in contexts with low social cohesion

•  Finalized instrument

Example 2 Jang, E.E., McDougall, D.E., Pollon, D., Herbert, M., & Russell, P. (2008). Integrative mixed methods data analytic strategies in research on school success in challenging circumstances. Journal of Mixed Methods Research, 2(3), 221-247.

•  Focus – developing a multidimensional understanding of school success for schools serving low-income immigrant families and children (Ontario, Canada) •  Concurrent MM design, n=20 schools (purposefully selected) –  Interviews and surveys with teachers, principals –  Focus groups with students and parents –  Purpose of complementarity

•  Descriptive and reductive analyses –  Interview data analyzed ! 11 themes associated with school improvement –  Survey data analyzed ! 9 factors associated with school improvement

•  Integrative analyses –  Transformed survey results to narrative form –  Compared to qualitative themes

•  Next, created new survey ‘scales’ from interview themes –  Of 75 survey items, 63 were judged to relate to the interview themes –  3 interview themes not present in survey items –  Assigned the 63 items to one of the remaining 8 themes –  That is, used the structure of meaning in the interview data to ‘rescale’ and then reanalyze the survey data

•  New ‘blended’ scales showed more variation than original survey factors

•  Further analyzed blended themes •  For each theme, identified schools with a mean score significantly different from overall mean (extreme cases) •  Returned to qualitative data to provide contextually rich narrative of the nature and contours of the theme at the selected schools •  Wrote narrative case profiles by theme ! understanding the contextual meanings of ‘high’ and ‘low’ for that theme

•  Example for ‘parent involvement’ theme –  ‘High’ school •  Community with 25 different languages •  School active in multiple parent programs, some in partnership with local service agencies •  One teacher serves as community liaison with parents and families •  Principal walks around community getting to know families •  Principal personally visits parents of children placed ‘at risk’ and generates with them a ‘game plan for their child •  Parents perceived school as welcoming and ‘on their side’

–  ‘Low’ school •  Similar demographics as ‘high’ school, more central city •  Same recent history of academic success •  Recently, principal and teachers have concentrated on school safety •  Principals recognizes importance of strong parental involvement; school needs to turn energies to this domain

Example 3 Kling, J.R., Liebman, J.B., & Katz, L.F. (2005). Bullets don’t got no name: Consequences of fear in the ghetto. In T.S. Weisner (ed.), Discovering successful pathways in children’s development (pp. 243-282). Chicago: University of Chicago Press.

•  “Moving to Opportunity” housing demonstration project, experimental study, US 1990s •  Key question: How does a substantial change in neighborhood affect low-income families?

•  Original design –  Baseline survey –  Two-year survey –  Administrative data analysis –  Focus on: job training and job earnings, children’s school progress

•  Addition of a “qualitative” component –  To better understand program “uptake”, gangs and crime, and to listen well to issues not yet on the table –  Conducted observations of program contexts and activities, consulted program counselors, interviewed program participants –  This account – 12 interviews with participants (both experimental and control groups)

Four contributions to study Interview data: 1.  Informed survey 2.  Generated conceptual framework re mechanisms of change 3.  Provided “deep institutional understanding” of MTO on the ground 4.  Contributed to lessons for housing policy

And in MM language … What MM purposes invoked? •  Development •  Complementarity •  Initiation

•  Comments, critiques, conversations?

… Thank you