Minimizing Confirmatory Bias in Custody Evaluations Daniel J. Rybicki, Psy.D., DABPS Dennis J. McGlothin, Esq. G. Andrew H. Benjamin, J.D. Ph.D. ABPP

Fourth Annual WA AFCC Conference Frontiers of Family Law March 15, 2014 Seattle, WA

1

Overview and Road Map Moving Toward a Uniform Procedure that Minimizes Confirmatory Bias Introduction and Reasons for Workshop Definitions and Scope of Problems

McGlothin

Rybicki

WAC, RCW and Statutes, WAC Statutes APA Professional Ethics and Safeguards Benjamin AFCC Model Standards as Professional Guidelines Rybicki Timeline of Confirmatory Bias: Beginning, Mid-Way, Report Stage and Afterwards Panel Discussion, Solutions and Suggestions Question and Answer Segment

2

Psychological Terms and Key Concepts Defined Definitions and Scope of the Problem Daniel Rybicki, Psy.D.

www.danielrybicki.com [email protected] 3

Caution Regarding Impressions and Perception

4

I’ll believe it when I see it

I’ll see it when I believe it

5

Faulty Data – “Tain’t Necessarily So”

Computer Pornography Example

Photo Evidence Example Audiotape, Video file

6

Rosenhan Study (1973) On Being Sane in Insane Places

7

Experimenter Bias

We won’t be needing THAT sample

8

Reliability

Validity

9

10

Sources of Error Types of Bias Confirmatory bias  Primacy effects  Vibrancy effects  Spurious correlation 

11

Confirmatory Bias and Distortion Confirmatory Bias: Internal process which confirms previously held perspective, presumption, belief Confirmatory Distortion: Evaluator intentionally engages in selective reporting or skewed interpretation of data to intentionally bolster a favored hypothesis Inclination to Intentionally seek information that will CONFIRM a hypothesis and Disinclination to find Disconfirming data

12

Dynamics of Confirmatory Bias & Distortion • Cognitive Dissonance reject information you don’t like • Suggestibility (positive test strategy, questions pull for desired outcome)

• Anchoring (false and impertinent information gains salience, priming effect, makes sense)

• Expectancy & Selective Attention, Recall • Confidence Levels and Credibility (overconfident and persuasive, albeit wrong)

• Primacy & Information Integration Adapted from Martindale, 2005

13

Additional Sources of Error, Contamination of Evidence Contamination by Internet Information Contamination by Consultants Impression Management Situational Factors, Sampling Error

14

Common Errors in Custody Evaluations and GAL Investigations Procedural Errors “methodological shortcuts” Failure to observe one of parents and child Failure to apply relevant statutes, case law Lack of equivalent assessments Systemic Errors “thinking thinking shortcuts shortcuts” Thinking too fast; overfocus on one or two proposed explanations; Premature conclusions Assertion Errors “application shortcuts” Misuse or Overuse of Research; overgeneralization without considering limits of application (e.g. joint custody is best; tender years doctrine) Drozd et al. 2013

15

Cognitive Errors and Biases System 1 Thinking: Intuitive, Rapid, Unconscious Processing Relies on emotional centers of brain; adaptive functions; binary processes; rules of thumb; hard to disengage once applied “gut feeling” System 2 Thinking: Analytical, Analytical Slow Deliberation Follows rules, compares objects on several attributes, weighs the information Active approach to decision making. “Sustaining doubt is harder work than sliding into certainty” (Kahneman 2011) Experts are vulnerable to biased thinking that finalizes judgments too quickly

(Adapted from Drozd 2013; Kahneman, 2011)

16

Hazards of Analytical System 2 Thinking Attentional Blindness Ignore the obvious; Invisible Gorilla study (Chabris & Simons, 2010)

One theory trumps other theories; Attachment more salient than Safetyy DV Intelligence and Education do not Prevent Attentional Blindness At Risk for Over Confidence in conclusion Clinical judgment may Fail to Employ Empirical Tools Repetitive successful use of a particular method increases risk – failure to “get out of the box”

Anchoring, Failure to see Obvious

Over Use of Simple Heuristics, Sloppy Thinking A h d tto B Anchored Belief li f or P Prior i Mi Mind-Set dS t

Rose Colored Glasses

17

Specific Errors and Biases in Parenting Assessments

p1

Selective Evidence, Confirmation Bias Gathers facts to support conclusion; disregard or discounts data that challenges viewpoint Premature Termination of Evidence Accepts first alternative that “might work”, conflicting data is ignored Wishful Thinking or Optimism Bias Sees things in positive light; Hope springs eternal; people will change down the road Choice-Supportive Bias Distorts memories to make chosen options seem more attractive, inevitable (cognitive dissonance) Recency Bias More attention to recent information; forgets earlier data From Drozd et al. 2013

19

Specific Errors and Biases in Parenting Assessments p2 Repetition Bias Willing to believe what told most often and by most number of sources Dichotomous Thinking Getting stuck validating specific claims, pro/con, rather than consider big picture issues S Source Bias Bi Reject data if biased against source (Fox v. MSNBC) Incremental Decision Making and Escalating Commitment Decision forms solid, then perpetuates in layers “pearls” Illusion of Control Underestimate future uncertainty on belief of more control Research Bias Because it is published, it is true – for this case11 20 From Drozd et al. 2013

Disaster Neglect May be more pernicious behavior in the future than anticipated Availability Heuristic “Saliency” Believes it because easilyy comes to mind Anchoring Effect Acts based on relative comparison “reduced to $40” sales tool

21

Top Ten Biases Overlooked by Custody Evaluators Martindale 2010 AFCC newsletter

Jiminy Cricket Bias : false belief evaluator can detect deception as easily as Pinocchio Troxelogical Bias : Applies own life experience to make sense of lives of others (based on Troxel v.v Granville) Neuman Bias : Recommendations little more than naïve optimism without basis in the record Imperium Curia Bias : Baseless belief in power of the court. Court orders make it so. 22

Top Ten Biases Overlooked by Custody Evaluators Martindale 2010 AFCC newsletter

p.2

UPAE Bias : Unfortunate Past As Excuse – gives sympathy for parent with troubled past, extra “slack” and naïve hope Intervention Bias : Therapeutic Endeavor in the midst of Evaluation. Conjectura Interdictum (Prohibited Influence) Bias : No such thing as too much information, regardless of how obtained

23

Top Ten Biases Overlooked by Custody Evaluators Martindale 2010 AFCC newsletter

p.3

Associative Bias : Positive bias from shared belief, interest, experience Empathy Bias : Identifies with parent, walking in their shoes

Marital Mindset Bias : Attention on each litigant’s strength and weakness as a spouse rather than on parenting strengths and weaknesses 24

Parenting Plan Evaluator’s Cognitive Error Checklist

p.1

Error Type Self-interested Biases

Solution Check for over-optimism and/or harsh criticism

Over-committment to Recommendations

Look for evidence that does not support your recommendations

Group-Think

Were there dissenting opinions in the data sources? Adequately explored? Check collaterals that don’t support common viewpoint

From Drozd et al. 2013

25

Parenting Plan Evaluator’s Cognitive Error Checklist

p.2

Error Type

Solution

Bias of Memorable Data

Consider how thoughts about case may guide analysis

Confirmation Bias

Are there credible alternatives discussed? What information does not support recommendations?

Anchoring Bias

Do you know how data was anchored? Are there unsubstantiated numbers, extrapolation from history, 26 motivation to use certain anchor

From Drozd et al. 2013

Parenting Plan Evaluator’s Cognitive Error Checklist Error Type

p.3

Solution

Halo Effect

Positive assumption from one area to another? Eliminate false inference by seeking comparable examples

Overconfident Optimistic Bias

Consider how family will manage without court monitoring, involvement of professionals

Disaster Neglect, Loss Aversion From Drozd et al. 2013

Recommendations overly cautious, not cautious enough; Review both extremes

27

Parenting Plan Evaluator’s Cognitive Error Checklist Impact of Professional History with Similar Cases Complacency Burn-Out Glossing Over Q Quick Think Cookie C C Cutter

p.4

Consider issue as if you were a new evaluator assigned to the case

Review and Critique Your Own Report as if done at Consultant for Unhappy Parent

28

From Drozd et al. 2013

Forensic Experts Biased by Adversarial Allegiance

Murrie et al. 2013

National Research Counsel (2009) warning: Accuracy and reliability of many popular forensic science techniques are unknown, error rates rarely acknowledged, forensic scientist prone to bias because not independent of retaining parties Study of forensic psychiatrist and psychologist group n=100; trained on use of PCL-R (Hare) and Static-99R Review 4 case files, prosecution and defense group; Give Risk Assessment ratings. Interpersonal Affective and Social Deviance ratings. Scores coincide with which side “retained” them 29

Legal Standards, Professional Ethics and Procedural Safeguards

G.H. Andrew Benjamin, J.D., Ph.D., ABPP

30

Defining Comprehensive Parenting Evaluation • Allegation/issue/concern driven • Exercises fair process to obtain multiple measure corroboration (WAC 246-924-445)

31

Guidelines for Forensic Evaluators Conducting Parenting Evals • Maintain professional integrity by examining the issues at hand from all reasonable p perspectives p • Seek information which will differentially test plausible rival hypothesis • Engage in consultation if issues exceed competence – http://www.apa.org/practice/guidelines/child-custody.pdf

32

Forensic Psychologist as Court Resource

Key Values and Functions

Integrity and Transparency Role Commitment , Role Boundaries Competence, Application of Research Respect and Empathic Skepticism Humility (leave Ego at home) Recognition of Limits of Tools Information Gathering, Preservation of Records Critical Analysis of Data

Broad-based, multiple-measured evaluation of both parties • Psychology should borrow from the law: “Parents and other parties are likely to advance their concerns in i a fforceful f l and d contentious t ti manner (p. ( 6)… Multiple methods of data gathering enhance the reliability and validity of psychologists’ eventual conclusions, opinions, and recommendations (p. 14).” APA’s Guidelines for Child Custody Evaluations in Family Law Proceedings (2009). 34

Methods and Sources of Data Interviews and Observations Home Visits y g Testing g Psychological Collateral Data

Systematic Analysis

Record Review

Logical Nexus Systems Design Parenting Plans 35

He Said

She Said

They Said

36

Forensic Model of Assessment • Use Multiple Forms of Data (observation, self-report measures, other testing, collateral reports) • Look for Convergence • Explain elements of Divergence • Consider Plausible Rival Hypotheses • Outline Methods, Systematically Investigate and Describe How Arrive at Conclusions

37

G. H. Andrew Benjamin, J.D., Ph.D., ABPP

Statutory Guidelines (e.g., WAC 246-924-445) Professional Standards (APA Ethics Code, APA Guidelines for Conducting Child Custody Evaluations)

Forensic Science and Practice Models

38

Implement a common language for attorneys, judges and behavioral health professionals • APA Ethical a Principles p s of o Psychologists sy o og s s aand d Code of Conduct (2003) 2.01e, 3.11. • What allegations are lawyers of parties indicating should lead to restrictions on final parenting plan? 39

• Use legal factors to organize psychological data within discussion section of report about each concern/allegation/issue (RCW 26.09.187)

• Concrete discussion of data about limiting factors • http://www.apa.org/practice/guidelines/child -custody.pdf (p. 864) 40

Clinical Judgment • The validity of clinical judgments is unrelated to experience level of the professional (Garb, 1989 2005) 1989,

• Errors of clinical judgment include inaccuracies because of over-reliance on memory, confirmatory bias and hindsight bias, and over-reliance on unique data 41

www.afccnet.org Copy available at website for AFCC

42

Consider Using Decision Tree Approach

Drozd, Olesen, and Saini 2013 43

Confirmatory Bias Issues : At the Outset of Case

44

Define Role and Scope Avoid Ex Parte Communication Use of Agreed Stipulation in addition to Court Order of Appointment Explanation of Process, Informed Consent Respect, Tolerance, Empathy

45

Overview of PETP’s EvaluationParallel Process Throughout Eval. • Phase 1- Pre-eval procedures • Phase 2- Clinical Interview • Phase 33- Observations Obse vat o s o of Parents Pa e ts aand d Child(ren) • Phase 4 - Collateral Information • Phase 5 - Interview and Report Findings to Parties • Phase 6 - Presentation to Attorneys 46

How to seem fair? • We are in a mine field of malpractice, ethical complaints, unpaid bills and a lot of client dissatisfaction dissatisfaction….

47

Practice Management: The appearance of fairness

Limit Ex Parte communication: y contact with either lawyer y is written,, and – Any always copied to other lawyer – All contact with parties or collaterals throughout the case occurs in writing, and reviewed by the party and collateral to assure a meeting of the minds has occurred 48

Phase 1- Critical Pre-evaluation procedures • Screen and refer inappropriate cases • Obtain concrete referral questions • Distribute and receive back disclosure forms and outline of the steps of evaluation process • Collect the first advanced fee • Distribute initial testing and consider the results to inform the first interview

49

Confirmatory Bias Issues: During Data Collection

50

Phase 2- Critical Procedures • Finish disclosure process • Administer psychological testing • Focus Foc s on allegation dri driven en psychosocial interview • Integrate data immediately • Send out psychosocial section of evaluation and allegation sections 51

Conducting Interview • Standardized questions (Emery, et al. ((2005)) • Informed by party concerns • Parties directed back to lawyers for advice, support

–No multiple relationships • http://apps.leg.wa.gov/wac/default.aspx?cite=246-924-445

52

Use of Written Survey, Personal History Use of Written Collateral Forms Use of Semi-Standardized Interview Format Have Parties Review Sections when Written

53

Release-of-information forms • Only people with first hand information or the professionals who received i d contemporaneous reports from parties • Provide declarations of nonprofessionals 54

Phase 3- Observations of Parents and Child(ren) •

Fair sampling of parent/ child (ren) interactions

•

Did parent believe it was adequate sampling?

55

Phase 4– Collateral Information Incorporation • Send written summary of interview – http://apps.leg.wa.gov/wac/default.aspx?cite=246-924-445

• Interview p particular p people p before the closing interview with influential party • Reject material that does not directly bear on allegations or protecting the children 56

Collateral Source Investigation Austin, 2002

Austin & Kirkpatrick, 2004

Which Sources (family, friends, third party) Standardized Questions, Additional Focused Items Degree of Alignment, Extent of Knowledge, Possible Error or Bias Behavioral Focus, Examples Investigative Interview, Skeptical Analysis

Risk of Too Much Weight on Collateral Information

57

Austin, 2002

Discriminative Value = Credibility Value x Informational Value Higher credibility value if data source is neutral, less aligned Higher informational value if data source has opportunity to have direct observation Convergent value with information comes from multiple sources Consider hazard of reverberating data and feedback loops

58

Confirmatory Bias Issues : Data Integration and Report Writing

59

Scientific Method: High School Lab example Plausible Rival Hypotheses Alternative Considerations Expression of Limitations When Deviate from Procedural Expectation, Explain Why

60

Examine from All Sides

Guidelines: Paint by Numbers?

See How Pieces Tell the Story

Big Picture and Design 61

Phase 5 - Interview and Report Findings to Parties • Clarify factual inconsistencies. • Discuss evidence that supports findings • Provide additional time to submit contradictory evidence in declarations with releases 62

Phase 6 - Presentation to Lawyers • Meeting with lawyers and the GAL to receive evaluation report and discuss questions about methodology and findings • Follow up phone conference after lawyers have reviewed written evaluation if clarification is necessary 63

Confirmatory Bias Issues : Testimony at Trial

64

Expert Witness On the Stand • Answer questions directly and concisely • Respond to understandable questions only • Draw reasonable inferences between the data of the case and your expert knowledge 65

References • Drozd, L.M., Olesen, N.W., and Saini, M.A. (2013). Parenting Plan & Child Custody Evaluations: Using Decision-Trees to Increase Evaluator Competence and Avoid Preventable Errors. Sarasota, FL: Professional Resource Press • Garb, H. N. (1989). Clinical judgment, clinical training, and professional experience. Psychological Bulletin, 105(3), 387-396. • Garb, H. N. (2005). Clinical Judgment And Decision Making. Annual Review of Clinical Psychology, 1(1), 67-89.

Thanks for your kind attention

Benjamin, McGlothin, & Rybicki 2014