Recommendations for the Forensic Disciplines Constantine Gatsonis Department of Biostatistics Brown University Science & the Congress Project , Sept 2012
Sept 2012
Gatsonis, Brown University
NAS 2009 report on
Strengthening Forensic Science in the US Major recommendation: Create new Agency for Forensic Sciences Additional Recommendations address need for: • • • • • •
Standardized Terminology and Reporting Scientific Research in the Forensic Disciplines Best Practices and Standards Quality Control, Assurance, and Improvement Codes of Ethics Education and Training
• Reform in Medicolegal Death Investigation System • AFIS and Database Interoperability Sept 2012
Gatsonis, Brown University
1
Forensic sciences are interdisciplinary Molecular biology • Nuclear DNA analysis • Mitochondrial DNA analysis Examples
Pattern recognition • Fingerprint analysis • Handwriting Analytical Chemistry
Hair and fiber analysis Sept 2012
Gatsonis, Brown University
Errors and error rates •
•
Examples: –
Laboratory analyses are subject to measurement error (i.e. uncertainty about true quantity)
–
Fingerprint analyses can lead to false identification of individual prints, e.g. because of observer error, low specimen quality, etc
–
DNA analyses can lead to false identification of individuals, e.g. because of contamination of samples, laboratory errors, etc
Fuzzy use of language. Need to state precisely the error under consideration. Sept 2012
Gatsonis, Brown University
2
Step 1 in the evaluation of the accuracy of forensic analyses : Define the task • Individualization: – Can a piece of evidence be associated with a particular source?
• Classification: – Can a piece of evidence be associated with a particular class of sources? • A few forensic modalities have potential for individualization. • More of them have potential for classification. Sept 2012
Gatsonis, Brown University
Example: A fingerprint study A set of pairs of prints is interpreted Data from each interpreter: Analysis results Truth
match
No match
Pair of prints comes from same individual
True False Positive Negative
Pair of prints comes from different individuals
False True Positive Negative
Sept 2012
Gatsonis, Brown University
Errors!
3
A general approach for accuracy • Borrowing from the paradigm of diagnostic testing • The well known 2x2 table for dichotomous test and truth: Forensic analysis results Truth
“yes”
“no”
“yes” (Target condition present)
True False Positives Negatives
“no” (Target condition absent)
True False Positives Negatives
Sept 2012
Gatsonis, Brown University
Need for research to reliably assess accuracy • The paradigm of accuracy assessment can be useful in many settings. • It requires substantial research effort. • It needs to be supplemented by other research to address important questions in individualization: – Definition of “match” – Estimation of random match probabilities • Of key importance in judicial setting: Paradigm addresses performance over repeated instances of the analysis. It does not necessarily guarantee the correct answer in a specific case. Sept 2012
Gatsonis, Brown University
4
Learning from the evaluation of diagnostic tests • Long tradition of studies of accuracy of diagnostic modalities. • “Moving target problem”: Technology evolves, often quite rapidly. 0.0 0.2 0.4 0.6 0.8 1.0
Sensiti vity
• Variability in reader performance. • Assessing/monitoring effectiveness (i.e. performance in everyday use) is challenging.
0.0 0.2 0.4 0.6 0.8 1.0 1 - Specificity
Sept 2012
• Major research effort and infrastructure is needed Gatsonis, Brown University
Analysts can vary in accuracy
Variability among mammographers interpreting common set of scans. (Beam, Arch Int Med, 1996) Sept 2012
Gatsonis, Brown University
5
High tech and new tech may not necessarily better
MRI and MRSI for localizing cancer in prostate Radiology, 2009 Sept 2012
Gatsonis, Brown University
In summary • Much research remains to be done to assess the accuracy of forensic identification methods based on pattern recognition. • Technology transfer of methods from other fields can be very helpful. • Interdisciplinary research enterprise is needed to support the generation of reliable finding, to understand the limitations of the analyses, and to make appropriate presentations of the results to the courts.
Sept 2012
Gatsonis, Brown University
6