Kaiser Permanente National Guideline Program: Dynamic Updating Process Craig W. Robbins, MD, MPH Medical Director, Evidence Based Practice, KP Care Management Institute Gladys I. Tom, MS Senior Project Manager, Evidence Services Unit, KP Care Management Institute

Disclosure of Interests (last 3 years) Craig W Robbins, MD, MPH

I certify that, to the best of my knowledge, no aspect of my current personal or professional situation might reasonably be expected to affect significantly my views on the subject on which I am presenting, other than the following: I am employed by the Colorado Permanente Medical Group which works exclusively with the Kaiser Foundation Health Plan in the US. I am not here to specifically promote Kaiser Permanente but merely to share our experience in guideline development.

KP Dynamic Updating Process: Lessons Learned  Updating clinical questions in a rigorous way is labor intensive regardless of whether external guidelines or external systematic reviews are available to use.  Concentrating our evidence analyst resources on the clinical questions most in need of updating will help balance efficiency and rigor in our guideline development processes.

Slide 3

Kaiser Permanente: Largest Non-Profit Health Care Program in the United States Permanente Medical Groups

Kaiser Foundation Hospitals

Kaiser Permanente

Kaiser Foundation Health Plan

 Founded in 1945  8 regions in 9 states and District of Columbia  9+ million members  15,000+ physicians  165,000+ employees  KP Care Management Institute (CMI)  KP National Guideline Program (NGP)

KP National Guideline Program The primary purpose of the Kaiser Permanente National Guideline Program is to provide the organization with the best available systematically derived clinical guidance to improve care delivery and optimize the health of Kaiser Permanente members.

Kaiser Permanente Evidence Network  17 KP National Guidelines • ~200 unique clinical questions across these guidelines • Supported by full Systematic Reviews, Evidence Based Methodology

 3.5 Staff FTEs dedicated to KP NGP work • 1 Senior Project Manager, 2.5 Analysts

 1.2 Physician FTEs dedicated to KP NGP work • 7 Physician EBM Methodologists

 External vendor: Doctor Evidence • Search & Data Extraction • Technology platform  Data repository, analysis & documentation

KP National Guideline Program Portfolio

KP NGP Methodology Update 2009-2011  Prior KP guideline methodology developed internally  Prior significant emphasis on distinguishing between “evidencebased” vs. “consensus-based” recommendations  Decision to adopt GRADE methodology • Separate assessments of evidence quality and strength of recommendations

 Decision to use AGREE II and AMSTAR tools to assess external CPGs and SRs for use within KP  Overall, we are now more systematic, transparent, & explicit in our guideline development processes

8

Kaiser Permanente National Guideline Program: Process & Methodology Clinical Questions New New Evidence CQ Scheduled Update

No

High Priority CQ

Adoptable CPG?

Existing SR by CQ?

Yes Yes

Prioritize CQs

Existing CPG

Additional KP CQs?

Key to Abbreviations: •CQ Clinical Question •CPG Clinical Practice Guideline •KP Kaiser Permanente •SR Systematic Review •GDT Guideline Development Team •GQ Guideline Quality Committee •NGD KP National Guideline Directors

Low Priority CQ

No

Evidence Search

Abstract Article Review

Inclusion/ Exclusion

Data Extraction

Critical Appraisal

Qualitative/ Quantitative Synthesis

Assess Evidence Quality

Systematic Review

Rationale

Recommendations

Yes Appraise SR

No No

Recommendations Acceptable? Yes

Acceptable SR?

No

Yes

Assess Implementability of Recommendations

New Relevant Yes Studies? No © 2012, Care Management Institute, Kaiser Permanente

GDT Approval

Internal Review

GQ NGD Approval National Guideline

Implementation

Kaiser Permanente National Guideline Program: Process & Methodology Guideline Implementation National Guideline

Regions (EHR, CDS, Operations)

Patient Education Tools (KP.ORG, National Patient Instructions

Improved Health of KP Members

© 2012, Care Management Institute, Kaiser Permanente

Clinician & Staff Education Tools (Clinical Library)

Disease Management Accreditation (NCQA)

Key to Abbreviations: •EHR Electronic Health Record •CDS Clinical Decision Support •KP.ORG KP publicly-accessible Internet site •NCQA National Committee for Quality Assurance

Challenges in Systematic Review (SR) & Clinical Practice Guideline (CPG) Development  CPGs not transparent to clinicians • Evidence basis not always explicitly linked • Difficult to assess rigor of development

 Very resource-intensive

• Infinite needs - Finite resources • Balance between efficiency and rigor • Investment in developing expertise & infrastructure • Updating SRs & CPGs  Every 2 years?  Dynamic updating, based on evidence & impact?

KP NGP Dynamic Updating Process: Goal To continuously direct our limited evidence services resources to the clinical questions/recommendations most in need of updating, thereby reaching an effective balance between rigor and efficiency in our guideline development processes.  To do so, we’re developing processes to monitor the KP NGP portfolio’s consistency with current evidence at the level of the individual clinical questions/recommendations.

12

KP NGP Dynamic Updating Process: Status  Confirming relevance of clinical questions with guideline clinical leaders  Prioritizing retained clinical questions for literature monitoring  Updating retained clinical questions and their search strategies  Monitoring literature searches limited to “Top 5” general medical interest journals plus specialty specific journals by topic  Using qualitative and quantitative signals to identify the need for updating

13

KP NGP Dynamic Updating Process: Status  For clinical questions of the following priorities: • Low—update decision deferred until higher priority items addressed • Intermediate—monitoring for qualitative signals of the need to update • High—monitoring for qualitative & quantitative signals of the need to update

 Prioritizing clinical questions for needed updates  Updating prioritized clinical questions and their associated recommendations (using updated external or internal SRs)  Every two years, having at least two clinicians on the guideline team document that all included clinical recommendations remain valid

14

Clinical lead team assess for clinical importance Not critical/ important

Critical/ important

Within GL prioritization (lead team)

Rank order priority across GLs (NGDs)

Lit search update by priority

High Priority Yes

No

Create/update MA of critical outcomes (mini databases-DrE) Quantitative assessment: Meets ANY of these criteria (critical outcomes): 1. The new CI crosses 1.0 when the old one didn't 2. The new CI includes appreciable harm or benefit while the old CI did not; AND both the new & old CI included 1.0 3. The new CI does NOT include appreciable harm or benefit while the old one did; AND both the new & old CI included 1.0 Qualitative assessment: New evidence clearly has the potential to affect clinical decision-making: 1. New studies change interpretation of effectiveness 2. New studies change the balance of benefits/harms 3. New studies introduce a potentially alternative intervention 4. New studies introduce a clinically important expansion of treatment

Update warranted

match CQ/PICO?

No

Go to slide B

Yes

Remove from GL

Yes

Top 5+ journal screen: New evidence

No

Update not warranted

Qualitative assessment: New evidence clearly has the potential to affect clinical decision-making: 1. New studies change interpretation of effectiveness 2. New studies change the balance of benefits/harms 3. New studies introduce a potentially alternative intervention 4. New studies introduce a clinically important expansion of treatment

No

Update not warranted Document in guideline: •Date of update •Statement reflecting new evidence does not change recommendation

Yes

Update warranted

Literature Monitoring: AHRQ SR Surveillance* Method I  Qualitative Signals of Need for Update • Opposing findings • Substantial harm • Superior new treatment • Important changes in effectiveness • Clinically important expansion of treatment • Clinically important caveat

 Quantitative Signals for Need for Update • Change in statistical significance (from non-significant to significant) • Change in relative effect size of at least 50 percent *from AHRQ Draft Report—Surveillance and Identification of Triggers for Updating Systematic Review: Implementation and Early Experience (2012) 16

Literature Monitoring: AHRQ SR Surveillance* Method II  Still Valid—does not need updating. • No new or only confirmatory evidence & experts assessed CER conclusion still valid

 Possibly out of date—may need updating. • Some new evidence and/or a minority of experts assessed CER conclusion as having new evidence that might change the conclusion

 Probably out of date—may need updating • Substantial new evidence and/or a majority of experts assessed the CER conclusion as having new evidence that might change the conclusion

 Out of date • New evidence that rendered the CER conclusion out of date or no longer applicable *from AHRQ Draft Report—Surveillance and Identification of Triggers for Updating Systematic Review: Implementation and Early Experience (2012) 17

Virtual Collaborative Workspace  IT platform as a strategic enabler of our dynamic updating processes  Integrated Components: library management system, study data repository, meta-analysis tool, GRADE-Pro, documentationpublication platform

 Functions: • • • • • •

Clinical question definition & cataloguing Evidence identification Evidence analysis & synthesis Rationale & recommendation development SR & CPG production CDS integration (automated eventually)

KP Dynamic Updating Process: Lessons Learned  Updating clinical questions in a rigorous way is labor intensive regardless of whether external guidelines or external systematic reviews are available to use.  Concentrating our evidence analyst resources on the clinical questions most in need of updating will help balance efficiency and rigor in our guideline development processes.

Slide 19

Questions?  How are your organizations approaching the resource challenges of developing and updating guidelines?

20

Contact List  Craig Robbins, MD, MPH • [email protected]  Gladys Tom, MS • [email protected]