Accreditation Council for Graduate Medical Education

Psychiatry Review Committee Update February 27, 2015 George A. Keepers, MD, Chair Review Committee for Psychiatry

ACGME Annual Educational Conference Manchester Grand Hyatt San Diego and Hilton San Diego Bayfront San Diego, California February 26-March 1, 2015

Disclosures • No conflicts of interest to report

Discussion Topics

• RC for Psychiatry Membership • RC for Psychiatry Activities • ACGME Initiatives

Review Committee Membership • George A. Keepers, MD RC Chair • Mark Servis, MD RC Vice Chair • Iqbal Ahmed, MD • Robert J. Boland, MD • Carlyle H. Chan, MD • Josepha Cheong, MD • Stephen P. Cuffe, MD* • Mina Dulcan, MD • Anne L. Glowinski, MD

• • • • • • • • •

*Terms end June 30, 2015

M. Philip Luber, MD Gerald Maguire, MD Robert J. Ronis, MD Heather Schultz, MD* (resident member) Andrea Stolar, MD Richard F. Summers, MD Dorothy E. Stubbe Larry Faulkner, MD ABPN Ex-Officio Annelle Primm, MD APA Ex-Officio

Incoming RC Members • Stephen A. Epstein, MD (Replaces Dr. Cuffe) • MedStar Georgetown University Hospital, Washington, DC • Cristin C. McDermott, MD (Resident Member) • University of Pittsburgh Medical Center, Pittsburgh, PA • Jed Magen, DO (AOA Member) • Michigan State University, East Lansing, MI • Ronald Paolini, DO (AOA Member) • Eisenhower Army Medical Center, Fort Gordon, GA • Deborah Simpson, PhD (Public Member) • Aurora Health Care, Milwaukee, WI Terms begin July1, 2015

ACGME RC for Psychiatry Staff ACGME Leadership • Mary Lieh-Lai, MD, Senior Vice President for Medical Accreditation (312) 755-7405 – [email protected] RC Staff • Louise King, MS Executive Director - (312) 755-5498 – [email protected] • Karen Lambert Associate Executive Director, RC for Internal Medicine (312) 755-5785 – [email protected] • Debra Martin Accreditation Administrator - (312) 755-7471 – [email protected]

RC for Psychiatry Meetings

• February 13-14, 2015 • Agenda Closed December 5, 2014

• June 5-6, 2015 • Agenda Closing March 27, 2015

• Note – These are the same agenda closing dates for the submission of new applications.

RC for Psychiatry Report Program Accreditation Academic Year 2014-2015 Psychiatry Brain Injury Medicine Addiction Psychiatry Child and Adolescent Psychiatry Forensic Psychiatry Geriatric Psychiatry Pain Medicine Psychosomatic Medicine Sleep Medicine

193 0 46 124 40 55 0 54 4

Current RC Projects • ACGME Next Accreditation System • Revision Program Requirements • General and Child Psychiatry

Aims of NAS • Enhance the ability of the peer-review system to prepare physicians for practice in the 21st century • To accelerate the movement of the ACGME toward accreditation on the basis of educational outcomes • Reduce the burden associated with the current structure and process-based approach • Note: this may not be evident right away

The Next Accreditation System in a Nutshell • Continuous Accreditation Model • Resident and Faculty Surveys • ADS • Milestones • Scheduled Site Visits Replaced • 10 year Self Study Visits • Focused Site Visit • Full Site Visit • Program Requirements revised every 10 years instead of every 5 years

Data Elements of NAS The following are the primary annual data elements: • • • • • • • •

Program Attrition Program Changes Scholarly Activity Board Pass Rate Clinical Experience Data Resident/Fellow Survey Faculty Survey Milestones

Where did most of the NAS data elements come from? • In 2009, data modeling project began to identify factors that predicted high and low program performance • A number of statistical methods were used • Model was replicated • Data elements were assessed to determine “relative risk” to predict low performance • Selection of Elements needed to be • Obtainable • Meaningful • Correlates w/prior decisions • Passed statistical “muster” • Used in combination • Understand that this is a work-in-progress • New data elements likely in future

Role of Review Committee in NAS • Utilize data and judgment to: • Concentrate efforts on problem programs • Determine whether accreditation standards are violated and provide useful feedback for programmatic improvement

• Determine whether these violations (citations) rise to a level requiring alteration in accreditation status • Motivate programs to rapidly improve, rather than playing the “accelerating accreditation action game” • Over time, understand and refine the nuances of the process

• Conduct complete review of the program q10 years, using a “PIF-less”, team based, department wide evaluation of programs

NAS Year 1: Ground rules • Basic operational principle of NAS: • RC will take an accreditation action on every program annually. • All Psychiatry programs will receive notice regarding accreditation status between January and July.

• At February 2015 meeting, RC reviewed NAS data submitted for AY 2013-2014 • ADS annual update information submitted in fall of 2013 • Faculty and Resident/Fellow survey data from early spring of 2014

NAS Year 1: Ground rules • All programs w CW or probation seen by reviewers • All programs identified by NAS data as “troubled” underwent further scrutiny by RC staff and members • What data elements were triggered? • Not all data elements have same importance/weight • Board scores and resident survey have more weight • Are programs still getting used to data elements (e.g., scholarly activity table)? • Are there patterns/trends in data?

PD Responsibility: Accurate Data •

Program Director: • • •

•

Must provide complete and accurate information Review all information before “hitting” the submit button DIO should also review before submission Common Omissions: • Faculty credentials (degree, certification, re-cert) • Participating sites • Complete scholarly activity • Updated response to citation(s) • Complete block diagram

PD Responsibility: Identifying Faculty for Faculty Roster •

Identifying who to list in ADS: Only physicians count as core faculty • Core Programs: Only faculty who spend at least 15 hours or more/week on residency (including clinical, didactic, research and administration) • Subspecialty Programs: Only faculty who spend at least 10 hours/per week • Review instructions in ADS: List minimum # for size of program Core faculty members complete: • Scholarly activity Report • Faculty survey •

•

Highlights from New P&P Manual • No longer “propose” adverse actions •

Review status options (next page)

• Core and Subs have same status options • Probation now an option for subs

• Core + Subs re-coupled…again • At time of self-study, RC will see entire department/unit • Probation for core = probation for subs

Accreditation Status Options New Applications

New Application

Significant change: no “proposed” withhold; applications either receive “initial accreditation” or “accreditation is withheld.”

Initial Accreditation Accreditation Withheld

Reapplication anytime or Appeal

Accreditation Status Options Initial Accreditation Significant changes: maximum length for “initial accreditation” = 2 years; can move to “continued accreditation w/o outcomes”; can move from “initial accreditation” to “initial accreditation w warning”

Initial Accreditation SV in 2 yrs.

Continued Accreditation (CA)

Continued Accreditation without Outcomes CA Initial Accreditation w/Warning SV in 1 yr. WD

Withdrawal of Accreditation (WD)

Accreditation Status Options Continued Accreditation

Accredited Program

Significant changes: “Continued Accreditation with Warning” appears as such on website; an adverse action can not be granted without a site visit; subs now can also be put on “probationary accreditation.”

Continued Accreditation (CA)

CA w/Warning Other (e.g. egregious)

Probationary Accreditation* Withdrawal of Accreditation**

Site Visit

* Probation cannot exceed 2 years ** Does not require Probation first

CA w/Warning CA

Citations in NAS Citations are not new • Identify areas of non-compliance • Linked to specific requirements • Responses required in ADS • Citations are given and removed by RC (not by staff) Citations received in NAS (after July 1, 2013): will require an RC member to review annually. Citations received in OAS (given prior to July 1, 2013): will go away after two cycles of continued accreditation in NAS with no new citations.

Areas for Improvement (AFI) • AFIs are new in NAS • “General concerns” • May be given or removed by staff (RC rules) or RC-Psychiatry • May not be specifically linked to a requirement • Do not require written response in ADS • Expectation that AFIs will be monitored locally • PD and GMEC will work to resolve • AFIs will be tracked by RC

Citations vs AFI’s • In OAS, the main mechanism to provide feedback was through citations • In NAS, we have 2 methods: citations and AFI’s • Citations require annual review by a member of the RC Citations will likely be used more sparingly, in hopes that AFI’s trigger appropriate local program improvement.

Letter of Notification (LON) • New Departmental LON • Summarizes actions for entire department • Sent to core program directors, subspecialty program directors and DIO

LONs from February 2015 Meeting • Will receive or be copied on departmental letter ……and…. • If program receives new citation(s): • will receive separate LON.

• If program DOES NOT receive new citation(s): • Core: will receive separate LON as well • Sub: will NOT receive separate LON

• If program DOES NOT receive new citation(s), but DOES receive AFIs: • Will receive separate LON with AFIs

Site Visits in NAS Three types • Full site visits • Focused site visits for an “issue” • Self-study visits every ten years

Full Site Visits • Application for a new core program • At the end of the initial accreditation period • RRC identifies broad issues/concerns • Other serious conditions or situations identified by the RRC • Notification given ~ 60 days • Minimal document preparation • Team of site visitors

Focused Site Visits • To assess selected aspects of program: • Potential problems identified during annual review • To diagnose reason for deterioration in performance • To evaluate complaint

• Minimal notification given (~ 1 month) • Minimal document preparation expected • Team of site visitors

Self-Study Visit What is it? • Not fully developed • Will review core + subs together • Scheduled every ten years • Conducted by a team of visitors • Minimal document preparation • Interview residents, faculty, leadership

© 2013 Accreditation Council for Graduate Medical Education (ACGME)

Self-Study Visit (cont.) • Examine annual program evaluations • Response to citations • Faculty development

• Focus: Continuous improvement in program • Learn future goals of program • Will verify compliance with core requirements

© 2013 Accreditation Council for Graduate Medical Education (ACGME)

Ten Year Self-Study Visit SelfStudy VISIT

Self-Study Process Yr 0

Yr 1

Yr 2

Yr 3

Yr 4

Yr 5

Yr 6

Yr 7

Yr 8

Yr 9

Yr 10

AE

AE

AE

AE

AE

AE

AE

AE

AE

AE

Milestones • Next step in the ACGME Outcome Project • Milestone definition: Specific behaviors, attributes, or outcomes in the general competency domains to be demonstrated by residents by a particular point during residency • Aggregate resident milestone performance used as an indicator of a program’s effectiveness

Milestones • Measure a residents progress in training • Assessed every 6 months

• Grouped by the 6 Core Competencies • Established by professional consensus • Assist in program self evaluation and accreditation

Milestones are a paradigm shift Previous Assessment • • • •

Global Summative Comparison Non-directive

Milestone Assessment • • • •

Focused Specific Individual Prescriptive

Milestones • General Psychiatry • Final version posted to ACGME website • First reports enter November 1 to December 31, 2014

• Psychiatry Subspecialties • Final version posted • All subspecialties will formally begin to use the Milestones in 2015 • First Reporting November 1 to December 31, 2015

Membership Working Group • • • • • • • •

• • • • • • •

Chris Thomas (chair) Sheldon Benjamin Adrienne Bentman Robert Boland Deborah Cowley Jeffrey Hunt George Keepers Gail Manos

Donald Rosen Kathy Sanders Mark Servis Kallie Shaw Alik Widge (resident) Susan Swing (ACGME) Louise King (ACGME)

Advisory Group • • • • •

Chris Thomas (chair) Timothy Brigham (ACGME) Susan Swing (ACGME) Carol Bernstein Larry Faulkner

• • • •

Deborah Hales Victor Reus Richard Summers Beth Ann Brooks

Milestones in a Nutshell • Move accreditation from structure and process-based to outcomes-based • Progression from entering resident to independent practitioner • Articulate shared expectations across programs • Set aspirational goals of excellence • Framework & language for discussions

ACGME Milestones Reports • RCs will receive only aggregate data • Programs may receive individual reports • ? Individual data to the Specialty Boards— the ABPN will not request Milestones reports on individuals

© 2014 Accreditation Council for Graduate Medical Education

Threads • 66 “Threads” • Developmentally linked milestones • Not required • Assist in assessment tool development

Milestones • NOT AN ASSESSMENT TOOL • Do not have to assess all 22 Subcompetencies or 66 Threads for each resident at the end of each rotation or each task

• Do not discard all current assessment methods • Map the Milestones onto existing tools

• Add new ones based on Milestones • Use the assessment tools to guide the milestones progress determined by the CCC

Competency Evaluation Mock Orals Operative Performance Rating Scales

Nursing and Ancillary Personnel Evaluations

End of Rotation Evaluations

ITE

Self Evaluation

EPAs

Sim Lab

Clinical Competency Committee

Case Logs Unsolicited Comments

Student Evaluations Clinic Work Place Evaluations

OSCE Peer Evaluations

Assessment of Milestones

Patient / Family Evaluations

Screen Shot – Core Pediatrics Milestones Reporting Form on ADS

Competency

Subcompetencies

Milestone level with mouse-over description

Milestones • Resident NOT required to meet EACH Level 4 item to graduate • Residents NOT assured of graduation solely on basis of Level 4 achievement • Milestones are NOT the only measure of competency • Milestone Levels 2, 3 & 4 do NOT necessarily correlate to PGY 2, 3 & 4

Milestones Reporting

• Phase II specialties • November 1 – December 31, 2014 • May 1 – June 15, 2015

Clinical Competency Committee

Clinical Competency Committee • Minimum 3 program faculty members • Can include non-physician members • Tasks: • Review all resident evaluations semiannually • Prepare Milestones rating of each resident • Advise PD of resident promotion, remediation, probation and dismissal

Clinical Competency Committee • May already exist under a different name • Plan ahead • What should be reviewed: • Continue to use current methods of evaluations • Incorporate new Milestones assessments

• Important for coordinator to be present at meetings • Issues: • Time constraints • Large residency programs • Small fellowship programs

Clinical Competency Committee • The role of the Program Director in the CCC is undefined • Chair • Member • Ex-officio • Not a member of the CCC

Clinical Competency Committees • Learn about/understand the Milestones • Decide how to assign Milestones • Which clinical rotations? • Which tasks? • What focused evaluations? • What summative evaluations? • Teach the faculty: • Definitions • The tools • FACULTY DEVELOPMENT IS KEY

Program Uses for the Milestones • Provide feedback to residents • Benchmark residents to program mean • Benchmark residents nationally • Determine program strengths • Identify program opportunities for improvement • Benchmark program nationally

Resident Uses for the Milestones • Receive specific feedback • Determine individual strengths

• Determine individual training goals • Learn rotation objectives

• Learn evaluation measures

CLER Program • Clinical Learning Environment Review • Institutions will be visited every 18 months • Data will not be used for accreditation, but……. • Programs must ensure that residents and fellows: • Are aware of patient safety/quality improvement efforts of the institution • Are actively participating in PS and CQI efforts

Psychiatry and Child and Psychiatry Adolescent Program Requirements • Initial Draft of Program Requirements • Review by ACGME and revised by RRC • Revisions posted for 45 day Public Comment period (ended July 23, 2014) • Public Comments and comments from ACGME reviewed by RC. All comments addressed and no more public comment period • Program Requirements revised • The proposed Psychiatry and Child and Adolescent Psychiatry Program Requirements were reviewed by the ACGME Committee on Requirements at its February 6, 2015 meeting for an effective date of July 1, 2015 • Until the Program Requirements are effective the CURRENT Program Requirements, are in effect.

Changes to the Psychiatry Program Requirement Summary • Program Personnel and Resources • Statement reflecting that RC accepts only ABPN certification was deleted to accommodate potential changes as a result of the Single Accreditation

• Other Program Personnel • Requirement that the Chair of Psychiatry must be certified by the ABPN challenged by CoR.

• Curriculum Organization and Resident Experiences •

Incorporation of Milestones and the CCC

• Program Evaluation and Improvement •

At least 70 percent of a program’s graduates who completed the program in the preceding three years should take the certifying examination in general psychiatry. (Outcome)

•

At least 70 present of a program’s graduates from the preceding three years who take ABPN certifying examination in general psychiatry for the first time must pass. (Outcome)

•

Percentage changed from 80 to 70 based on feedback from the field

Changes to the Child and Adolescent Psychiatry Program Requirement Summary • Institutions •

At least, 50 percent salary support and protected time of 50 percent FTE (at least 20 hours per week) for the program director

•

At least, 20 hours per week for direct program administration and education for the program director

•

Approved fellow complements of 20 or more require 30 hours per week effort and/or the appointment of an associate program director

• Other Program Personnel • Requirement that the Chair of Psychiatry must be certified by the ABPN has been challenged by the CoR

• Curriculum Organization and Resident Experiences •

Incorporation of Milestones and the CCC

Changes to the Child and Adolescent Psychiatry Program Requirement Summary • Program Evaluation and Improvement •

At least 70 percent of program graduates from the preceding five years should have taken the certifying examination. (Outcome)

•

At least 50 percent of a program’s graduates from the preceding five years who take the ABPN certifying examination in child and adolescent psychiatry for the first time must pass. (Outcome)

Psychiatry and Child and Adolescent Psychiatry Program Requirements

• Approved Psychiatry and Child and Adolescent Psychiatry Program Requirements and FAQs will be posted to the ACGME website at www.acgme.org within the Psychiatry specialty page

Single Accreditation System

Psychiatry AOA Programs • As of December 2013, there are 14 AOA Psychiatry residency training programs

Accreditation of AOA Programs AOA-Approved Program Yes

AOA-Approved as of July 1, 2015

Program is under aegis ACGMEaccredited sponsoring institution

No Sponsoring institution applies for ACGME accreditation April 15, 2015 – June 30, 2020; Receives “Pre-Accreditation Status”

Yes

No

C

Program cannot apply under the terms of the agreement between AOA, AACOM and ACGME; program begins standard ACGME application process at any time through the DIO of an ACGME-accredited sponsoring institution.

Program submits ACGME application with ACGME-accredited sponsor endorsement July 1, 2015 - June 30, 2020; Receives “Pre-Accreditation Status”

No

Program had matriculated residents/fellows as of July 1, 2015

Yes

A

B Review Committee assesses substantial compliance with current ACGME requirements

Review Committee assesses substantial compliance with current requirements, with two exceptions: 1. Program may have AOA-certified co-program director 2. AOA-certified faculty members are acceptable

Timeline for Accreditation • To apply, programs must be associated with ACGME-accredited sponsoring institution or institution with “Pre-Accreditation Status” • Window for institutional accreditation open 1 April 2015 - 30 June 2020 • New institutional application process

Webinars • Previous webinars available for review at: http://www.acgmenas.org/index.html under “ACGME Webinars” • CLER • 2013 Coordinator Webinars • NAS Phase I and Phase II: Overview of Next Accreditation System • Milestones, Evaluation, CCCs • Stand-alone slide decks for GME community: NAS, CCC, PEC, Milestones, Update on Policies • Self-Study

Questions

Thank you