How to Prepare for a Successful ACGME Site Visit aka How to Run a Strong Fellowship Program Association of Specialty Professors April 30, 2011 John (Jack) Buckley, MD, MPH [email protected]
Objectives • Outline the components of an organized fellowship program (i.e. documentation) – This helps drive the development of substantive curricula
• Preparing for the Site Visit • Review what happens at and after the site visit
Here’s what you need: • Program policies and procedures • Detailed fellowship curriculum (readily available to fellows/faculty) • Assessment techniques • Files for each fellow (past and current) • Evidence that you review your program on a regular basis and that you take action steps based on your data • Evidence of duty hour monitoring • Trail of communications (IM SubSpecialty PR: II.A.4.n)
GETTING ORGANIZED and KEEPING ON TASK Consider a Working Calendar (schedule tasks as much as possible) July
• School/hospital ID Badges for fellows • Standard “New Hire” orientation • Confirm procedural competence from IM residency • Core lecture series • Orient to ambulatory clinic • Division’s welcome lunch
Documentation: One method for organization 1. 2. 3. 4. 5.
Program Manual Can be combined for fellow/faculty ease of Program Curriculum use Trainee Documentation Program Evaluation Can be combined Program Improvement Plan
Required Documentation 1. Program Manual (readily available to learners, faculty)
Components: • Policies and Procedures covering: – order writing by residents/fellows and faculty on all teaching services, – lines of responsibility between the faculty and residents/fellows and other specialty residents in a given clinical setting, – moonlighting – care of patients by residents/fellows on non-teaching services (if applicable) • List of teaching methods: need Direct Observation methods (eg. OSCE, team-based learning, simulation, experiential) • Lecture schedules • List assessment tools – Beyond the global evaluation completed by faculty each rotation, include two assessment tools per competency – Ensure at least one Direct Observation tool
2. Program Curriculum:
A curricular document for each clinical/procedural/research rotation or experience
Components: • Description of Rotation & Educational Experience • Written competency-based goals and objectives at each educational level which must be distributed to fellows at the beginning of each rotation. Include definitions of benchmarks (milestones) needed for trainees to advance to the next year of training. • Teaching strategies, types of patients, types of patient encounters, procedural learning • Evaluation methodologies • Reading list
Required Documentation 3. Trainee Documentation
Components: • Create a portfolio (consider a hanging folder system in a file cabinet vs. electronic) with sections for each competency • Require the trainee to monitor and contribute to their own portfolio. This builds self-directed learning skills and tailors educational activities to the individual trainee, while enhancing learner “buy-in” to the process (they created it, they are more invested in it). • Fill Portfolio with: – Clinical Competency Committee reports – Monthly faculty evaluations, nursing evaluations, patient evals., etc. – Semi-annual Evaluations – Feedback sessions with PD and others (APD, KCF, mentors, Chair, etc…) – Documentation from other healthcare providers (letters, e-mail, etc…) – Self-reflective activities – Moonlighting authorization (if applicable) – Copies of projects and their timelines (QI, teaching, research) – Graduates: need THE summative letter
Required Documentation 3. Trainee Documentation (create a section for each of the six competencies) Medical Knowledge: MK • In-service exam scores • Divisional quiz results Patient Care: PC • Faculty global rating forms • Peer global rating forms • Procedure / Patient log book • Chart review/audit • Chart Stimulated Recall exercise • Mini-CEX (cognitive/procedural) or other documentation of direct observation(s)
Required Documentation 3. Trainee Documentation
Communication and Interpersonal Skills: CI • 360º evaluations: – Patient satisfaction survey reports – Nursing / ancillary staff evaluations – Faculty & peer evaluations • OSCE (if applicable)
Practice-Based Learning and Improvement: PBL • Practice improvement activities • Research activities (summaries, reports, abstracts, manuscripts) • Self-Evaluations • Presentations • Teaching activities (students, peers) • QI projects
Required Documentation 3. Trainee Documentation
Systems-Based Practice: SBP • Participation in multi-disciplinary conferences • Certificates of educational module completion (with scores) • Patient safety initiatives • M&M reflective exercise • Participation in outreach programs or other community service involvement Professionalism: • 360º evaluations • Letters of praise / complaints • Attendance records, Chart dictation delinquencies • Examples of mindfulness and sub-ordination of self-interests • Appreciative Inquiry exercise • Attendance / participation at ethics conferences • GME seminars on leadership
Curriculum committee should meet annually (May) and analyze the following: (minutes should be available for the site visitor and include faculty/resident attendance) Components: • Board scores • Trainee evaluations of rotations and program (completed at least once annually) • In-service exam performance (if available) • Post-graduate interviews with graduates and their employers /colleagues • Faculty surveys/feedback • Other: hospital surveys (consider the recent hospital survey on consulting services), prizes & awards • Clinical outcomes some examples……
Hospital Survey on 31 Consulting Services: Courtesy, Effectiveness, Timeliness Department
Nephrology Pulmonary/Critical Care Infectious Disease ICU Nursing General Medicine (MINES D) Ophthalmology Dermatology Diagnostic Radiology Hematology Gastroenterology Oncology Pathology - Anatomic Otolaryngology Psychiatry Pathology - Clinical/Lab Case Management Neurology General Medicine-Hospitalists
83 82 82 82 81 81 81 81 79 79 79 78 76 76 75 74 74 73
ICU Sepsis-Specific Mortality Sepsis Bundled Initiated June 2006
Sepsis Bundle Compliance
Program Improvement Process (Easily combined with #4)
Best done as a formal, annual event (late Spring) by the program’s curriculum committee Process: • Review all program related data/outcomes: – Evaluations (fellow, faculty, rotation, program, graduates, graduates’ employers, residents, nurses, patients),
• Minutes from all fellowship meetings (including GMEC) • Solicit input (formal or informal) from all stakeholders (e.g. rotation directors, resident/fellow focus groups, etc…) • Formulate: – Specific action items – Identify the responsible person – Establish a timeline for re-analysis
• “close the loop” by revisiting the plan at each curriculum meeting • Include faculty development plans
TEMPLATE for a Quarterly Curriculum Committee Agenda Pulmonary / Critical Care Fellowship …….this helps keep you on track to accomplish your program improvement goals ATTENDEES: Gabe Bosslet, MD (Associate PD) Jack Buckley, MD (PD) Katie Sears, MD (Fellow) Mark Farber, MD (KCF) Homer Twigg, MD (Div. Head) David Roe, MD (KCF) I. Follow-up of Previous Meetings / Action Plans: Status report on each action item (include annual PIP and formative items arising from last curriculum meeting) II. Standing Agenda Items: III. New Issues: 1. Announcements 2. Fellowship Program Data Review IV. New Action Items: 1. Item, Responsible person(s), Timeline
Program Evaluation Curriculum committee should meet regularly and analyze the following: (minutes should be available for the site visitor and include faculty/resident attendance) Components: 1. Board Scores 2. Trainee evaluations of rotations and program (completed at least once annually) 3. In-service exam (if available) 4. Post-graduate interviews with graduates and their employers /colleagues 5. Faculty surveys/feedback 6. Other: hospital surveys (consider the recent Hospital survey on consulting services), prizes & awards 7. Patient Outcomes data: eg.: VAP rate 8. Patient surveys 9. Trainee Research Productivity
Program Curriculum A curricular document for each clinical/procedural/research rotation or experience Components: 1. Description of Rotation & Educational Experience 2. Written competency-based goals and objectives at each educational level which must be distributed to fellows at the beginning of each rotation, including stated level of competence 3. Teaching strategies 4. Evaluation methodologies 5. Reading list
Program Improvement Process Program Resources Components: 1. Protected time for PD, Coordinator, secretary 2. Chair support 3. Faculty support 4. Office space 5. Computer support 6. GME, IM, ABIM, ACGME
Components: 1. Minutes from all follow up meetings related to Curriculum Committee concerns documenting a remediation plan(s) (e.g. rotation directors, resident focus groups, etc…) 2. Faculty Development Plans 3. GMEC meeting minutes and activities 4. Action plans 5. Timelines for re-analysis
What is required, specifically?
The ACGME’s Frequently Asked Questions (FAQ’s)
Annual ACGME Survey of your trainees
Preparing for the Site Visit
Nearly 100% of your effort should be in preparation. If you are prepared, organized, thorough, and knowledgeable…..it is a clear message to the site visitor you know what you are doing. • Programs generally receive at least 110 days of advance notice of the scheduled date of their site visit. • You will need to complete the Program Information Form (online) by the assigned deadline • The site visitor assigned to you will contact you 30-60 days in advance to discuss the site visit agenda
When to begin preparation?
think of it as a continuous process of program improvement.
Read every single word of the program requirements and implement everything they say
–“Musts” = Musts –“Shoulds” = Musts (unless you’ve documented a really good reason) –Don’t ignore anything!
Be aware of changes that require ACGME notification (sites, curriculum, change in number of fellows, anything “major”)
When in doubt, call the ACGME
Perfect PIF’s • Review and ensure implementation of all PR’s and address nuances that emerge in the ACGME’s FAQ’s – If you sense something missing in your program, it will come out during a site visit – Don’t allow any gaps – Be prepared to prove everything you claim to be doing – If you’ve never seen what goes into a PIF, take a look now
Filling Out the PIF • Answer questions clearly, briefly, and directly • When asked to list educational activities, consider enumerating the responses to facilitate clarity and organization • Check for grammar and spelling errors • Avoid all institutional slang
Filling Out the PIF • Ask several people to proof-read your PIF for internal consistency • Use the COMMENTS sections to explain potentially confusing components to your program • Common inconsistencies site visitors recognize: – – – – – – – –
# of residents in the program Block diagrams #months on rotation vs. narrative # procedures/resident vs. program total Faculty lists and CV’s Narrative should agree with data/blocks Be direct and to the point when answering questions! Succinctly explain nuances / oddities when given that option
Review changes from last site visit • Has your program expanded? • What were the citations (if any) from the last site visit? Be prepared to show the changes you have implemented • What was the conclusion and advice from your midreview (have a copy ready at the site visit) • Review and make sure you have read and implemented all new PR’s
The Site Visit • Site visitor typically contacts you weeks in advance and proposes an agenda – Meet with you and your Program Coordinator – Meet with your Division Head and/or Chair – Meet with the fellows – Meet with key clinical faculty – Wrap-up
Site Visitor’s meeting with the fellows • This will be private (no PD’s, no faculty, no coordinators) • Fellows who meet with the site visitor must be peer selected • No pagers!!! • No extras (eg. interventional or transplant fellow) • No “prepared” answers. Site visitors are trained to tell the difference
Common Site Visit Questions for Fellows • • • • •
What are the Program Goals and Objectives and where are they located? What procedures are you trained to perform upon graduation? How many patients do you see in clinic? Can you describe under what circumstances you demonstrate continuity of patient care? Do you ever get training in how to recognize fatigue?
…more Site Visit Questions for Fellows •
• • •
What area of the curriculum are you least happy with? What things are being done currently to address this? Is there any reluctance from faculty to fix this issue? Do you ever get videotaped or directly observed while seeing patients? If so, how do you learn from that encounter? As a resident, how do you know how you are doing? What kind of input do you have into the educational effectiveness of the program? What do you need to do to advance from PGY-4 to PGY-5? PGY-5 to PGY-6? PGY-X to Graduation?
….more Site Visit Questions for Fellows • • • •
How do fellows evaluate the faculty? What types of organized clinical discussions (didactics) do your faculty members participate in? In what ways do trainees who are not part of your program interfere with your education? What types of things would prevent a fellow in this program from speaking freely about concerns or problems going on with the program? State any examples when education IS NOT the primary focus on rotations or major assignments. Have you ever seen goals and objectives for any of your rotations? If so, when are they reviewed with you?
…..more Site Visit Questions for Fellows • • • • • •
Are fellows given an opportunity to evaluate the entire program? If so, when? Are fellows given an opportunity to evaluate faculty? If so, how often? What activities have you been involved in to apply the principles of quality improvement? Name one quality improvement activity/project that the residents have been involved in over the past year. How are fellows taught to examine their current patient care practices in the context of scientific evidence? How do fellows learn supervisory skills? How are they evaluated on their supervisory skills?
….more Site Visit Questions for Fellows • • • • •
How do fellows learn to achieve competence in communicating with patients, families, and other colleagues in the medical profession? Do patients, families and members of the health care team contribute to your evaluation of professionalism? Do fellows ever work in inter-disciplinary teams? If so, where? Are you ever taught about safety and medical errors? If so, where? How does the program track duty hours?
Preparing the Faculty for the Site Visitor • Explain the process to the faculty • Faculty interviews serve to confirm the educational components described in the PIF • The site visitor will want to see that they are committed, engaged teachers • Faculty should read the PIF carefully and be familiar with the documentation • These faculty should be sufficiently knowledgeable to answer questions like: – How does your program teach systems-based practice? – What is your process for program improvement? – What type of multi-source assessment is your program using?
What happens during the faculty interview? • Faculty may be asked about the competencies, what kinds of assessment tools are in place, how promotion and graduation are determined, how remediation is determined • They may also be asked about the learning environment (i.e. duty hours, supervision, feedback re: program improvement) • What do you do for faculty development? • Faculty may be questioned about scholarly activity output
2009 data Responses from 10 faculty to my challenge to list all six core competencies • • • • • • • • •
Medical Knowledge (10) Patient Care (7) Communication and Interpersonal Skills (7) Professionalism (8) Practice-Based Learning and Improvement (1) Systems-Based Practice (1) Procedures (4) Honor Thy Faculty (1) Love of country (1)
Questions for Program Directors • Topic areas you may be asked about: • Duty hours • Supervision • Quality of education and educational environment (e.g. are you getting the support you should; chain of command; free of intimidation) • Competencies – where they are taught and evaluated • Clarification of contents of PIF • How is attendance tracked during conference? • Describe a particular experience; review goals/objectives for a particular rotation)
Wrap-Up with the Site Visitor • Site visitors may give you a chance to explain observed inconsistencies (PIF, PD statement, resident statements) • Keys to Success: – Clear, accurate, consistent responses – Organized, attentive, knowledgeable – Open, honest, sincere
What happens after the site visit? • Site visitor submits a written report to the ACGME, which is then forwarded to the Residency Review Committee reviewers • The RRC reviews your program, which is based on your PIF and its attachments, and the site visitor’s report. The site visitor does not participate in the accreditation decision • Most RRC’s meet twice or three times per year, while some that review a small number of programs may only meet once • RRCs close out their agendas approximately 60 to 75 days before the meeting date. For a program site visited less than four months before a meeting date, there is no guarantee that it will be presented at the next meeting
Letter of Notification • Will be received by Program Director and Designated Institutional Official (DIO) approximately 60-90 days after the RRC review (cycle length will be communicated immediately after review by email) • Accreditation status • Length of training/number of trainees/level • Effective date of accreditation action • Approximate date of next site visit • Cycle length • Approximate date of mid-review • Progress report due (the letter will provide citation details) • Areas not in substantial compliance (if no citations and/or the program receives a 4-5 year cycle length this section may also include commendations)
Accreditation Actions • The following accreditation actions may be taken by a Review Committee in the accreditation of specialty programs, independent subspecialty programs, transitional-year programs, and sponsoring institutions:
Withheld Accreditation Initial Accreditation Continued Accreditation Probationary Accreditation Withdrawal of Accreditation
• Accreditation withheld, probationary accreditation, and withdrawal of accreditation, as well as a reduction in fellow complement by the RRC, are adverse actions and subject to an appeals process
Summary • Running a strong program is the best preparation for an ACGME site visit • All of the program requirements are spelled out (in detail) on the ACGME’s webpage • Good Preparation Pays Off – Documentation – Resident and Faculty readiness • In the eyes of the ACGME, “if it is not documented, it didn’t occur”