Ministry of Training, Colleges and Universities
Key Performance Indicators for Private Career Colleges September 29, 2015
Presentation to Career Colleges Ontario
Quality and Partnerships Unit Private Career Colleges Branch Postsecondary Education Division
Purpose
Describe the tools and methodology used to calculate Key Performance Indicators (KPIs) for private career colleges (PCCs).
Summarize how to navigate the Forum Portal in order to upload your files (www.pcckpi.ca).
Provide key tips on the data reporting process for new and repeat participants, including on:
Populating data files; Submitting error-free files; Increasing survey response rates; and Completing your audit.
2
Defining KPIs
Following in-depth consultation with students, PCCs and other PCC stakeholders, the ministry developed the following five KPIs for PCCs : PCC KPIs
Public College KPIs
Graduation Rate
Graduation Rate
Employment Rate
Employment Rate
Graduate Satisfaction with Training
Graduate Satisfaction with Training
Graduate Employment Rate in the Field of Study (GEFS)
Student Satisfaction
Employer Satisfaction with Training
Employer Satisfaction with Training
These indicators are seen to be in good alignment with those collected in the public college sector and will provide all stakeholders – students, PCCs and government – useful information to aid in decision making. Throughout the project, the ministry attempted to align the PCC KPIs, and associated tools and methodology required for their calculation, with those used in the public college sector.
3
Phased Implementation Plan
The first phase of KPI implementation commenced on July 12, 2013, with initial focus on PCCs approved to accept OSAP-funded students. All five KPIs will be calculated for OSAP-approved PCCs in 2014, with implementation for all PCCs in the sector in 2015. OSAP Default Rates will continue to be calculated for all OSAP-approved PCCs.
Phases of KPI Reporting PCC Type:
OSAPapproved
All Other PCCs
Phase 1 (2013) Mandatory: (3/5 KPIs) 1. Graduation Rate 2. Employment Rate 3. GEFS
No Reporting (preparing student files)
Phase 2 (2014) Mandatory: All KPIs Adding Graduate Satisfaction and Employer Satisfaction
No Reporting (preparing student files)
Phase 3 (2015)
Mandatory: All KPIs
Mandatory: All KPIs
Rationale • Basic KPI-reporting infrastructure in place from before 2006 moratorium. • Provides ministry the ability to leverage learn from initial implementation issues. • Promotes accountability for government funding. • Ability to leverage lessons learned from Phase 1. • Potential to align majority of PCCs with other ministry systems (e.g., new registration system) at point of Phase 2.
Increased Quality Assurance & Accountability 4
Some Key Terms
Entrants are individuals who enrolled in an approved vocational program provided by a PCC.
Graduates are individuals who completed all of the necessary requirements for an approved vocational program, including all academic and practical requirements.
An Enrolment File is a data (CVS) file used to report to Forum all entrants for one of the three defined Enrolment Terms.
A Graduate File is a data (CVS) file used to report to Forum all graduates for one of the three defined Graduate Terms. 5
Data Uploads
Data on all entrants and graduates must be collected and uploaded to Forum three times per year corresponding to three defined enrolment and graduate terms. Each of the three data upload windows is open for two months to provide schools with enough time to complete their upload for each term. Upload Window
Entrants and Graduates Included
Winter
July 1 – August 31, 2015
January 1 – April 30, 2015 (plus historical data for first time reporters)
Summer
October 1 – November 30, 2015
May 1 – August 31, 2015
Fall
February 1 – March 31, 2016
September 1 – December 31, 2015
Term
6
Graduate Outcomes Survey
Forum Research In. (Forum) uses the graduate information provided by PCCs to conduct a graduate outcomes survey starting six months after graduation. This survey collects both graduate outcomes (employment) and graduate satisfaction information.
One survey window follows each of the three data upload periods. Forum attempts to survey as close to the six month mark as possible).The graduate survey is conducted by telephone in English or French.
The survey consists of standardized questions regarding a specific reference week falling six months after graduation. Survey questions are aligned with those used in the public college sector.
The survey reference week is the first week following the six month period after the individual graduated.
7
Employer Satisfaction Survey
Forum derives the employer satisfaction survey population from respondents to the graduate outcomes survey who consented to their employers being contacted.
The employer survey collects information on the educational preparedness of PCC graduates.
One employer survey window follows each of the graduate outcomes survey windows.
8
Auditing
PCCs’ enrolment and graduate data must be audited according to the ministry’s KPI Audit Guideline, located at www.pcckpi.ca.
Auditors are to examine an appropriate sample of the enrolment and graduate data files to provide the ministry with a level of assurance that the information included in the enrolment and graduate data files was not misstated and correctly matches the records held by the PCC.
PCCs must submit their KPI audit report by September 1st of the year after the collection cycle started (e.g., September 1st, 2016 for the 2015 collection cycle).
Note: For OSAP-approved PCCs that participated in the 2014 collection cycle, any 2014 data files that have not yet been audited must still be audited by the PCC’s 2015 renewal of registration deadline, as per the former audit requirements.
In the final audit report, PCC auditors are required to provide a list of the enrolment and graduate files audited as a schedule to their final audit report, specifying the file reference numbers, original versus revised files and necessary corrections.
9
Payments/Invoices The ministry has developed a streamlined billing process that allows for a one-time billing at the end of each KPI collection cycle, based on the number of completed graduate outcomes and employer satisfaction surveys associated with each PCC. PCCs should expect to receive an invoice from the ministry related to KPI services in the winter of the year following the start of the collection cycle.
For example, for the 2015 collection cycle, PCCs should receive their KPI invoice in winter 2016/17.
PCCs should assume that they will pay approximately $20.00 - $30.00 for each completed graduate outcomes and employer satisfaction survey.
This will include a unit cost ($13.55 for graduate survey; $9.85 for employer survey) as well as a fixed cost allocation which is shared across all PCCs, where their graduates/employers completed the survey. The fixed cost allocation is associated with calculating the final KPIs and producing the preliminary/final reports.
10
Steps to Uploading Data
Log in to Forum’s Portal
Verify your Account Users
Prepare your Data Files
Upload your Files
Auditing your Files
11
1. Logging In
Your Primary User is the same person as your “Head Office Contact” as specified in RICC. This individual must log in to Forum’s portal at https://pcckpi.ca/login and create Secondary Users for each of your institution's campuses. If you are a Primary User logging into the Forum Portal for the first time here are the next steps:
Your login information – portal link, username, temporary password – will be sent to the “Head Office Contact” email address included in the ministry’s Registration Information for Career Colleges (RICC) system.
Your username will be your six-digit RICC Organization ID which starts with a “2”. If your Head Office Contact information is outdated, please update it in RICC and provide it to Forum at
[email protected].
Once your Primary User logs in with the temporary password, he/she will be asked to change it, after which the temporary password will no longer work.
You can change the password at any time after initial login. If you no longer have access to the email with your login information, please use the reset password link: https://reporting.forumresearch.com/PCC/ResetPassword.aspx.
12
2. Verifying your Account Users
After successfully logging into the Forum portal, you must verify the name and email of your PCC’s Primary Account User: the person who will oversee the KPI reporting process, your secondary users and their campus reporting.
You must also assign Secondary Users: one individual at each campus who is responsible for its enrolment/graduate file uploads.
You can modify or delete a Secondary User at any time. Your Primary User can also be one of the secondary users if he/she will also be responsible for uploading data files at specific campuses.
13
3. Preparing Your Data Files
Enrolment and graduate information is provided to Forum in two separate data files – one enrolment file and one graduate file – for each of the three upload windows. Therefore, each PCC provides Forum with six files per KPI collection cycle (as well as historical files for first time reporters). Visit www.pcckpi.ca for current versions of the file templates you must use to report your entrants and graduates. Once logged in, you can view the list of files that must be prepared and uploaded to Forum for each of the three upload windows during the 2015 Collection Cycle. First time KPI reporters must also upload historical data – two files for each term going back 200% of your longest vocational program’s duration.
For example, if your longest program is one year, you must report on all terms going back two years. NEW: The portal will now indicate the date you must go back to.
For the enrolment data file: remove any individuals who withdrew from their program within the grace period (25% of the program to a maximum of 8 weeks). It is important that you upload your files in chronologic order as a graduate will only be accepted into the system if he/she can be matched to a previously submitted entrant. If you do not have any entrants or graduates for a term, you will still be required to submit a null – or “blank” – file. 14
4. Uploading Your Files
Once you have prepared your enrolment and graduate files for a specific term (e.g., Winter), you must upload them one at a time. Each time you upload a file, you will have to provide the email addresses for your auditor and Key Executive (e.g., President, CEO, Owner, or Director). After a file is successfully uploaded, your auditor and Key Executive will receive an email, providing them with a link to view the file. When you submit a file that has an error, please read the error message carefully as it will explain how to fix the error in your file.
You can send your error reference code to Forum if you don’t understand what it means.
Be sure to leave adequate time for your auditor to review your uploaded files. Engage them early so they can schedule your audit it on their calendar.
15
5. Auditing – Fixing Errors
Any errors detected by the PCC auditor in the data files are to be identified by the auditor to the PCC and immediately corrected by the PCC as follows: 1. The PCC auditor is to identify to the PCC the changes required to correct errors in the graduate and enrolment data files; 2. The PCC is then to make the changes required to correct the errors and re-submit these files to Forum, generating new file reference numbers for the revised files; and 3. The PCC auditor is to verify that all errors have been corrected in the new file(s).
A list of the enrolment and graduate files audited, specifying “original” versus “revised” and all necessary corrections, is a requirement of the final audit report, provided by the PCC auditor to the PCC and uploaded to Forum. The PCC auditor must ensure that the revised files contain all of the required corrections.
Graduation rates are calculated based on the audited enrolment and graduate data reported by PCCs to Forum. The remaining four KPIs are calculated from the results of the Graduate Outcomes and Employer Satisfaction Surveys.
16
Now for some key tips…
17
Key Tips – Forum’s Portal
How do I get my log in credentials?
How do I log into the PCC User Management page?
Return to www.pcckpi.ca and instead of using the institution's Organization ID, you will use the email address that was used to set up the Secondary User account and the temporary password provided.
Where can I download my “Approved Program List”?
Make sure you assign the appropriate secondary user to each of your campuses from the drop down menu provided on the portal. Record the temporary password that is shown on the PCC management page after creating each Secondary User account. Provide this password to your Secondary Users as they will need it to log in for the first time.
How do I log in using my Secondary User account?
Each campus must be attached to a unique Secondary User account (email address). It is possible to use the same account (email address) for multiple campuses, but you would need to have a different password for each campus.
What do I do after creating my Secondary User accounts?
Visit https://www.pcckpi.ca/ click on PCC KPI Login, which is located in the menu on the left. When logging into the PCC User Management page, you will use the Organization ID assigned in RICC as your user name (six numbers beginning with “2”).
Can my Primary User upload data files for each of my campuses?
A temporary password will be emailed to your institution's Head Office Contact, as provided in RICC.
The list can be downloaded from the main menu on the submissions page. It’s recommended that you copy and paste program names and details from the Approved Program List when populating the submission files in order to avoid naming errors.
Can I develop my own enrolment and graduate file templates?
In order to reduce errors, we recommended that you make copies of our templates and not try to recreate your own. Templates are available under “Important 2015 Documents” at www.pcckpi.ca.
18
Key Tips – Forum’s Portal
How do I name my Excel data files? In order to upload your files, their names must not begin with a number and must not contain any spaces. “Enrolment2015w” is correct. “2015 Enrolment w” is incorrect. Are all fields (columns) in the file required? NO, Please refer to the Operation Procedure for the required fields (pgs. 26-31). Some fields are mandatory while others are optional. Can I enter Forum’s email or my Inspector’s email in the auditor field? NO, the auditors e-mail should not be associated with Forum or the ministry. What if I don’t have a KPI auditor at the time of my first login? If you have not yet acquired an auditor to conduct the KPI audit, please enter your Head Office Contact’s email address. Can I edit the auditor email at a later date? Yes, in the “submissions page” you can edit your auditor’s email for the given data file upload. Remember to update your auditor’s email information once you have acquired an auditor or if your auditor changes. You can do this via the PCC Management Page. What if my “Approved Program List”, as shown on the portal, is not up to date? Please contact your inspector and make sure your programs are approved in RICC as they are being provided. Request from Forum, a copy of the Program Change Template, and after populating, email a copy to the helpdesk
[email protected]. What if there are no enrolments or graduates in a term, do we have to submit a file? Yes, if there are no students for a given term, a NULL/BLANK file must be uploaded. IMPORTANT: If you are submitting a null file, when probed, select the box labelled “there are no records to submit for this period”. What if I can’t interpret the error message after submitting a file? Email the error reference code to the helpdesk at
[email protected]. 19
Key Tips – Compiling your Data
What if a program does not have an OSAP code?
What if my PCC does not maintain student identification numbers?
Section J allows a PCC to include its own terminology (e.g., program name, code, faculty, location code). If you don’t have this information, you can leave these fields blank.
What if a program is delivered via distance education (not FT/PT)?
YES. Date of birth is used to validate graduates on the phone. Only results from validated graduates will contribute to your KPIs.
What is Section J (PCC program information) in the enrolment data file?
Transfers between approved vocational programs are allowed at any time. Include Y in the M.1 column and the student’s former program in M.2.
Do I need to include a student’s date of birth?
If the student received OSAP, enter Y in G.1. If the student received non-OSAP funding, enter the primary non-OSAP funding source in G.2
What if a student transfers from one vocational program into another?
If you do not already have a unique student identification number assigned to each of your students, you are required to assign them prior to your first data upload. Each student ID number must be 15 characters or less. This number must follow the student from enrolment to graduation.
What if a student has more than one funding source?
Include “XXXX”
Include “CORR” if your program is approved to be delivered via DE.
Do I need to include employer information in the graduate file?
NO. Section L is an optional field. This information may, however, increase the employer survey response rate.
20
Key Tips – Successful Uploading
Populate your files in advance. As individuals enrol in/graduate from your programs, update your data files. Do not wait until the last minute to upload files. If there are problems, you will have more time to resolve them before the deadline. Ensure your key contact information is updated in RICC so that you receive all ministry/Forum updates. Ensure the information uploaded matches what is on RICC and update your information in RICC if need be. Review Forum’s FAQs before you contact Forum or the ministry. Most likely your answer is already provided. Read Appendix D and E of the KPI Operating Procedure to ensure you have the proper coding for each data entry field. Make sure that you include an accurate telephone number and date of birth and for each of your graduates.
The more graduates Forum can contact, the higher your response rate will be. Date of birth is used to validate graduates on the phone.
Update your Hard-to-Find Graduate lists on the Forum portal (www.pcckpi.ca). If possible, inform your students/graduates and any employers with whom you have a relationship with that they will be contacted by Forum six to 12 months after graduation to complete a graduate outcomes or employer satisfaction survey and encourage them to do so. Screenshot your error messages if you are having problems uploading your files. If you are calling Forum and leaving a message, please remember to leave contact information. 21
Contact Us Questions about data files, the portal, file uploads:
Policy-related questions:
Forum Research Inc.
Ministry of Training, Colleges and Universities
Email:
[email protected] Telephone: 1-866-206-7660.
(New) Email:
[email protected]
22
