Audit of the Management of Call Centre Operations Audit Report Project Number: 17031/09-10

SP-1036-12-12E

October 2012

Audit of Management of Call Centre Operations

Paper ISBN: Cat. No.:

978-1-100-21218-0 HS28-205/2012E

PDF ISBN: Cat. No.:

978-1-100-21219-7 HS28-205/2012E-PDF

Audit of Management of Call Centre Operations

Table of Contents EXECUTIVE SUMMARY ................................................................................................. i 1.0 BACKGROUND ....................................................................................................... 1 1.1 1.2 1.3 1.4

Context ...................................................................................................... 1 Audit Objective ......................................................................................... 1 Scope ......................................................................................................... 1 Methodology ............................................................................................. 2

2.0 AUDIT FINDINGS ................................................................................................... 3 2.1 2.2 2.3

A governance framework is in place and is working well ........................ 3 An effective risk management approach is in place .................................. 5 Controls over quality assurance could be strengthened ............................ 6

3.0 CONCLUSION .......................................................................................................... 9 4.0

STATEMENT OF ASSURANCE ............................................................................ 9

APPENDIX A: Audit Criteria .......................................................................................... 10 APPENDIX B: Call Centre SMSM organizational structure ........................................... 13 APPENDIX C: Glossary ................................................................................................... 14

Audit of Management of Call Centre Operations

EXECUTIVE SUMMARY Human Resources and Skills Development Canada’s (HRSDC) network of 14 specialized call centres is the key point of contact for citizens/beneficiaries to access their client-specific information under the Employment Insurance (EI), Canada Pension Plan (CPP), Old Age Security (OAS) and Employer Contact Centre programs. In fiscal year 2011–2012, these specialized call centres answered 8.69 million calls from Canadians. The call centre network is managed through the four regions and National Headquarters (NHQ). The Call Centre Directorate (CCD), within the Processing and Payment Services Branch (PPSB) in NHQ, is responsible for the overall planning and scheduling strategy, and developing the call centre network budget and forecasts. NHQ also oversees program delivery and provides guidance relating to quality and harmonization. The delivery of programs and services within each region is governed by the regional Executive Head, Service Management (EHSM). Audit Objective The objective of this audit was to provide assurance that the management control framework supporting the EI and CPP/OAS call centres is in place and applied consistently across all call centre operations. Summary of Key Findings  Roles and responsibilities are clear and accountabilities are defined.  A performance management strategy, comprised of a suite of indicators related to the network’s speed of service is in place. It is not clear how these indicators and related performance data align with the network priority of first contact resolution.  Risk management principles and practices are embedded in call centre operations.  A workload management strategy is in place, but call centres continue to face significant challenges in meeting current service levels.  Scheduling of call centre resources supports service delivery and people management.  Training regime, procedures and systems are in place to support call centre agents.  A quality assurance program is in place to monitor calls and generate reports for coaching. Guidelines are not consistently applied. Opportunities exist to implement a methodology that takes into account operational changes.

Internal Audit Services Branch, HRSDC

i

Audit of Management of Call Centre Operations

Audit Conclusion Overall, a management control framework supporting the EI and CPP/OAS call centres is in place and applied consistently across all call centre operations. However, the audit found areas for improvement related to quality assurance. Recommendation 1. To ensure that relevant and useful information is used in its National Quality Assurance Program (NQAP), PPSB should: a. Implement a call monitoring methodology that will reflect a reasonable distribution of monitored calls over time and consider operational changes, e.g. recent updates to procedures may call for increased monitoring. b. Ensure timely and complete coaching sessions between Citizen Service Officers (CSOs) and Team Leaders (TLs) are conducted and documented.

Original signed by:

_________________________________________________ Vincent DaLuz, CA, CIA Chief Audit Executive Department of Human Resources and Skills Development Audit Team Members: Senior Director – Brigitte Marois, CGA, CMA Audit Principal – Isabelle Barrette, CMA Audit Principal – Rajni Patel Nathan Ferguson Emmanuel Ouellette Amanda Elliott

ii

Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

1.0

BACKGROUND

1.1

Context

HRSDC network of 14 specialized call centres is the key point of contact for citizens/beneficiaries to access their client-specific information under the EI, CPP/OAS and Employer Contact Centre programs and to address enquiries regarding the status of their files. Call centres also process transactional requests such as changes of address, direct deposit or marital status, renewals and/or cancellation of benefits. In fiscal year 2011–2012, these specialized call centres answered 8.69 million calls from Canadians. Of this, approximately 800 EI call centre agents answered over 5.3 million calls and approximately 400 CPP/OAS call centre agents answered over 3.0 million calls. The call centre network consists of four regions along with NHQ. The CCD in NHQ is responsible for the overall planning and scheduling strategy and developing the call centre network budget and call volume forecasts. NHQ also oversees program delivery, provides guidance relating to quality and harmonization, and leads the implementation of any enhancements to the network. The delivery of programs and services within each region is governed by the regional EHSM. 1.2

Audit Objective

The objective of this audit was to provide assurance that the management control framework supporting the EI and CPP/OAS call centres is in place and applied consistently across all call centre operations. 1.3

Scope

The scope of this audit included the CCD, PPSB at NHQ and the 14 EI and CPP/OAS specialized call centres located across Canada (five EI sites, three CPP/OAS sites, six sites housing both EI and CPP/OAS). The following call centres were visited in February and March 2012:    

Edmonton EI & CPP/OAS call centres Montreal EI & CPP/OAS call centres Bathurst EI & CPP/OAS call centres Toronto EI & Scarborough CPP/OAS call centres

Internal Audit Services Branch, HRSDC

1

Audit of Management of Call Centre Operations

1.4

Methodology

A number of methodologies were employed during the conduct of this audit, such as:  Interviews/discussions with management and staff from PPSB at NHQ and from specialized call centres;  On-site observation at regional call centres;  Comprehensive review and analysis of relevant documentation; and  File review and analysis of quality assurance reports and related action plans.

2

Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

2.0

AUDIT FINDINGS

2.1

A governance framework is in place and is working well

Overall, roles and responsibilities are clear and accountabilities are well defined. A performance management strategy, comprised of a suite of indicators related to the network’s speed of service, is in place. Frequent updates on these indicators and performance against them are provided. However, it is not clear how these indicators and related performance data align with the network priority of first contact resolution. Analysis Roles and Responsibilities Since 2010, the specialized call centre network has undergone two key organizational changes. The first has been the implementation of the Service Management Structural Model (SMSM) – see Appendix B for call centre SMSM organizational structure. The second organizational change to the network has been the implementation of a co-managed model which delegated more management responsibilities to the regions. Under the co-managed model, EHSMs became accountable for a broader spectrum of call centre management responsibilities. Overall, interviews with call centre management and staff indicated that roles and responsibilities were clear and understood. However, they also identified uncertainty about the role of the call centre Business Expertise Consultant (BEC). The audit team noted that roles and responsibilities for the BEC varied amongst regions. In some instances, the BEC was seen as a technical expert who provided advice to the Business Expertise Advisor (BEA), whereas in others, the BEC was seen as a generalist that was primarily concerned with business enhancements, process improvements and trends analysis. Management in the regions reported that this inconsistency has been a cause of confusion or uncertainty about the BEC’s role in call centres. There may be a need to review existing roles of the BEC in order to improve consistency and clarity of the position. On the other hand, the TL’s role is consistent across the network. TLs’ range of responsibilities include taking CSOs attendance, monitoring indicators, providing feedback to CSOs, conducting coaching sessions based on BEAs’ quality assurance reports and handling call escalations. The audit team noted that TLs are not required to have program specific knowledge and this poses certain challenges. For example, while conducting feedback sessions with CSOs based

Internal Audit Services Branch, HRSDC

3

Audit of Management of Call Centre Operations

on the monitoring reports completed by BEAs, TLs are playing an intermediary role between the CSO and the BEA. The TL might have to liaise with the BEA to obtain advice and clarification which creates a duplication of efforts. There may be a value added opportunity to equip TLs with the skills needed to better support CSOs and advance the Department’s vision of first contact resolution. In November 2011, the CCD developed and disseminated a draft Accountability Framework and Performance Measurement Strategy to the regions. The objective of this framework was to identify and distinguish NHQ and regional call centre accountabilities within the new reporting structure. The strategy clearly outlined network targets as well as regional and national accountabilities for workload management, budget, human resources, quality monitoring and the day-to-day management of the network. Interviews with regional management and staff indicated that communication is in place and has been facilitated through various national calls such as the National Operations Committee and Call Centre Managers conference call. Performance Management Strategy The CCD has developed a comprehensive suite of key performance indicators which are related to the network’s speed of service. These indicators, and performance against them, are systematically provided to the regions, often on a daily basis. However, the audit team could not find clear alignment between these indicators and the current mandate and priorities for the network as defined in the 2011–2014 PPSB National Integrated Plan and the 2012–2013 Report on Plans and Priorities. One of the Department’s priorities for the call centre network focuses on first contact resolution whereby citizen’s needs are fully resolved at the call centre level. The CCD provides the regions with a daily report on different performance measures. Data included in these reports are forecasted call volume compared to number of answered calls, service level (percentage), average speed of answer, abandoned calls (percentage), average handle time, vacation and other leave (percentage). In discussions with management, the audit team was informed that first contact resolution of transactions is in fact currently measured, but these results are not published. Regional call centres are also provided reports on performance against expected call handling capacity, i.e. what is deemed to be achievable with their current level of funding. Management informed the audit team that these reports facilitate discussions with the CCD about improvements to a call centre’s performance.

4

Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

The current service level target is to have 80% of callers speak with a live agent within 180 seconds. The audit team reviewed monthly performance trends dating from April 2009 to March 2012 and noted that the network has not met this service target in the three fiscal years reviewed. The fiscal year average for CPP, OAS and EI call centres can be seen in the chart below. Average service level performance by year (target 80%) Fiscal year

CPP-OAS call centres

EI call centres

2009–2010 2010–2011 2011–2012

67% 37% 58%

42% 51% 32%

Regional management indicated that meeting current performance targets, in particular service levels, poses a major challenge for the network. Reasons cited include insufficient levels of resourcing, high staff turnover in some call centres resulting in less experienced CSOs, and increased complexity of the calls received. EI call centres have seen an increased number of calls as a result of the backlogs in processing EI claims. CPP/OAS is also seeing an increasing client base every year as more Canadians reach retirement. 2.2

An effective risk management approach is in place

Call centre operations are supported by risk management principles and practices. A workload management strategy communicates call handling capacity and comprises a number of tactics to address call volume. A central process for scheduling of CSOs is in place to support both service delivery and people management. Analysis Risk Management Risks are being identified at the branch level as part of the Regional and National Integrated Business Planning Processes. However, Regional Integrated Business Plans do not necessarily identify risks at the call centre level. Although, call centres have not documented their own risk registers, the audit team noted a variety of measures that CCD has implemented to mitigate operational risks. Minor opportunities for improvement have already been brought to management’s attention during debrief meetings. The audit team also reviewed Business Continuity Plans at the regional and CCD levels and found they include detailed action plans in the event of a number of scenarios.

Internal Audit Services Branch, HRSDC

5

Audit of Management of Call Centre Operations

Workload Management The CCD applies a Resource Determination Model to determine the appropriate mix of resources to meet forecasted demand and uses the draft accountability framework to establish what is achievable within the network’s funding envelope. The CCD, in consultation with the regions, has developed additional strategies to manage the network’s workload, i.e. having a mix of part-time and full-time resources to build latent call handling capacity into the network. While call centres are not meeting their current service levels, workload management strategies are in place to best manage national demand and workforce availability within the current network’s funding envelope. Scheduling of Call Centre Resources A central process is in place for scheduling CSO resources that takes into account information about planned off-phone activities, planned vacation time, and incorporates breaks, lunch, and time to review any procedural updates. This process strives to create schedules that will optimally serve the network. The CCD has also developed a leave management strategy with the goal of promoting planning at the regional level and recognizing the needed flexibility by call centre management teams. 2.3

Controls over quality assurance could be strengthened

There is a training regime in place for newly hired CSOs as well as seasonal refresher training. To support them in their role, agents access a web-based reference tool which is also used to communicate updates or changes to procedures. Our interviews with agents indicated that they would prefer more inclass training sessions to validate their understanding of updated or new procedures. Call centres have a NQAP in place to monitor calls and generate reports for coaching sessions. The audit team identified instances where the NQAP guidelines were not consistently applied or followed. Taking into account the interview and audit results as well as the way the NQAP is currently carried out, it is not clear to what extent the quality assurance program provides relevant and useful information to achieve consistency and improve client service.

6

Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

Analysis Training Training is a shared responsibility between NHQ and the regions. The CCD is responsible for coordinating the development of national program training while the regions are responsible for delivering program and legislative training. The New Hire Call Centre Curriculum (NHCCC) is a training program that teaches agents technical and soft skills to deliver quality service to citizens. Staff indicated that the NHCCC training packages adequately prepare agents for their duties at the call centres. In addition to the initial training, “refresher” sessions are provided, for example annual T4 and Guaranteed Income Supplement training is provided to OAS/CPP agents. Staff expressed that they would benefit from additional training sessions such as “current issues” and “soft skills”. To support them in their day-to-day functions, call center agents access the Common Reference Tool (CRT), a web-based reference tool that includes policies, procedures and other information related to EI and CPP/OAS in a structured manner. The CCD is responsible for maintaining and updating the CRT. Training material is increasingly being provided to agents in short memos as opposed to holding traditional in-class settings. It is expected that agents assimilate this information and any updates to the CRT during specifically allotted reading periods totaling one hour per week. Agents expressed that it can be difficult to go through and ensure proper understanding of the material received. Majority of agents interviewed said they would prefer more in-class or group training to better assimilate the material. In addition to the CRT, call centres agents have access to a regional Business Expertise support line manned by BEAs when they require assistance. Most regions have taken the initiative of independently developing tracking tools to collect information about the use of their support line with the goal of analyzing the information for trends and training needs. At the time of this audit, this regionally collected information was not available to the CCD for analysis. We were informed that NHQ is currently working with the regions to implement a national support line for agents.

Internal Audit Services Branch, HRSDC

7

Audit of Management of Call Centre Operations

Quality Assurance In 2008, the CCD implemented the NQAP with the objective of achieving consistency and improving client service. The program is effectively a live call monitoring regime. This program was revised in 2010 following observations and comments gathered during the evaluation phase. The NQAP requires each agent to have five calls monitored per quarter. Calls are evaluated by BEAs using a scorecard of 26 technical and soft skills criteria. The criteria cover all technical aspects of the call such as the greetings and authentication, fact finding, resolving needs, promotion of services and the closing. In addition, soft skills aspects such as an appropriate tone, vocabulary, rate of speech, and demonstration of confidence are evaluated. A monitoring report containing general observations on the agent’s strengths and areas for improvement is written by a BEA. Within one week of monitoring, the report is to be discussed between the agent and his or her TL in a coaching session. The audit team reviewed a total of 156 monitoring reports from across the four call centres visited. The results are as follows:  For 37 reports (23.7%), the coaching session with the TL took place later than one week. The range in which sessions were late was two weeks to two months following the monitoring of calls.  In 22 instances (14.1%), there was no record of the coaching session being conducted by the TL. Management indicated that the NQAP has been designed to serve as a coaching tool. However, staff at call centres confirmed that the feedback received from monitoring reports is not always useful from their perspective since the small number of calls monitored during a quarter (versus the number of calls answered) may not provide a representative view of their performance. Additionally, these monitored calls are known to occur on the same day, at times consecutively. The audit team is of the opinion that this methodology may not provide a realistic depiction of the agent’s performance. The NQAP guidelines provide for a standard of quality and consistency. Taking into account the interview and audit results as well as the way the NQAP is currently carried out, it is not clear to what extent the quality assurance program provides relevant and useful information to achieve consistency and improve client service.

8

Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

Recommendation 1. To ensure that relevant and useful information is used in its NQAP, PPSB should: a. Implement a call monitoring methodology that will reflect a reasonable distribution of monitored calls over time and consider operational changes, e.g. recent updates to procedures may call for increased monitoring. b. Ensure timely and complete coaching sessions between CSOs and TLs are conducted and documented.

3.0

CONCLUSION

Overall, a management control framework supporting the EI and CPP/OAS call centres is in place and applied consistently across all call centre operations. However, the audit found areas for improvement related to quality assurance.

4.0

STATEMENT OF ASSURANCE

In our professional judgement, sufficient and appropriate audit procedures were performed and evidence gathered to support the accuracy of the conclusions reached and contained in this report. The conclusions were based on observations and analyses of the situations as they existed at the time against the audit criteria. The conclusions are applicable only for the Audit of the Management of Call Centre Operations. The evidence was gathered in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing.

Internal Audit Services Branch, HRSDC

9

Audit of Management of Call Centre Operations

APPENDIX A: Audit Criteria The conclusions reached for each of the audit criteria were developed according to the following definitions. Numerical Conclusion on Categorization Audit Criterion 1

Significant Improvements Required

Definition of Conclusion Requires significant improvements (at least one of the following three criteria need to be met):   

financial adjustments material to line item or area or to the Department; or control deficiencies represent serious exposure; or major deficiencies in overall control structure.

Note: Every audit criterion that is categorized as a “1” must be immediately disclosed to the Chief Audit Executive and the client Director General or higher level for corrective action. 2

Moderate Issues Has moderate issues requiring management focus (at least one of the following two criteria need to be met): 



3

Controlled

 

4

Well Controlled

 

10

Internal Audit Services Branch, HRSDC

control weaknesses, but exposure is limited because likelihood of risk occurring is not high; control weaknesses, but exposure is limited because impact of the risk is not high. well managed, but minor improvements are needed; and effective. well managed, no material weaknesses noted; and effective.

Audit of Management of Call Centre Operations

The following table outlines the audit criteria and examples of key evidence and/or observations noted which were analyzed and against which conclusions were drawn. In cases where significant improvements (1) and/or moderate issues (2) were observed, these were reported in the audit report. It is expected that:

Conclusion

A defined and communicated governance framework is in place to support call centre operations, outlining the decisionmaking, delivery mechanisms, roles, responsibilities and accountabilities.

3

A documented approach to risk identification, assessment and formal response is in place and being used to support call centre operations.

3

Observations/Examples of Key Evidence 

Overall roles and responsibilities are clear and understood with the exception of the BEC.



Accountabilities are documented in the draft Accountability Framework and Performance Measurement Strategy. These accountabilities have been communicated to regional management.



Key performance indicators have been developed and performance is regularly communicated to call centre management. However, the alignment between these indicators and the current mandate and focus on first contact resolution is not clear.



Risk management is embedded in the roles and responsibilities of management within call centres and CCD.



Within the CCD, the Intraday unit is ensuring that the network’s call volumes are monitored in real time to identify any issues quickly and take actions appropriately.



Call centre operations have implemented a number of strategies that mitigate risk. However, there is no documented risk assessment (risk registers) at the regional call centres or CCD level that presents risks specific to call centre operations.



Business Continuity Plans for call centres as well as supporting systems are developed and contain the appropriate amount of detail.



Strategies are in place to best manage national demand and workforce availability of the call centre network within the current network’s funding envelope.



The scheduling of resources is done in an appropriate and effective manner and strives to support good people management.

Internal Audit Services Branch, HRSDC

11

Audit of Management of Call Centre Operations

Appropriate controls are in place to facilitate provision of timely, accurate and consistent services to citizens.

2



There are policies, procedures and supporting systems in place to support call centre operations. o CSOs would benefit from improvements to the search functionality of the CRT and/or its structural layout.



In-class training initiatives are developed and implemented appropriately.



Procedures and guidelines are in place to ensure security and protection of personal information.



A quality assurance program is in place and monitoring activities are conducted. o The NQAP guidelines provide for a standard of quality and consistency. o Sample tests show that follow-up activities, namely coaching sessions and action plans, could be strengthened. o Given the resource intense nature of NQAP, call centres may benefit from an approach that is more focused on operational changes and reasonable distribution of calls to monitor calls.

12

Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

APPENDIX B: Call Centre SMSM organizational structure

Internal Audit Services Branch, HRSDC

13

Audit of Management of Call Centre Operations

APPENDIX C: Glossary BEA BEC CCD CPP CRT CSO EHSM EI HRSDC NHCCC NHQ NQAP OAS PPSB SMSM TL

14

Business Expertise Advisor Business Expertise Consultant Call Centre Directorate Canada Pension Plan Common Reference Tool Citizen Service Officer Executive Head, Service Management Employment Insurance Human Resources and Skills Development Canada New Hire Call Centre Curriculum National Headquarters National Quality Assurance Program Old Age Security Processing and Payment Services Branch Service Management Structural Model Team Leader

Internal Audit Services Branch, HRSDC