Statistical Quality Control and the Operational Risk Management Framework Agostinho, António1, Miguel, Alexandra2, Raminhos, Manuela3 1

Banco de Portugal, [email protected] 2

3

Banco de Portugal, [email protected]

Banco de Portugal, [email protected]

Abstract One of the main purposes of the Statistics Department of the Banco de Portugal is to ensure a statistical production with high quality standards aiming at fully meeting user’s needs, in particular those of the ECB, the Eurostat and other national and international organizations. Statistical quality control is chiefly concerned in making sure that several procedures and working arrangements are in place to provide for effective and efficient statistical processes, to minimise the risk of errors or weaknesses in procedures or systems or in source material.

Operational risk

management (ORM) is the entire process of continuously and systematically identifying, analysing, responding to, reporting on and monitoring operational risks. The statistical quality control and the ORM in the framework of national central banks’ statistical activities are “two sides of the same coin”: one focuses on the means to deal with the risks of errors, data mismatching and information’ gaps (by controlling) and the other focuses on the objectives (by managing risks). A definition of statistical quality, as well as the various possibilities of assessing it, is also described, focussing specially on statistical quality reports. Keywords: statistical quality control, operational risk management, quality assessment Disclaimer: The analyses, opinions and findings expressed in this paper reflect the personal views of each of the authors, which are not necessarily those of the Banco de Portugal or of the Eurosystem.

1

1. The Banco de Portugal experience in the field of Statistical Quality Control The Banco de Portugal’s strategy to further enhance the overall quality management procedures along all the phases of statistical compilation and dissemination processes underwent a restructuration of its organization. To cope with the rising demand of resources emerging from statistical production requirements, it was decided to centralize the statistical function in one specific Department of the Banco de Portugal, the Statistics Department, fully committed to statistical processes (conception, development, compilation and dissemination), which centralized the entire statistical function, including its dissemination, and, in the beginning of 2004 by creating a special Unit - the Statistics Audit Unit (SAU) - an innovation in the ESCB, covering the following main functions: •

Data quality and legal provisions’ full compliance assessment

•

Audit operations to the statistics produced in the Statistics Department

•

Support to the Head of the Department in the field of Internal/ECB Audits.

Besides all the internal quality control procedures that are implemented in each statistic compilation domain of the responsibility of the Statistics Department, one of the ways to address Statistical Quality Control is through the development of statistics audit operations. In this context, it should be referred that, to assess the quality of the statistical procedures in place, the quality dimensions, as laid down in the IMF Data Quality Assessment Framework (DQAF), are used. This framework provides a flexible structure for statistical quality assessment along the different phases of the statistical process from data collecting through processing up to dissemination. At European level, a reference should also be made to the European Statistics Code of Practice by European Statistical System and (ESS) to the Public commitment on European statistics by the European System of Central Banks (ESCB). Once analysed and evaluated the different phases of statistical production procedures, the organizational and functioning aspects and the efficiency of the procedures in place, SAU issues, when it applies suggestions / recommendations on current practices or suggests new procedures and/or organizational arrangements.

2

Moreover, after each statistics audit operation SAU defines a considerable set of structured quality indicators, to be compiled on a regular basis (generally on a yearly basis), on the quality of the statistical production performing regularly Statistical Quality Reports. These indicators are conditional to the specific nature of each statistic and to the corresponding milestones in its statistical production process and are systematized and presented to the compilation unit and to the Head of the Statistics Department under the form of a “Statistical Audit Report” and a “Statistical Quality Report” whose recommendations and suggestions’ investigations will further consolidate the management indicators that are currently used to monitor the quality of the statistical compilation processes in the Banco de Portugal.

2. Operational Risk Management (ORM) ORM is an integral part of the ESCB/Eurosystem governance and management processes. It is part of the overall risk management within the ESCB/Eurosystem. The objectives of ORM in the ESCB/Eurosystem are to better manage uncertainty and to enable a better-informed decision-making process with regard to risk management. ORM does not cover the management of financial risks (i.e. credit and market risks). Operational risk management is the entire process of continuously and systematically identifying, analysing, responding to, reporting on and monitoring operational risks. Within the scope of this policy operational risk is defined as the risk of negative financial, business and/or reputational impacts resulting from inadequate or failed internal governance and business processes, people, systems, or from external events. In order to facilitate a harmonised risk assessment and response and consistent reporting, a common language, i.e. the operational risk taxonomy, and a common risk tolerance policy should be used by all the NCB. The operational risk management process comprises the following steps: •

risk identification – Identification and record of all the risks related to all of the tasks, processes and objectives 3

•

risk assessment - The risk assessment considers both impact and likelihood as well as existing controls/control objectives as a basis for determining how the risks should be managed

•

risk response - Different risk response strategies can be applied to a specific risk: risks can be avoided, mitigated, transferred, or accepted

•

risk reporting – It aims to present an overview of the risk situation at a given point in time

•

and monitoring - It is an ongoing process that continuously checks the status of the key operational risks and related controls/ control objectives, verifies that they remain in line with the operational risk tolerance policy, ensures that action plans are implemented according to agreed schedules, scans the business environment and best practices to detect emerging new operational risks and define control objectives, and ensures that incidents are proactively monitored and reported.

The ESCB/Eurosystem risk taxonomy, which provides a clear and common language for all operational risks, is based on three interlinked components: root causes, risk events and risk impacts. Each risk event may have multiple root causes and impacts. •

The taxonomy of root causes distinguishes four categories of level 1 (people, governance and business processes, systems and external events) complying with the operational risk definition

•

The risk events taxonomy is divided in seven categories (errors or failures, infrastructure disruption, occupational incidents, frauds, disasters, attacks, other events)

•

The ESCB/Eurosystem ORM policy distinguishes three types of impact: business, financial and reputational impact.

The ORM assessment exercises imply the conduction of the following steps, in a first stage:

4

•

Step 1: identify the tasks and processes

•

Step 2: conduct a criticality assessment of each task and process, and

•

Step 3: conduct a "quick scan" analysis.

Only high-level processes should be identified, i.e. those leading to ultimate deliverables. The criticality assessment is mainly a “one-off exercise” that is to be updated in case that new processes are implemented and/or existing processes are subject to major changes. The criticality assessment aims at prioritisation of tasks and processes to provide focus in subsequent risk assessments. This is achieved by categorizing and ranking the processes by determining in a structured manner the risk impact in terms of the achievement of business objectives and/or financial and/or reputation if one or several of the worst case scenarios would materialise. The "quick scan" is performed on the ESCB/Eurosystem processes assessed as most critical according to the criticality assessment mentioned above.

3. Statistical Quality Control versus Operational Risk Management The operational risk associated with the statistical function was identified as being basically: •

the risk of reputational and impact image (credibility and confidence) damages to the Central Bank; and/or

•

the inability to achieve its business objectives (satisfaction of the users’ statistics needs and fully compliance with statistical commitments towards national and international organizations).

In keeping with ORM line of action, the Statistics Department of Banco de Portugal has identified and characterized, in a first step, seven high level processes, being the first five directly involved in the statistical compilation processes and the last two ones with its dissemination and a special service related with an administrative database storing credit-related information: 5

•

Monetary an financial statistics

•

Balance of Payments statistics and international investment position statistics

•

National Financial Accounts

•

Securities statistics

•

Non-financial corporation’s statistics from the central balance-sheet database

•

Statistics Dissemination

•

Central Credit Register (CCR)

For each of the first five statistical compilation processes identified in the Statistics Department five sub processes were identified: i) Methodological design; ii) Collection of raw statistical data; iii) Quality control; iv) Data processing; and, v) Analysis of the results for each of the process outputs. For de statistics dissemination process three additional sub processes were also identified: i) Statistical information repositories; ii) Consistency analysis of the results; and, iii) Dissemination of the statistical outputs. The Central Credit Register process is an administrative database storing credit-related information supplied by the participants (financial institutions that grant credit) on an individual basis, for their assessment of the risks attached to granting/extending credit. For this process four additional sub processes were also identified. In a second step Banco de Portugal has conducted a criticality assessment considering the worst case scenarios categorized and ranked for each task and process, identifying the worst possible scenarios for each one. The scenarios considered have been measured according the risk of negative financial, business and/or reputational impacts. Banco de Portugal identified for each one what would be the worst consequence, considering that controls have failed, which are abbreviated presented as follows: •

Integrity => loss of quality information and correctness of the output

•

Confidentiality => intentional or unintentional use or dissemination of raw data

6

•

Availability => total or partial unavailability of statistical compilation or dissemination

•

Internal events => intentional or unintentional use or dissemination of data with loss of confidentiality

External events were considered unlikely in the statistical compilation or dissemination context. Moreover, for each phase of the cycle of evolution of each process (sub-processes) the potential risks have been identified, namely: •

Errors or failures in internal and/or external data reporting with impact on loss of quality information and correctness of the output

•

Errors or failures in the quality requirements of data reporting with impact on loss of quality information and correctness of the output

•

Human errors or failures in compilation, processing, validation, analysis and delivery of output with impact on loss of quality information

•

Lack of confidentiality due to intentional or unintentional use or dissemination of elementary data

•

Deliberate acts manipulating, occulting or damaging elementary confidential data

•

Errors or failures in the communication, reporting, validation, exploratory analysis and delivery of output with impact on loss of quality information

•

Lack of skilled staff or expertise

•

Natural catastrophes

On the national level, for each of the mentioned potential risks, were identified: i) its descriptions, ii) its possible causes, and iii) the respective mitigation factors. However implicit in the different phases of statistical production procedures, and as part of the compilation’ cycle, a set of quality control procedures and working 7

arrangements are already implemented covering all the phases of statistical production process – data collecting, data processing and analysis and statistical dissemination – which contribute to the mitigation of the several risks identified and mentioned above.

4. Work ahead in the field of ORM applied to statistics The statistical quality control and the ORM in the framework of national central banks’ statistical activities are “two sides of the same coin”: one focuses on the means to deal with the risks of errors, data mismatching and information’ gaps (by controlling risks) and the other focuses on the objectives, identifying, analysing, responding to, reporting on and monitoring operational risks (by managing risks). To enhance the quality statistical control, the Statistics Department, in particular the statistical audit unit follows the approach of: •

Identify all statistical processes and tasks that may be subject to statistical auditing

•

Use the Self Assessment Questionnaire (SAQ) – first step of a statistic audit operation - to assess the respective processes and tasks

•

Identify the risk events to be considered in the ORM (from the above information along with other sources).

•

Identify the risk treatments/mitigation (issuing recommendations namely in the Statistical Audit and Statistical Quality Reports that are realized on an yearly basis following each of the statistical audit operation performed by SAU, sharing the good practices among the various units)

•

Monitor the risk (assessment of the implementation of the recommendations included in the Statistical Audit Reports through regular follow-ups).

According to Banco de Portugal’ experience in Statistical Quality Control some common fields were found with ORM. In fact, a deep collaboration has been revealed, so far, with the statistics auditors. Statistics auditors were former statistics compilers (they know the business) what provides them the capacity of enable an efficient analysis 8

of quantitative information and of the efficiency of the procedures in place. Finally, statistics auditors and statistics compilers are under the same management (ensuring a consistent leadership).

9

References [1] Agostinho, A. & Valério, M. J. (2007), Statistics Audit: the experience of Banco de Portugal [2] Agostinho, A. & Valério, M. J. (2008), Statistical Quality Control: the experience of the Banco de Portugal [3] Suplemento ao Boletim Estatístico do Banco de Portugal (Janeiro 2012) A Gestão da Qualidade nas Estatísticas do Banco de Portugal [4] ESCB/Eurosystem (2008) Operational Risk Management Guiding Principles [5] ESCB/Eurosystem (2008) Operational Risk Management Policy

10