Evaluating Health Systems Presentation to the IOM Committee on Planning the Evaluation of PEPFAR

Lola Dare, CEO, CHESTRAD

1

The Current Situation Outline • • • • •

Key Messages Health Systems – What are they? Situation of Health Systems in Africa African HSS Leaning Sites Performance Management Findings: Ondo State Demonstration HSS Learning Site (2004 – 200) • Enduring Issues and Opportunities • Concluding Thoughts: IOM Evaluation of PEPFAR

2

Main Messages • Health Systems are diverse, complex and very context specific, mutli-method and approaches will be required • Country case studies needs to reflect this diversity and complexity • Building blocks or sub-systems provide a platform for identifying commonalities within this diversity • Sub-systems act in synergy and interact, evaluation should examine the effect of GHI investments (Including PEPFAR) on this interaction to achieve disease/issue specific outcomes

Health Systems: What are they?

4

Health Systems – What are they? • Diverse – – – –

Epidemiological Social Determinants Economic Capacity Political economy

• Complex – – – –

Building blocks or sub-system Diseases, issues National, sub-national, district, community Back office or front office

5

Health Systems – What are they?

• Suffer from the M-dilemma – Multi– More Approach – Management

• Challenged in many countries • Weak, fragmented and uncoordinated in Africa

6

Situation of Health Systems in Africa

7

Financing FlowsSituation: in disarray The Current Revenue (Sources)

(Collection & Pooling)

1Revenue

Managers

Overseas Development Assistance (Loans, grants & in-kind donations)

National Planning Commission

Federal Medical Centres

Budgets (Allocations, derivation, MDG appropriations)

Federal Ministry of Finance

Federal teaching and specialized hospitals

Firms and Employers (Taxes, VAT etc)

National Health Insurance Scheme

Federal Min. Of Health Parastatals

Individuals (Fee for service)

State Ministries of Finance

State Min. of Health

Donations (Cash and Kind)

Local Government Finance Departments

S. Hosp. Mgt. Boards

Health Facilities

LGA PHC Departments

(Allocation & Purchasing)

Public Sector Facilities

Providers

Private for profit facilities

Community Based providers

NGOs Providers

Faith Based Providers

1Some of these revenue managers (Federal medical centers, federal teaching and specialized hospitals) are also providers and are included in public sector facilities)

Source: CHESTRAD, 2008.

HMOs

Human Resources: Balancing Health Worker Ratios WHO Region

% of Total Health Workforce

Ratio

Health Service Provider

Health Mgt and Support

Africa

83

17

4.9

Eastern Mediterranean

75

25

3.0

South East Asia

67

33

2.0

Western Pacific

78

23

3.4

Europe

69

31

2.7

Americas

57

43

1.3

The World

67

33

2.0 9

Adapted from WHR, 2006

Quality: Standards, Accreditation and Certification (13 Core Areas of Compliance) •

Accreditation Status

Mean Compliance % by Facility Type Basic Health Center

Comprehensive Health Center

Total

Non Accreditation

71.9

76.9

72.3

Remedial Accreditation

21.6

23.1

21.7

Provisional Accreditation

6.5

0.0

6.0

Total

153

13

166 10

Management, Leadership and Governance: Lessons from the Ondo State Demonstration Learning Site

• 225 facilities offering primary, secondary and tertiary care and no trained health administrator • Facility and sector administration often led by clinicians- principally doctors and nurses who are trained to heal and care but not to manage • The ‘special cadre’ in the health workforce: Is this the doctor? Skills mix and leadership tensions • HRH planning more of trade unionism than of performance and service delivery

• Outcome indicators: Performance or reducing mortality and morbidity for all at all costs • Balancing political reality in the technical and management debate Lessons from the Ondo State Demonstration Learning Site

Integration: Within and Between

Tertiary Support

1st Referral

• Unclear structure functions & linkages • Systematic analysis of needs & response capacity inadequate • Mainly relate on priority disease or programs

Health Center (1st Contact)

Home/Community Based Care

12

Coordination: The Current Situation

13

African HSS Learning Sites

14

Improving outcome from specific causes - Health System efforts at different PHC levels Level of Care

VPDs

EoC

Malaria

TB

HIV/AIDS

2nd Referral

+

+++

+++

+

+

+

++

++

++

++

+++

++

++

+++

++

++

+

+

++

++

(Tertiary)

1st Referral (Secondary)

1st Contact (Primary)

Home based (Community)

Code

Maximum

+++

Medium

++

Minimum

+

The African HSS Learning Sites Program Operational Research sites to: -- Explore the impact of a basic package of health systems interventions, implemented in synergy, on health outcomes

Where are the Learning Sites? Early start up sites

ETHIOPIA

-- Emphasise the MDG priority burdens including malaria, obstetric care, HIV/AIDS and immunization

--

Test innovations, promote knowledge and learning on the interaction between improvement in human resources, expanded financing and health systems performance.

CAMEROON UGANDA `

ONDO STATE (A demonstration Learning Site)

ZAMBIA ``

S. AFRICA

16

Evaluation Question • A basic package of systems intervention will strengthen district health systems performance and improve health outcomes. • The Basic Package of HSS interventions – human resources – Financing – quality management – access to drugs and medical supplies, management and leadership – communication and advocacy – performance management 17

Evaluation Framework

Financing Options (Budgets, loans, grants, community contributions)

Actions from related sectors Finance Public Administration

Human Resources

Infrastructure & Equipment

Drugs, Supplies & Quality Mgt

Education

Health Systems Output (Equity, Efficiency & Quality) Agriculture Environment

Outcome (

Health Status,

Mortality & Morbidity)

P E R F O R M A N C E M A N A G E M E N T & E V A L U A T I O N 18

Performance Management Findings: Ondo State Demonstration HSS Learning site (2004 - 2008)

Outcome: Organization and Infrastructure Improved tier organization and integration (Ondo State, Nigeria)

Supervision

Tertiary SSH,1

RH, 3

Secondary DH, 6

Cottage Hosp, 9

Primary

Comprehensive HC, 18

Basic Health Centre, 203

Referral

Community Home Based, 4.1m

20

Outcome: Supplies and logistics chain management • Essential drug list updated • Decentralized facility based management of the DRF • Private sector-led procurement and distribution • State regulation of quality and pricing • Elimination of stock out of essential drugs • A Public Private Partnership

21

Outcome: Human Resources (Total Workforce by Sector and Year 2003-2007) Sector

Total Workforce by Year

2003 – 2006

2003

2004

2005

2006

Change

% change

Health

2,876

3,072

3,041

3,662

786

27.3

Education

27,345

27,438

29,137

30,390

3,045

11.1

Agriculture

2,939

2,713

1,821

1,814

-1,125

(38.3)

Finance

1,531

1,728

1,030

1,028

-503

(32.9)

World, Lands & Housing

2,160

1,488

1,110

1,109

-1,051

(48.7)

136

150

164

171

35

25.7

General Administration

2,009

1,329

1,105

1,065

-944

(47.0)

Parastatals (Utility Boards)

3,004

2,522

1,996

1,785

- 1,219

(40.6)

Others

1,527

1,508

1,075

1,166

-361

(23.6)

Total Workforce by Sector

43,527

41,948

41,479

41,190

- 1,337

(3.1)

Justice

Overall attrition 3.1%; social sector HR gains (Health 27.3%; Education 11.1%) 22

Outcome: Financing Government Health Expenditure per year including HHR in US$ millions (2003 – 2007)

23

Early Impact: General Outpatient Utilization (Simple Malaria) • All facilities increase - 178% (increase) • Secondary facilities - 216% increase • Tertiary facilities -- 20% reduction (? appropriate utilization) 24

Sustainability Assessment (Assessment Score: 1=None 2= Insignificant; 3=Moderate; 4=High; 5 = Significant)

Stakeholder and performance management database assessment of 5 domains Domain

Financing QIM

PMP

PM

HR

Diagnostics

Domain Assessment

Implementation Status

4

3

5

3

3

2

3.3

Institutional Home

5

3

5

4

4

3

4.0

Technical Skills & Competence

3

3

3

3

3

2

2.8

Managerial Capacity

3

2

3

2

3

2

2.5

Financial Capacity

3

2

3

2

2

2

2.3

Ownership

4

3

4

2

3

2

3.0

3.7

2.7

3.8

2.7

3.5

2.2

3.4

Sub-System Assessment

Interpretation of Assessment: 1,2 = Not sustainable; 3= Sustainable at current levels but scale up capacity limited 25 4 = Sustainable and scaleable with some external support; 5 = Sustainable with minimal external support

Enduring Issues and Opportunities

26

Enduring Issues for HSS evaluation I • Purpose – Achievements, Effect, attribution, Impact and Accountability • Methodology – Effect, Attribution, Contribution or Impact – Operational or rigorous – Study design considerations • Timing of the evaluation – For the outcome to reasonably occur – Moving train evaluation • Tools – DHS/ DCA – WHO District Health System Assessment Surveys – African HSS Learning Sites – TGF Model Impact Platforms

27

Enduring Issues for HSS evaluation II

• Indicators Selection – Tracking of progress (Attribution)

– Monitoring of achievements (Attribution) – Outcome/Impact indicators (Contribution and Accountability)

• Application – Generation of knowledge and learning – Performance improvement/management • Capacity for conducting evaluation – Infrastructure : ICT – Skills: Design and application – Change in information management behavior

28

Enduring Issues for HSS evaluation III • Investment in Evaluation: Data and information management systems – Programme M&E budget – Country Information systems: district, sub-national and national • Coordination – Design and Tools – Data collection – Dissemination especially across evaluation projects within a large programme – Ownership • Open Access to the dataset for secondary analysis 29

Concluding Thoughts

30

Opportunities or Threats? • • • • • • • • • •

Model Evaluation Platform (TGF) IHP+ Evaluation Framework Country Health Sector Surveillance Systems (CHeSS) Joint Assessment of National Strategies and Plans Joint Platform for HSS Funding Harmonization for Health in Africa Performance Based Funding (The Global Fund) Results Based Financing (The World Bank) Institute for Progressive Financing (UK-Gov Proposal) Global Health Initiative (USG Proposal) 31

Concluding Thoughts I Considerations for the Evaluation of PEPFAR

• The evaluation questions – Health Systems Effects of PEPFAR investments – Harnessing opportunities for HSS in PEPFAR countries drawing from the lessons of PEPFAR 1 authorization • Evaluation Framework – Investment/Interaction across HSS subsystems (Input and Process) – Specific output linked to programme goals and targets (Attribute) – Disease or Issue Specific Output (Contribute) – Impact (Contribution) 32

Concluding Thoughts II Considerations for the Evaluation of PEPFAR

• Country Case Study section to reflect – Reflect the diversity, complexity and context • Indicator Selection – Building blocks/sub- systems approach linked to disease/issue specific outcomes and impact

33

[email protected]; [email protected]