United Nations Development Programme

Papua New Guinea

TERMS OF REFERENCE INDEPENDENT EVALUATION OF PROVINCIAL CAPACITY BUILDING PROJECT

INTRODUCTION AND RATIONALE The Papua New Guinea “Organic Law on Provincial and Local Level Governments” (OLPLLG) was implemented in 1997, which essentially aims to devolve substantial financial management functions and responsibilities -- planning, budget and finance – to the sub-national level (Provincial, District, and Local Level Governments). In support to the OLPLLG, the Financial Management Improvement Program (FMIP) was developed to promote transparency and build capacities in financial management within and between the national, provincial and local-level governments of PNG. Specifically, it aims to achieve the following goals:

• • • • •

Sound fiscal management Effective decentralization of financial management Allocation of resources in accordance with government priorities Utilization of resources to achieve value for money Enhanced accountability and transparency

The FMIP is directly implemented by the Department of Finance and supported by the Asian Development Bank through the Financial Management Project (FMP), AusAID through the Provincial Financial Management Training Program (PFMTP) and the UNDP-AusAID, through the Support for Provincial Financial Management Training (SPFMT 2000-2003) and the Provincial Capacity Building Project (PCaB October 2004 – December 2006) . UNDP significantly supported the process of decentralization in PNG through the SPFMT which is a parallel and complementary project that provided follow-on support to the various financial training courses conducted under the AusAID component – Provincial Financial Management Training Program (PFMTP) from April 2000 – March 2006 and currently through the Advisory Support Facility’s Support to Financial Training Branch. Through the SPFMT, a number of experienced public sector UN Volunteers were deployed in several of PNG’s provincial treasury offices to provide hands-on coaching and mentoring to the staff to help them absorb the new functions and responsibilities as stipulated under the Decentralization Act of 1995. The project, which ended in December 2003 was deemed successful and was regarded as best practice. To build on and consolidate the achievements of the SPFMT, and to assist the Government of PNG (GoPNG) to further develop an integrated approach to strengthen financial management at the national and provincial level, UNDP implemented the intermediate Provincial Capacity Building Project (PCaB), which consists of three main components: 1. 2. 3.

Capacity Building for Provincial Training and Pilot Internship Phase ICT support for Provincial Financial Management and study mission IFMS Scoping and Design Mission of a full-scale decentralization intervention

UN House, Level 14, Deloitte Tower, Douglas Street, Port Moresby Postal Address: P.O.Box 1041 Port Moresby, Port Moresby, NCD, Papua New Guinea Telephone: (675) 321-2877, Fax: (675) 321-1224, E-mail: [email protected]

Components 1 and 2 focus on continued and enhanced capacity building measures and follow-up support to activities of SPFMT project phase I in order to ensure sustainability of these activities in the provinces, particularly through hands-on financial management training at provincial level, a pilot internship program and a provincial training officer network to help build national capacities. The results of this intermediate short-term and time-bound phase will feed into a scoping and design mission (Component 3) for a full-scale decentralization intervention to be fielded in the last quarter of the project. The project commenced in October 2004 and will end in December 2006. The Project was extended from April 2006 to December 2006 to allow for sufficient time to consolidate project outputs but also to allow for more time for the donors to scope and agree on their future support to FMIP and their respective roles in this collaborative support. Additionally, AusAID provided cost sharing for the PCaB project to take over Provincial Support Advisors (PSAs) from the PFMTP project which ended in March 2006 and the newly created positions to assist with the IFMS implementation - the IFMS Support Advisors (IFMS SAs). OBJECTIVES OF THE EVALUATION As part of UNDP’s overall governance programme and in order to guide the design of a more long-term decentralization support programme, UNDP in partnership with AusAID, ADB, and the Department of Finance (Executing Agency), wish to undertake a comprehensive evaluation of previous governance initiatives (i.e. SPFMT and PCaB) which will include a review of the project design, implementation strategy, sustainability, issues and constraints, results and achievements, good practices, lessons learned and recommendations for possible future programme focus, including its management and institutional implementation framework. The PCaB project is a component of the overall FMIP Programme and is closely linked to other donor projects therefore the project strategy needs to be revisited and if necessary revised to take into account the developments of these other components. In addition, given that the project has attracted significant UNDP and donor contributions, there is a need for an in depth evaluation of its performance. The main stakeholders of the evaluation are the Government of Papua New Guinea, Donors (ADB and AusAID) and the Provincial Treasurers. This evaluation will provide the UNDP CO and its partners with a clear analysis of the project progress and key lessons learned, and strategic recommendations that could enhance UNDP’s programme in supporting Government’s efforts to implement the “Organic Law on Provincial and Local-level Governments”. The analysis and recommendations presented by the project evaluation will provide the basis for further development of UNDP’s governance programme in Papua New Guinea and should therefore include forward looking recommendations on the exact scope of UNDP’s support to FMIP from 2007 onwards. 1.

Within this context, there are five primary objectives of the evaluation:

1) To evaluate the project's achievements against the project’s original objectives in the project document. In doing so the evaluation will: a)

Evaluate the outcomes/results so far and approach used to achieve them against the objectives/outputs and indicators of performance included in the original design or as adjusted in annual work plans as well as operational matters and counterpart funding, etc.

2

b) c) d) e)

Evaluate the capacity and structure of supervisory and advisory management in place, including the delivery of inputs in terms of quality, quantity and timeliness To evaluate whether the recommendations of the project TPR have been implemented and their impact on the project with respect to outputs/outcomes Evaluate the progress in transferring capacity locally as well as the involvement of the direct beneficiaries in the original project design and annual work plan development. Evaluate achievements with respect to the objectives of the FMIP project

2) To evaluate whether the project has contributed to the broader development outcomes. In doing so the following issues need to be considered: a) b) c)

Have the outcomes been achieved, and if they have not, has progress been made towards their achievement? An analysis of the underlying factors beyond the project's control that influence the achievement of development outcome. Has the partnership strategy for achieving these outcomes been effective/appropriate?

3) To review the current context of the project vis-a-vis the context in which it was originally signed. Issues to be addressed include: a) Evaluate whether the project strategy is still relevant. b) Evaluate the continuing relevance of the project (approach, objectives, modalities of implementation, etc.) with regard to the prevailing context. c) Review the implementation strategy adopted to date for the project and compare with the project strategy given in the document. In particular, review the criteria used in the selection of activities and the degree to which and success with which they have been used by the project. 4) To identify linkages and synergies to other programmes, projects and activities. Issues to be addressed include: a) b)

Review and identify similarities and synergies with FMIP, PFMTP (AusAID), and FMP (ADB) component and identify possibilities to fine-tune the PCaB project with regard to the other project components. Propose synergised or complementary activities to save cost and to guaranty for more efficiency in project delivery and sustainability.

5) To formulate a concept paper on the scope of a possible a second phase project. This work will take into account: a) b) c) d) e) f) g) h)

A review of the findings of objectives 1 to 4 of the terms of reference. The progress made so far. The recent developments of the FMIP Programme and donor support (AusAID, ADB and others) Increased attention to results based planning and budgeting in the revised project strategy that reflects tailored national MDG targets and indicators to ensure that UNDP supports the achievement of MDGs The views of a wide range of potential stakeholders Consultations with the UNDP country office on the United Nations Country Programme and new country programme processes to adjust the new project strategy accordingly Using a comprehensive and consultative process in the production of the logical framework and implementation schedule. Identify clear and measurable benchmarks for future monitoring and evaluation

3

SCOPE OF THE EVALUATION The evaluation mission will be fielded for two weeks, beginning September 2006, to review previous and ongoing governance initiatives supported by UNDP, with the end in view of designing a full-scale decentralization intervention in support to the GoPNG’s development priorities. The evaluation will draw on experiences and lessons learned from SPFMT Phase I and the intermediate PCaB Project. The mission will place special emphasis on exploring the involvement of UNVs and PSAs (and other international and national volunteer organizations) in the full-scale intervention as this modality in combination with the hands-on mentoring approach has already been identified as ‘best practice’ during the implementation of SPFMT project. The mission is expected to perform the following duties: 1.

Assess the overall performance of the project with reference to its respective strategies and objectives; suitability and efficiency of implementation arrangements including monitoring and evaluation, organizational and management structure; contribution to capacity building; role and effectiveness of UNVs and PSAs, inputs, outputs, and indicators as stated in the respective project document;

2.

Review the partnership and cooperation between the stakeholders as well as links with the national and local governments; examine the project’s collaboration and engagement with civil society and local communities; review links to other UNDP programmes, UN agencies and donor organizations;

3.

Evaluate the project outcomes in terms of relevance and effectiveness in light of the emerging country context, priorities and development strategies; how the project addressed local challenges, its main achievements and overall impact, sustainability of project achievements, as well as the remaining structural gaps not addressed by the project. The evaluation and design process will focus specifically on the progress made by the existing project and recent developments in the FMIP Programme and in particular the PFMTP component. Where there are linkages and synergies (at the policy level, on resource mobilisation and substantive areas such as gender, governance, and strategic partnerships) to other projects, programmes or activities these need to be highlighted and the implications for the activities of the project be addressed.

4.

Identify good practices and initiatives that worked well or that can be potentially applied to other projects/developed upon in the upcoming phase, as well as gaps, weaknesses and initiatives that should be avoided in future;

5.

Identify key and significant lessons for donors (UNDP, AusAID, ADB) and the PNGGO, that can be drawn from the experience of the project and its results (in terms of design, strategy, sustainability, impact, results and achievements);

6.

Provide clear analysis and concrete recommendations on how the evaluation results can contribute to the design of a subsequent multi-donor decentralization project in PNG.

The scope of the work should be as comprehensive as possible, independent and include, but not be limited to, the following: a)

Identify lessons learned, both from the project being evaluated and other projects in the FMIP Programme and if appropriate other projects implemented in PNG.

4

b) Assess the geographical spread and take careful account of the gender dimensions of the project objectives and achievements in involving and meeting the needs of all key stakeholders, both men and women. c) Review the management structure called for in the project document in the light of the present political context. In particular review the possible role to be played by provincial and district organisations so that any future structures to be established will be relevant to the political and social context now prevailing in PNG. d) Evaluate the management structure established under the project, both with a view to determining its ability to achieve the project’s objectives and its effectiveness in ensuring that the activities implemented by the project will result in sustainable outcomes. e) Review the approach to planning taken by the project management and compare with that given in the project document to determine the effectiveness of the former and the appropriateness of the latter. f) Assess the role of donors in the project, with particular reference to their specific requirements for use of the funding they wish to provide. g) Assess the level of funding which can be expected to be available to the project in the immediate future (both Donor and GoPNG counterpart funding). In particular, the mission will determine what funding can be made available for project staffing and logistic support costs such as residential accommodation, travel and communication costs. h) Prepare a Phase II project design or adjust the existing project design in accordance with the requirements of UNDP, based on the UNDP Programme Manual and advice provided by the UNDP PNG Country Office i) Involve stakeholders in logical framework preparation of Phase II design (new document if required) to build local ownership and to test assumptions.

PRODUCTS EXPECTED FROM THE EVALUATION An Evaluation Report with findings and above mentioned assessment areas, recommendations, lessons learned and best practices, as described in the points under scope of the evaluation. This will also include an “action item” list or description of best practices in a certain area or in the appropriate niche for UNDP interventions in a specific programme country. There should be a strong emphasis on the design of a future project. A Concept Paper on the scope of a possible phase II intervention The mission findings and recommendations shall be thoroughly discussed with the UNDP Country Office and the project management. On the basis of comments provided by UNDP and the project management, the mission shall finalize the reporting documents. Finally, the consultants shall share the report findings at a Project Steering Committee Meeting, organized by PCaB. EVALUATION TEAM The mission team will be composed of six members. The team leader will be an independent consultant. The other members will comprise of one representative of GoPNG (DoF), the UNDP PNG Governance Programme Officer and a governance specialist from the UNDP-Pacific Sub Regional Center (PSRC). A representative from AusAID and a representative from ADB will also be invited to join the evaluation team.

5

All duties of the team will be carried out in close collaboration with the UNDP Country Office as well as PCaB. METHODOLOGY OR EVALUATION APPROACH 1.

Documentation Review (desk study) will include the review of: • Common Country Assessment/UN Development Assessment Framework (CCA/UNDAF), UNDP Country Programme Outline, SPFMT and PCaB Project Documents, SPFMT Evaluation Report, Annual Progress Reports (APR), Quarterly Progress Reports, Internship Programme Report, Audit Reports, etc.

2.

Consultations/meetings with UNDP, the stakeholders of the project, relevant partners (local and national government agencies, counterpart institutions, donor community, civil society) including but not limited to the following: • GoPNG concerned line departments at national level (Department of National Planning and Monitoring and Department of Finance) • University of Papua New Guinea, Pacific Adventist University and PNG Institute of Public Administration • International donor community - UNDP, ADB, AusAID, EU, World Bank

3.

Field visits will be conducted with the following: • Provincial Administrations and Treasury Offices of Milne Bay, Morobe, and Central

Action Initial visit of the evaluation team leader to the UNDP country office in Papua New Guinea to plan the details of the evaluation mission Documentation review (desk study) Field visits Evaluation mission Drafting of the evaluation report Debriefing Preparation of final report TOTAL

Duration 2 Days 1 Day 5 Days 2 Days 2 Days 1 Day 1 Day 14 Days

In order to meet this deadline, the team leader would be required to undertake document preparation after working hours, including weekends. IMPLEMENTATION ARRANGEMENTS The mission will be fielded for two weeks, beginning in September 2006. PCaB will be in charge of preparing a briefing package for the members as well as organizing all required meetings and interviews with the various parties concerned. Field visits will be scheduled by PCaB to ensure the process of consultation with partners and key stakeholders. The team leader’s contract, Daily Subsistence Allowance (DSA) and international return ticket shall be met by UNDP PNG. All local travel and related costs for the evaluation mission shall be met by PCaB. QUALIFICATIONS AND EXPERIENCE OF INTERNATIONAL CONSULTANT, TEAM LEADER

6

The international consultant/team leader is expected to be an acknowledged authority on the matters related to local governance, decentralization and capacity building. He/she must have solid experience in programme evaluation as well as formulation, recognized for his/her understanding and appreciation of the situation with regard to development issues, and respected for his/her expertise and opinion on the subject. He/she has the ultimate responsibility for delivery of the inputs and for the ultimate success of the mission, but can only achieve this through the collective effort of all team members. The candidate must have a strong understanding of the PNG context, particularly in relation to the government’s decentralization/local governance strategy. The Team Leader will be expected to work full time on this task for the duration of the mission. The competencies and skills of the candidate will include: Education

• Postgraduate degree in the relevant field (management, political science, economics, international relations, public administration)

Experience

• Minimum of ten (10) years of extensive relevant experience in developing strategies and projects in the area of governance/decentralization.

• Team Leader experience in the evaluation of technical assistance projects, preferably with UNDP or other UN development agencies and/or major donors. • Experience in leading multi-disciplinary teams to deliver quality products in high stress, short deadline situations. • Proven capacity for working across different sectors and institutional levels from policy and decision making, to administration and management, local government and communities. Competencies

• • • •

Competencies

• • • •

Language

Excellent writing, communication and interpersonal skills and ability to operate in multi-disciplinary environment. Objectivity, impartiality and maturity, and judgment necessary to deal with delicate situations. Demonstrated ability to assess complex situations in order to succinctly and clearly distil critical issues and draw forward looking conclusions. An ability to assess institutional capacity and incentives, and formulate proposals for strengthening and improvement. Strategic planning skills and experience in programme formulation. Excellent facilitation skills in group discussions. Demonstrated understanding of gender analysis and its applications to implementation of projects Commitment to inclusive and participatory approaches to delivery of policy advice and technical assistance Previous work experience in PNG and knowledge of the country will be an asset.

• Proficiency in English is compulsory, knowledge of PNG local language an asset.

7