REPCar Regional Coordination Unit Caribbean Environment Programme United Nations Environment Programme 14-20 Port Royal Street, Kingston, Jamaica Tel: (876) 922-9267 to 9 • Fax: (876) 922-9292 E-mail: [email protected]
TERMS OF REFERENCE of the MID-TERM REVIEW 1. BACKGROUND Project rationale Agriculture is critical to the economies of the countries of the South-western Caribbean region. Increasing world demand for cash crops and the growth in competition for a share of global markets have resulted in significantly increased pesticide use in the sub-region. When not properly used, agricultural pesticides pose an environmental threat. The contamination with highly toxic or persistent pesticides is associated with environmental risks and health hazards, both to humans and the coastal environment and its associated coastal economies. With these problems in mind, UNEP’s Caribbean Environment Program is working, with the support of the Global Environment Facility GEF, to reduce runoff of agricultural pesticides into the Caribbean Sea, through a project with Nicaragua, Costa Rica and Colombia. The main objective of the project is to mitigate the degradation of the marine environment caused by the use of pesticides in Colombia, Costa Rica and Nicaragua, implementing comprehensive management practices and specific measures to control the use and application of pesticides in the agricultural sector. This will further assist countries in meeting their obligations under Annex IV of the Land Based Sources of Marine Pollution Protocol of the Cartagena Convention for the Protection and Development of the Wider Caribbean Region. Project history Considering the precautionary principle for approaches to marine environmental protection and resource conservation, the project idea was developed in 1998. Subsequently, a PDF-B was developed in 2000, producing National Reports on the status of pesticide use and management in the participating countries. One of the key issues identified in the PDF-B phase was the benefit associated with demonstration projects of best practices and training in the areas of alternative technologies or Good Agricultural Practices (GAP). Based on the National Reports, a Regional Report was prepared and a full size project was developed and submitted to GEF in 2001. The project was endorsed by the governments of
Colombia, Costa Rica and Nicaragua and approved by the GEF council in 2002. However, the project commenced at the end of 2006 and is projected to be completed by December 2010. Executing Arrangements The project is a co-operative effort of national, regional and local stakeholders. The main stakeholders for this project are farmers, agrochemical distributors, health, agricultural and environmental ministries and agencies, environmental non-governmental organizations (NGOs) and other community-based organizations, relevant international organizations, and academic institutions. These stakeholders are represented on National Coordinating Committees (NCC) and Regional Project Steering Committee (PSC) meetings. The NCCs and the National Coordinators liaise directly with the demonstration projects on Good Agricultural Practices, the participants in the coastal monitoring programme and other national implementing agencies. The Project Steering Committee (PSC) is the regional primary decision-making body, guiding the overall execution of the project. The National Coordinating Committees (NCC) are represented at the PSC by the National Coordinator (representing the Ministry of Environment) and a delegate from the Ministry of Agriculture. Both ensure proper liaison between the PSC and the National Committees. The overall execution of the project and coordination at the regional level is the responsibility of UNEP’s Caribbean Regional Coordinating Unit (UNEP-CAR/RCU), located in Kingston, Jamaica. UNEP-CAR/RCU is the secretariat to the Caribbean Environment Programme (CEP) and for the Convention for the Protection and Development of the Marine Environment for the Wider Caribbean Region (Cartagena Convention). UNEP-CAR/RCU is the Executing Agency of the project. The project is implemented by UNEP’s Division of Global Environment Facility Coordination (UNEP-DGEF). The Executing Agency, UNEP-CAR/RCU, maintains regular contact with the POPs (Persistent Organic Pollutants) Unit of DGEF. Project components and activities There are 4 activity components of the project: a) Working with affected stakeholders, each country is promoting Good Agricultural Practices (GAP) in the specific sectors that contribute the most to pesticide runoff through the implementation of a series of demonstration projects. With the participation of farmers, these projects monitor the environmental, social and economic impacts of different management practices and provide educational activities for identified stakeholders. b) A regional coastal monitoring programme is supported by the project to monitor pesticide runoff into the coastal environment through pesticide residue analysis of selected matrices. The capacity of some targeted labs is increased in this component, which will
2 | 11
provide the basis for long-term monitoring by academic and oceanographic institutions in the region. c) A series of activities are undertaken to institutionalize improved pesticide management and to strengthen the capacity for reducing pesticide runoff. National policy frameworks and regulatory systems for the use and control of pesticides will be reviewed and crop certification programmes will be supported. Information on the project’s activities and results will be disseminated by various means. d) UNEP-CAR/RCU is responsible for project execution, and is assisting the participating national authorities with coordinating the substantive part of activities at the national level. At the time of the mid-term evaluation, the project has achieved: (i) Establishment of project coordination structures at the regional and national levels; (ii) Selection of targeted crops and start-up of demo projects in the 3 project countries, as well as the establishment of a baseline on the status of pesticides use for the selected crops; (iii) Design of a regional coastal monitoring programme, completion of first monitoring assessment, and improvement of the capacity of participating research institutes; and (iv) Production of promotional materials, establishment of project website is running and an interactive map-based information platform is under development. Relevance to the GEF and POPs Activities This project was initially developed to support the “Contaminant-based” Operational Programme 10 and “…help demonstrate ways of overcoming barriers to the adoption of best practices that limit contamination of the International Waters environment…”. Since then POPs related activities have become part of the Operational Programme on POPs: reduce and eliminate production, use and release of POPs. The activities are also consistent with provisions of the Stockholm Convention on POPs. At the time of project implementation, Costa Rica and Nicaragua have ratified the Convention, while Colombia had signed and its ratification is currently in process. The project will also implement various provisions of the Cartagena Convention, specifically the Protocol Concerning Pollution from Land-Based Sources and Activities. At the time of the implementation of the project, Colombia and Costa Rica had only signed the Protocol although all three countries are signatories to the Cartagena Convention. Budget The total project budget is USD 9,920,000 of which USD 4,295,000 is from the GEF and USD 5,625,000 is co-financing from official and private entities in the participating countries and from CropLife LA. An important part of the co-financing is coming from the implementing agencies responsible for the demo projects and the coastal monitoring programme.
3 | 11
2. OBJECTIVE AND SCOPE OF THE REVIEW The objective of this mid-term review is to review and evaluate the implementation of planned project activities and outputs against actual results to date, and as far as possible establish the initial project impact (ref. objectives & outcomes as described in the logical framework, Annex 1), sustainability and execution performance. The focus will be on the following questions: a) Are the regional and national coordination mechanisms and the institutional arrangements adequate, effective and timely to coordinate project activities at the regional, national and local levels? b) Are there clear reasons for the changes made to the original project workplan, are the changes relevant and were these approved? c) Has the establishment of baseline information been appropriate and sufficient? d) Is the project approach to validating and demonstrating Good Agricultural Practices effective? Will these models serve as reference methods and contribute to improved use of pesticides in agriculture? e) Has the coastal monitoring programme been designed and implemented in such a way that the results will effectively contribute to the monitoring of potential environmental impacts? f) Are the stakeholders adequately involved in the development and implementation of project activities, particularly for the demonstration projects on GAP, relevant and sufficiently covering the interests of the projects? g) Will the project facilitate long-term sustainability after the completion of project activities, particularly at the local demonstration site level and for the areas of influence of the demo projects? h) What are the prospects for further collaboration with other GEF and non-GEF projects in the region and globally? The review will assess the following, among other things: • Delivered outputs: assessment of the project’s success in producing each of the programmed outputs to date, both in quantity and quality as well as usefulness and timeliness. • Project outcomes and impact: evaluation of the project’s success so far in achieving its outcomes. • Sustainability: analysis of the risks that are likely to affect the persistence of project outcomes in the short, medium and long term. • Execution performance: determination of effectiveness and efficiency of project management and supervision of project activities. The sustainability assessment should address financial sustainability, socio-political risks, and stakeholder ownership, national institutional framework and governance, and environmental risks. At this point in the implementation of the project, it is not expected that any of the project activities have been integrated, progressed and replicated. However, the extent of synergies created and potential for creating additional synergies with similar activities should be discussed.
4 | 11
In addition to evaluating the status of project implementation, recommendations should be made in support of improving project implementation, particularly, but not limited to, the following areas: • The structure and operational modalities of project coordination, especially at the national level, which should have stakeholder representation from both the public and private sector. Is the structure sustainable and how can it ensure active involvement and facilitate information flow among all stakeholders? • Sustainability of the project. How can positive impacts of demonstration projects be sustained? Can the demo site impacts be up-graded to cover other crops and areas? How can the coastal monitoring programme be sustained, and include other coastal areas and countries of the region? • Collaboration with other GEF or non-GEF projects within and outside the region. The evaluation should also give consideration to the processes that have affected the attainment of project results to date, such as: • Was the project properly prepared? Were objectives and components clear and feasible? Were the capacities of executing agency and country counterparts (Ministries of Environment) adequately considered when the project was designed? • Is the project in-line with the current priorities of the participating countries? Are the participating countries committed (both in terms of time and financially) to the project? • Has the project involved the relevant stakeholders through consultation or information-sharing during its preparation? • Were structural problems of the project adequately identified by the executing agency, were modifications proposed? • Has the project done appropriate financial planning and reporting? Has there been diligence in the management of funds and reporting? Has the co-financial contribution from the different project partners been at the expected level? • What were the reasons of delays in the implementation of the project and its components? What are the consequences? Have efforts been made to overcome these constraints? Furthermore, the evaluation should highlight lessons learnt where possible, both positive as well as negative, from the standpoint of the design and implementation of the project geared towards the demonstration and promotion of Good Agricultural Practices (GAP) and specific measures to improve the management for the use and application of pesticides in the agricultural sector. The evaluator will identify lessons learnt from the project, from the following perspectives: • Mechanisms to establish coordination at the national level, promoting ample stakeholder participation from public and private sectors. • The engagement of farmers, and how to change their attitudes and behaviour taking into consideration environmental and health impacts related with the use of pesticides. • Assessment of baseline information and strategies developed by the coastal monitoring programme to analyze and disseminate this information for improved decision making.
5 | 11
Information exchange and management, based on Internet and GIS techniques. Lessons for regional cooperation among Colombia, Costa Rica and Nicaragua.
The scope of the evaluation is similar to the one specified in “Global Environment Facility Guidelines for Implementing and Executing Agencies to conduct Terminal Evaluations, May 7, 2007” to evaluate the activities supported by GEF through this project. The evaluator should, during his review, assess the quality of the implementation of the Monitoring and Evaluation plan.
3. TERMS OF REFERENCE The evaluator will rate the overall implementation success of the project and provide individual ratings of design and implementation aspects: (a) Assess whether the project is appropriate in terms of addressing the need for reducing the runoff of pesticides to the Caribbean Sea. (b) Determine the continued relevance and appropriateness of the project to relevant GEF and GEF POPs activities, national and regional priorities and how the project complements UNEP’s global and regional mandates. (c) Assess whether the project document and work plan are adequate, clear and realistic to enable effective and efficient implementation; whether the project is executed according to the plan and how well the management is able to adapt to challenges during project execution to enable implementation. The evaluator should also establish how well the project has identified and managed its risks so far. (d) Determine progress made to date in meeting the project's objectives and planned outcomes, in particular: o Develop and promote a set of GAP protocols for some of the major crops of the Meso-Caribbean Basin (MCB) region that are environmentally sound, socially acceptable and economically feasible, and that are transferable to other similar parts of the region and world. o Increased capacity for the monitoring of pesticides runoff and information on pesticide residues in the coastal and marine environment of the Caribbean is made available to the wider public. o Improved awareness on the importance for preventing pollution of the coastal and marine environment from agricultural-based sources. (e) Review overall sustainability of the project and specifically review the extent of synergies created with other relevant projects. (f) Determine whether this type of project or components of the project, in particular with regards to training and technology transfer, has/have potential for being replicated, either in terms of expansion or extension. The evaluator shall also identify any step towards replication taken within the framework of the project and the relevance and feasibility of these steps.
6 | 11
Regarding the demo projects and coastal monitoring programme: (g) Determine the quality and comprehensiveness of terms of reference, proposals and protocols produced for the demo projects. (h) Review sustainability of the implementation of GAP, how the demo site impacts can be up-graded to other crops and areas. (i) Determine the quality and comprehensiveness of the terms of reference (scope), proposals and protocols produced for the coastal monitoring programme. (j) Review the sustainability of the coastal monitoring, how monitoring activities can be funded and continued after the lifetime of the project. (k) Review the composition and running of the advisory panels. (l) Assess the proposals for awareness raising and complementary training in GAP. Regarding Project Coordination: (m) Establish the level of involvement and effectiveness of the institutional arrangements. Review the working relationship and sharing of responsibilities between the National Coordination Committees and National Project Coordination Units and, the Regional Project Steering Committee and the Regional Project Coordination Unit. (n) Determine the level of participation of various stakeholder groups in the implementation of the project (i.e. relevant ministries, farmers’ organisations, pesticides industry, universities, etc.). (o) Assess the effectiveness of supervision and administrative and financial support provided by UNEP-CAR/RCU, UNEP-DGEF and GEF. (p) Assess the cost-effectiveness of the project activities and whether the activities are achieved within the planned and/or reasonable time and budget. (q) Analyse and compare pledged and actual co-financing obtained to date. Note: The evaluation should also include a breakdown on projected and actual costs and co-financing for the project (table attached in Annex 2 Financial Planning). (r) Review the adequacy of the developed M&E system. Review data and reported results of the Monitoring and Evaluation activities undertaken by implementing and executing agencies. In addition, assess the quality indicators identified in the Logical Framework. (s) Identify problems encountered during project implementation and lessons learned in the design and execution of this project. (t) Provide recommendations that may assist with re-direction of the project or reevaluating the project approach to improve delivery of expected outcomes and impacts. The ratings will be presented in the form of a table with brief justifications based on the findings of the main analysis.
7 | 11
4. METHODOLOGY AND RATING The mid-term review will be conducted as an in-depth evaluation using a participatory approach, whereby the UNEP-DGEF POPs task manager, staff of the Regional Coordination Unit (UNEP-CAR/RCU), the National Coordination Units (Ministries of Environment) and other relevant agencies are consulted throughout the evaluation. The consultant will also discuss with with the UNEP-DGEF POPs Task Manager on any logistic and/or methodological issues to properly conduct the review in as independent way as possible given the circumstances and resources offered. The findings of the evaluation will be based on: • Visits to Colombia, Costa Rica and Nicaragua: o Meetings with National Coordination Units (Ministries of Environment), o Meetings with other members of the NCC, o Field visits to demo projects and coastal monitoring sites, meetings with project coordinators. • Interviews with project personnel and staff of the Regional Coordination Unit (UNEPCAR/RCU) in Kingston, Jamaica. The visit is also aimed at obtaining copies of all relevant documents and project reports. • Additional interviews by phone with other stakeholders. • Desk review of the project document, outputs, financial and monitoring reports (such as the quarterly reports to UNEP, relevant correspondence, internal reports, if available. • Desk review of reports and minutes of meetings of the Project Steering and National Coordination Committees. The proposed schedule for the evaluator is as follows: March 9 2009
March 10-11, 2009 March 12, 2009
Medellín and Apartadó, Colombia Bogotá, Colombia
March 13-14, 2009
San José, Costa Rica
March 15-16, 2009
To be determined, Costa Rica San José, Costa Rica
March 17, 2009
Briefing with PCU, project manager. Meetings with NCU and members of NCC. Review relevant documentation. Meet AUGURA’s demo project staff, visit demo project farms. Meet/telephone conference with INVEMAR’s Coastal Monitoring Programme staff Meet/telephone conference other stakeholders Concluding meeting with NCU Meetings with NCU and members of NCC. Review relevant documentation. Meet demo project staff CORBANAPROAGROIN-BANACOL. Meet demo project staff, visit demo project farms. Meet staff Coastal Monitoring Programme CICACIMAR
8 | 11
March 18, 2008
March 19-20, 2009 March 21, 2009
Bluefields, Nicaragua Managua, Nicaragua
March 23-27, 2009
Meet/telephone conference other stakeholders Concluding meeting with NCU Meetings with NCU and members of NCC. Review relevant documentation Meet BICU’s demo project staff, visit demo project farms. Meet CIRA’s Coastal Monitoring Programme staff Meet/telephone conference other stakeholders Concluding meeting with NCU Meet CAR/RCU staff Review relevant documentation Preparation of the report.
The success of project implementation will be rated on a scale of 1 to 6, with 1 being the highest and 6 being the lowest. The following implementation aspects should be used for rating purposes: • Attainment of objectives and outcomes • Achievement of outputs • Cost-effectiveness • Stakeholders participation • Country ownership • Implementation approach • Financial planning • Monitoring and Evaluation Each of the items should be rated separately and then an overall rating given. The following rating system is to be applied: • HS = Highly Satisfactory • S = Satisfactory • MS = Moderately Satisfactory • MU = Moderately Unsatisfactory • U = Unsatisfactory • HU = Highly Unsatisfactory In addition, replicability and sustainability will be rated on the following 1 to 4 scale: • L = Likely • ML = Moderately Likely • MU = Moderately Unlikely • U = Unlikely
9 | 11
5. EVALUATION REPORT FORMAT The evaluation report shall be a detailed report of no more than 30 pages (excluding annexes), written in Spanish or English and then translated to the other language, and should include: i) An Executive Summary (no more than 3 pages) ii) Introduction and background iii) Objective, scope and methodology iv) Project Performance v) Conclusions and rating of project implementation success vi) Lessons learnt vii) Recommendations viii) Annexes The final report should be sent to the following person: Mr. Jan Betlem POPs Task Manager UNEP-DGEF P.O. Box 30552 Nairobi, Kenya Tel: (254) 20-7624607 Email: [email protected]
With a copy to: Mrs. Maryam Niamir-Fuller Director UNEP/Division of GEF Coordination P.O. Box 30552 Nairobi, Kenya Tel: (254) 20-7624165 Email: Mr. Nelson Andrade Coordinator UNEP-CAR/RCU 14-20, Port Royal St. Kingston, Jamaica Tel: (1876) 922-9267 to 9 Email: [email protected]
Mr. Segbedzi Norgbey, Chief, Evaluation and Oversight Unit UNEP, P.O. Box 30552 Nairobi, Kenya Tel.: (254) 20-7623740 Email:
10 | 11
6. RESOURCES AND SCHEDULE OF THE REVIEW The contract for this evaluation will begin on February 9, 2009 and end on April 30, 2009. The consultant will submit a draft report to UNEP DGEF Task Manager latest on April 10, 2009, with a copy to the Coordinator UNEP-CAR/RCU for initial comments. Comments on the final draft report will be sent to the consultant by April 15, 2009, after which the consultant will submit the final report no later than April 30, 2009. The total budget for the review is estimated at US$ 17,500. The cost for travel and accommodation (about US$ 10,000) will be adapted according to the final schedule and itinerary. The budget for the consultant fee is US$ 7,500. In accordance with UNEP policy, all UNEP projects are evaluated by an independent evaluator. For this Mid Term Review, the contract will be issued by UNEP-DGEF. The evaluator should not have been associated with the design and implementation of the project. The evaluator will work under the overall supervision of the UNEP DGEF Task Manager. The evaluator should have the following minimum qualifications: i) at least 10 years technical experience in the project-related areas; ii) expertise in environmental health-related issues; iii) experience with implementation and management of GEF projects, in particular demonstration projects; iv) experience in project evaluation; and v) experience in working in the region and excellent command of spoken and written English and Spanish.
7. SCHEDULE OF PAYMENT The evaluator will receive an initial payment of 60% of the total contract amount (enough to cover travel and accommodation costs) which is due upon signature of the contract. Final payment will be made upon satisfactory completion of work. The fee is inclusive of all expenses such as travel, accommodation and incidental expenses. In case, the evaluator cannot provide the products in accordance with the TORs, the timeframe agreed, or if the products are not of the required standard and quality, the payment to the evaluator could be withheld, until the products are modified to meet UNEP's standards. In case the evaluator fails to submit a satisfactory final product to UNEP, the product prepared by the evaluator may not constitute the evaluation report.
11 | 11