| Prospect | To get a qualitative understanding of the results of adaptation projects and actions. |
| Potential use of M&E findings | Supporting the management of an ongoing project and informing follow-on activities or similar projects. Gaining insights for reporting. |
| Description | Beneficiaries and stakeholders involved or affected by the project are being consulted about their views, experiences and perceptions. This may take the form of participatory workshops or one-on-one interviews. A questionnaire may be used or an open-ended discussion with guiding questions. Participants may be asked about their views about the project and its results, as well as about challenges and success factors or lessons learned. |
| Benefits and limitations | A qualitative approach offers the potential for in-depth insights into how and why the project has or has not been able to deliver results. It can serve as helpful complement to quantitative indicators. Main limitations are the subjective nature of the gathered information which could lead to false conclusions, particularly if only a small set of stakeholders are included. There may also be cultural or psychological factors influencing the responses, e.g. if beneficiaries hesitate to speak openly about project results. Therefore, care should be taken in the design and execution of the qualitative assessment. |
| Resources needed | Qualified interviewers. Know-how to develop the assessment details. Time and financial means to conduct the assessment. |
| Example from practice | UNDP applies a qualitative vulnerability reduction assessment in which beneficiaries are surveyed at the beginning and the end of an intervention. They are being asked to rate their level of vulnerability. Comparing the results to the baseline serves as a measure of project’s effectiveness. |
| Links | UNDP’s guide to vulnerability reduction assessment |
| Prospect | Continuously documenting the progress of an adaptation project and knowing whether its objectives are achieved. |
| Potential use of M&E findings | Supporting the management of an ongoing project and reporting its adaptation-specific results upon completion. |
| Description | During the design of a project, its objective, intended results and activities will be defined. A theory of change connects the various intended activities and results and illustrates how collectively they work towards the achievement of the objective (see Bours et al.,2014, for details). To be considered an adaptation project (or one with significant adaptation benefits), the theory of change should outline how the project contributes to adaptation. Indicators can then be formulated for various points in the theory of change. The indicators should be able to demonstrate how the project contributes to adaptation. |
| Benefits and limitations | The benefit of using a theory of change as planning tool is to systematically outline how a project will achieve its objectives. Its main advantage over other results management tools such as “logical frameworks” is its non-linearity, meaning that activities and results can be interrelated in a way that fits the complex realities of most sustainable development projects. Monitoring the indicators over time assists the project management. |
| Resources needed | Resources are mainly required for designing the project/programme and its indicators. Once operational, resources are mainly needed for data gathering and analysis. Since this takes place in close relation to project activities, the resources needed are seen as moderate. |
| Example from practice | • The GIZ guidebook “Adaptation made to measure” includes an example from an adaptation project in India. |
| Links | Adaptation made to measure. A guidebook to the design and results-based monitoring of climate change adaptation projects. (GIZ, 2013; also available in French and Spanish) Bours et al. (2014): Guidance note 3: Theory of Change approach to climate change adaptation programming CARE (2014): Participatory Monitoring, Evaluation, Reflection and Learning for Community-based Adaptation: A Revised Manual for Local Practitioners UKCIP (2011): AdaptME toolkit |
See specific purpose #7 “Assessing whether vulnerability has been reduced”
| Here, the term “impact evaluation” refers to so called “rigorous impact evaluations” that use experimental, quasi-experimental or similar designs. These designs enable robust conclusions about the causality of actions and results. | |
| Prospect | Understanding the impacts a project has had on the target group/system and how and why they occurred (learning). |
| Potential use of M&E findings | Informing the design and implementation of adaptation interventions and increasing the effectiveness of adaptation through transfer of lessons. Using evidence of results to advocate for resource allocation. |
| Description | Impact evaluations (IEs) aim to assess the difference a project or programme has made in comparison to not having had the intervention. IEs differ from regular project monitoring in important ways: (a) they aim to establish cause-and-effect relationships between actions and results in order to identify the actual contribution of a project to any observed progress, and (b) they go beyond the project indicators and also assess unforeseen impacts. To do so, IEs compare the effects of an intervention to other groups or areas without a similar intervention. Different techniques are employed based on, amongst others, the size and randomness of the treatment and control group (an overview of evaluation designs is provided by a GIZ guidebook from 2015). IEs are typically done after completion and by entities which were not directly involved in the project’s implementation. IEs can focus on individual, institutional or systemic impacts. |
| Benefits and limitations | Impact evaluations (IEs) provide a robust assessment of results combined with insights into how they were achieved. Understanding how and why projects were successful is the main benefit of IEs. Lessons learned can be used to inform and improve similar interventions. In addition, due to the rigour of their methods the results obtained are usually very robust. The main limitations of rigorous IEs are: (1) IE methods require certain conditions, e.g. a similar control group, which are not always possible. (2) IEs are very data and resource intensive. The high costs and data requirements are a key limiting factor of their application. (3) For IEs to determine results, time needs to have lapsed since the intervention took place. Moreover, the process of a participatory IE can take significant time, too. Consequently, results of IE may only be available a year or more after the project has ended. |
| Resources needed | Expertise for the design and implementation of the evaluation. Various types of reliable data about the intervention, the effects and the control groups. Due to the efforts required for a rigorous IE, significant financial resources are needed. |
| Example from practice | Relatively few rigorous IEs of outcome-oriented (in contrast to merely capacity building) adaptation projects have been conducted to date. • GIZ has commissioned an IE of an adaptation and biodiversity programme in Morocco. The participatory process at the beginning of the IE was described in a webinar on 5 October 2016. |
| Links | Impact Evaluation Guidebook for Climate Change Adaptation Projects by GIZ in collaboration with UNDP (2015) Experimental Project Designs in the Global Environment Facility (GEF) – an advisory document by the Scientific and Technical Advisory Panel of the GEF (2012) Bours et al. (2014): Design, monitoring, and evaluation in a changing climate: Lessons learned from agriculture and food security programme evaluations in Asia |
| Prospect | Knowing the quantified economic and health benefits achieved through an adaptation measure. |
| Potential use of M&E findings | Reporting achieved benefits of adaptation interventions and providing accountability based on quantitative figures. Advocating for resource allocation. |
| Description | The approach consists of multiple steps: (1) The boundaries of the geographical area and population need to be defined. (2) The baseline value of economic assets and the health profile of the population before the start of the intervention and their development over time need to be determined. (3) Damages from extreme events and other impacts related to climate change need to be measured. (4) Estimations of the damages that would have occurred without the adaptation measures need to be made. This may also need to take into account extreme events that did not occur during the lifetime of the assessment but could have happened based on expected climatic changes. These steps can be operationalized in different ways, with different methods and underlying assumptions. Health impacts can be quantified based on the concept of Disability-Adjusted Life Years (DALYs) which is used by the World Health Organization |
| Benefits and limitations | The key benefit of this M&E approach is the quantification of actual adaptation outcomes, i.e. the “success” of adaptation. An additional advantage is that this approach can also be used before implementation based on climate scenarios and on assumptions about the likely effects of adaptation measures on asset protection and health. There are two main limitations: (1) the approach is not applicable to projects or actions that predominantly focus on capacity building. It works best for projects and actions that directly reduce the impacts of climate extremes. It also requires a certain time span in order to detect outcomes, typically at least 3 years. (2) The approach is very data intensive and therefore costly to implement. |
| Resources needed | Expertise to design and apply the approach. Various types of reliable data about economic assets, health status, the intervention and climate change impacts are needed. Overall the approach is very data-intensive and requires significant financial resources. |
| Example from practice | Case study of the application of “Saved health, saved wealth” to a coastal protection project in Viet Nam including Excel calculations. |
| Links | Stadelmann, M. et al. (2014): Universal Metrics to Compare the Effectiveness of Climate Change Adaptation Projects. Handbook of Climate Change Adaptation. Springer Publishing. Open access as 2011 working paper Guide to apply the “Saved health, saved wealth” approach and Excel Tools for calculation |
