On 08th June 2022, a series of webinars was held ahead of the Evaluation Cooperation Group (ECG)* meeting of the Heads of Evaluation of the independent evaluation offices.- The Evaluator General Karen Rot-Münstermann, represented IDEV. These webinars were organized by and for the staff of evaluation offices, to promote knowledge, experience-sharing, and networking among peers. The European Bank for Reconstruction and Development (EBRD), the World Bank Independent Evaluation (IEG), the Asian Development Bank, and the African Development Bank Independent Evaluation (IDEV) all organized sessions.
IDEV organized a panel discussion to share its experience in using robust design methodology to measure the impact of the Last Mile Connectivity Project (LMCP) Phase 1 project in Kenya.
Measurement of the impact of energy interventions is difficult and often requires a creative approach because of methodological challenges. The choice of an impact evaluation of this project was motivated by the innovative design of the Kenyan project and the Bank’s plan to scale up similar interventions, increasing energy access in other Regional Member Countries.
IDEV Division Manager Rufael Fassil guided the evaluators tasked with the impact evaluation. He informed the audience that there were very few impact evaluations of capital-intensive infrastructure operations such as this one, and very few impact evaluations in general undertaken in the African continent. Conventional impact evaluation approaches would not have sufficed in tackling this large infrastructure project. The evaluation called for creative solutions to assess core areas of investment about what works or does not work, how things work, for whom, and why.
IDEV evaluators Racky Balde and Eustace Uzor then explained how they put together a creative mixed approach. Given the population served and the parameters identified, the challenge was to ensure their sampling methodology was robust, that the evaluation accurately covered the relevant project area, and had the most representative sample of beneficiaries. Racky Balde described the deployment of a spatial sampling approach using a high-resolution settlement layer map of Kenya from the Centre for International Earth Science Information Network (CIESIN) – Columbia University in collaboration with the Facebook Connectivity Lab and Digital Globe. To select households for evaluation purposes, Google Earth was used to validate the list of treated and control households based on a design rule of a 600-meter radius around a transformer. The methodology adopted was a Fuzzy Regression Discontinuity Design. It is a two-stage least square regression model based on the probability of being eligible in the area. A qualitative analysis with triangulation complemented it.
The evaluators presented the findings, some contradicting the expected outcomes and impact. Evidence had shown, for example, that consumption had increased, whereas mean income in the project area had not increased. An important lesson for the Bank in its future projects for last-mile connectivity is associating electricity access projects with income-generating investments. The evaluation also pointed out that while the subsidy reduces the electricity cost, the electricity affordability remains an issue for households connected by this project.
The session moderator, Jacqueline Nyagahima, then pursued the learning process with a question-and-answers session, leading to informal discussions. Regarding the choice of evaluations and their timing, Rufael Fassil explained that quality, time, and cost, are 3-way trade-offs in a work program, since evaluations have to be timely to be useful. On the necessary time to evaluate impact after the completion of a project, the evaluators concurred that contrary to health sector investments, impact from an electrification project could be done relatively soon after completion since it involved electricity connection numbers in a given area.
The event concluded that this pilot initiative would certainly be useful to inform future electricity access projects of the Bank because it had helped the Bank understand what has worked and what has not. It will also serve to improve the performance of the next phases of the project in Kenya and other ongoing Bank-financed electricity access programs in Africa.
* The Evaluation Cooperation Group (ECG) was established in 1996 to promote a more harmonized approach to evaluation methodology, following an assessment of the five major MDBs (the African Development Bank, the Asian Development Bank, the European Bank for Reconstruction and Development, the Inter-American Development Bank and the World Bank).
The presentation from the webinar is available here.