Once the mini-grid is operational, it’s important to measure the project’s impacts. Mini-grids have (1) direct impacts, like increased access to electricity, and (2) indirect impacts, such as improved livelihoods. A well-designed impact evaluation measures both types of impacts, accounts for external factors and assesses how well a project benefits all members of the community equally. There is varied level of rigor that can be included in an impact evaluation, and many impact evaluations can include counterfactual studies.
Donors, government agencies, investors, communities and/or project managers may require impact evaluations. The project developer or an external evaluator can conduct the evaluation.
Key Steps: Designing an Impact Evaluation
- Identify the impact to be measured, the indicators that will measure the impact and the assessment criteria.
- Identify who will participate in the evaluation process. Include all relevant stakeholders.
- Create a participatory evaluation process with a plan for making the results publicly available.
Further Explanation of Key Points
Measuring Direct and Indirect Impacts
Project evaluations should measure both direct and indirect impacts. A mini-grid’s direct impacts might include increased access to electricity, increased use of electricity, improved energy services, new productive uses of energy and decreased fuel costs. Indirect impacts might include economic growth, increased incomes and improved health.
It’s not always possible to measure an impact directly, so evaluators use indicators, measurable variables that serve as proxies for desired results. To evaluate improved access to electricity, for example, the developer might use indicators such as the number of new household connections, increase in kWh of energy consumed or percentage of very poor households that receive energy services. Indicators of the quality of energy services might include number of hours of available electricity or number and duration of outages.
Measuring indirect impacts is more difficult. The evaluation must control for other variables that could influence results. For example, individuals in the sample who own land might increase their incomes by using electricity in agriculture. The mini-grid project made the increase in income possible, but some of the increase was due to land resources, not increased access to electricity. So evaluators would need to control for the variable of land use to measure the mini-grid’s impact on income from agriculture.
At the beginning of the project, developers need to collect baseline data, the starting point for each indicator. During project assessment, the evaluator measures the indicators again and compares the values to the baseline data. The difference between the two measurements indicates the impact of the project.
Impact Evaluation Techniques: Experimental Designs
Impact evaluations are identified by the type of methods used to generate the counterfactual analysis. Impact evaluations that use formal experimental designs yield the most accurate results. In an evaluation with an experimental design, the evaluator randomly selects a sample of individuals from a community and randomly assigns half of the individuals to a treatment group and half to a control group. For a mini-grid project, the treatment group would receive the energy services while the control group would not. To measure the project’s impact, the evaluator compares the benefits gained by the treatment group against those gained by the control group.
Experimental designs have drawbacks. First, they’re expensive, because they require skilled evaluators to design the experiment, collect accurate data and evaluate results. Measuring impact with an experimental design may increase project costs for mini-grid developers. Perhaps more importantly, many development practitioners would consider it unethical to provide energy services to some members of the community but deny them to others simply to measure the project’s impact more accurately. The control group may be reluctant to participate since they would not receive energy services.
Impact Evaluation Techniques: Non-experimental Designs
Non-experimental impact evaluation techniques such as cross-sectional comparisons and panel designs focus on comparing intervention groups before and after implementing an intervention. Though not as rigorous as counterfactual studies, non-experimental approaches are simple, practical and less expensive ways to assess the impact of mini-grid projects. The developer can collect individual data throughout the life of the project and compare it to aggregate socio-economic or energy data from local development authorities or national agencies.
Cross-sectional comparisons measure impact through individual surveys at the end of project implementation. For mini-grid projects, cross-sectional comparisons can show how broader project outcomes—like increasing a community’s total energy supply—impact individuals. Surveys can identify the type and quality of energy services an individual household receives. Evaluators also collect data about improvements to health, education and income.
Panel designs also measure impact on the individual level, but evaluators measure impacts before, during and after project implementation. Evaluators engage directly with individuals, such as holding a face-to-face meeting, to determine how their livelihoods change as a result of energy services. By directly engaging with individuals over time, evaluators can identify and control for the impacts of other development projects, leading to a more accurate picture of a mini-grid's impact.
Equity in Impact Assessments
To ensure equity, the mini-grid developer needs to assess a project’s impact on different members of the community, including women and other marginalized groups. The project’s monitoring and evaluation system should track impacts by gender, income and other socio-economic factors. Otherwise, developers might not realize that some groups are benefitting significantly more than others.
For example, a general impact evaluation might identify increased economic activity as a result of new energy services. A gender-focused analysis, however, might reveal that only men are benefiting from that economic growth. The project might be providing energy services for productive uses across the community, but if women don’t have the financial resources or access to credit to use that energy, they will not benefit equally from the project.
While monitoring project impact, the mini-grid developer must watch for unintended consequences. Mini-grid projects might create or exacerbate inequalities in a community. For example, in an area where income levels vary widely, a uniform tariff might limit electricity access to the wealthier members of the community. Families that couldn’t afford the tariff would not gain access to energy.
Challenges in Impact Evaluation
The impacts of energy projects on development goals can be difficult to measure. Energy is not usually an end in itself, but rather an input into other sectors. The chain of impact from the energy service to livelihood improvements can be long, convoluted and difficult to track. Access to electricity impacts many different sectors.
In addition, the benefits of energy projects may take years to emerge. Communities might see the benefits of energy services after the project itself has ended. To measure real impact, an evaluation might have to continue long after the life of the project.
Furthermore, to achieve a desired impact, an energy project might depend on developments in other sectors. For example, increasing energy services can spur economic growth, but only if the local economy has raw materials, machinery for production, financing for economic activities and markets. Project developers must understand how energy is linked to these factors in order to assess a project’s economic development impact.
Adams, S. et. al. (2006). A Guide to Monitoring and Evaluation of Energy Projects.
This resource is a guide for developing monitoring and evaluation programs for a wide range of energy services, including grid-based and off-grid electricity and improved biomass cookstoves.
Energy Sector Management Assistance Program (ESMAP) (2017). Gender: Social Inclusion in the Energy Sector, Online Resources.
ESMAP’s online resources provide tools for mainstreaming gender considerations into energy sector activities. The resources include sample questionnaires and checklists.
USAID (2016). Evaluation Policy.
USAID’s Evaluation Policy communicates USAID’s approach to evaluation, including the purposes of evaluation, the types of evaluations that are required and recommended, and the approach for conducting, disseminating, and using evaluations.
USAID. Evaluation Toolkit.
USAID’s Evaluation Toolkit curates the latest USAID guidance, tools and templates for initiating, planning, managing and learning from evaluations.