II. PERFORMANCE MONITORING AND EVALUATION:
HOW ARE INDICATORS USED?

>> This is USAID >> Intersectoral Partnerships >> ISP Handbooks

The ability to assess the impact of inter-sectoral partnering, as with any development result, depends upon two facets of development programming: performance monitoring and evaluation. Performance monitoring refers to the ongoing process of collecting and analyzing data to measure the performance of a program, process, or activity against expected results. To facilitate this, a defined set of indicators is constructed to track regularly the key aspects of performance. An evaluation, on the other hand, is a relatively structured, analytic effort undertaken periodically to answer specific questions regarding programs or activities. Evaluation focuses on why results are or are not being achieved; on unintended consequences; or on issues of interpretation, relevance, effectiveness, efficiency, impact, or sustainability. Performance indicators are key to both monitoring and evaluating intended results. An indicator is a particular characteristic or dimension used to measure intended changes. It is the quantitative or qualitative value of a single element within the concept that is being analyzed. Performance indicators are used to observe progress and to measure actual results against intended results. They are most often used for three general purposes: 1) to complete policy design or evaluation; 2) to enlighten an audience about some specific situation; and/or, 3) to aid in allocation decisions. They serve to answer "how" or "if" an activity is progressing toward its objective, rather than "why" or "why not" such progress is being made.2 Together, monitoring and evaluation are used to: ensure financial accountability build the capacity of customers and local stakeholders to reflect, analyze, and take action provide organizational credibility and garner public support develop lessons learned that can lead to corrective action by local stakeholders develop lessons learned that can help implementing agencies improve program implementation (UNDP 1997). USAID Guidance on Indicator Selection and Collection USAID, like other government agencies, is required to develop useful and meaningful performance goals and indicators consistent with the Government Performance Results Act.3 Performance information informs judgements, choices, and funding. Agencies use it to determine whether a program should continue to receive funding. Implementing agencies, customers, and other local stakeholders also benefit from the information. Performance information can be used, for example, by a local reformer to maintain public support for reform by demonstrating progress to date. USAID staff and partners are required to develop a performance monitoring plan that includes performance indicators and their definitions, data source and quality, method of data collection, frequency and schedule of data collection, responsibilities for acquiring data, and a data analysis plan. As described in Box 1, USAID guidance identifies four criteria for selecting quality performance indicators: direct, objective, practical, and adequate (USAID 1998a). Box 1: USAID Criteria for Selecting Quality Performance Indicators Direct: A performance indicator is direct (or valid) if it closely tracks the result it is intended to measure. Indicators should be widely accepted for use by specialists in a relevant subject area, exhibit readily understandable face validity, and/or be supported by a specific body of technical research. Indirect or proxy indicators can be used when other considerations, such as cost or timeliness, make that necessary. When proxies are used, particularly those that are not generally accepted or widely used, the relationship between the indicator and the result it is intended to measure should be researched and well understood. Objective: An indicator is objective if it is unambiguous about 1) what is being measured; and, 2) what data are being collected. Objective indicators have clear operational definitions that are independent of the person conducting the measurement -- that is, different individuals would collect data for an objective indicator using the same indicator definition. Practical: An indicator is practical if it is one for which data can be collected on a timely basis and at a reasonable cost. Performance indicators should 1) provide data to managers at a cost that is reasonable and appropriate, as compared with the management utility of the data; 2) have data available on a frequent enough basis to inform regular program management decisions; and, 3) have data available that are current enough to be useful in decision-making. Adequate: The number of indicators tracked for a given result should be the minimum necessary to ensure that progress toward the end result is sufficiently captured. USAID guidance also identifies three criteria for collecting quality performance data: validity -- the extent to which a measure actually represents what it is intended to measure; reliability -- the stability of a measurement process; and timeliness -- frequency and currency. Types of Indicators USAID staff and partners involved in a program must exercise great care in choosing between different types of indicators and ensuring that the indicators selected meet the above criteria. Inputs, outputs, and outcomes Input indicators measure what has gone into an activity, such as technology added, money spent, or services provided. They do not measure the quality of the services or other concerns such as access or adequacy. Output indicators are the measure of the actual impact or results of programs, such as the number of wells built or teachers trained. This distinction needs to be kept very clear so one does not assume output results by measuring inputs. Indicators measuring outcomes reflect the impact of the inputs and outputs. Outcomes are at a much higher level than inputs and outputs. For example, a program that allocates money (input) to train community leaders (output) may lead to greater social capital (outcome). Objective vs. Subjective As mentioned earlier, objective indicators record items that would be recorded in exactly the same way by any reliable observer. Subjective indicators can be self-reported measures of the stakeholder or customer experience, or reports of outsiders observing a program. Subjective indicators can amplify the information that is gained from objective measures by describing quality rather than quantity. For example, an objective indicator may state that 20 teachers were trained, while the subjective indicator may reveal that of those 20, only 10 teachers felt prepared to teach what they had learned. Assets vs. Deficiencies Most social science research is designed to collect and analyze data about problems, or deficiencies. In a poor community, for example, deficiency indicators may measure crime and violence, or joblessness and welfare dependency. While indicators that measure deficiencies are often easier to collect, they may have the effect of lowering customer participation because they tend to create apathy. Asset indicators within a poor community, on the other hand, may assess the capacities of residents and workers, or the associational and institutional base of an area--not what is absent, or what is problematic, or what the community needs. Asset indicators can be very helpful when using performance information in a planning process because they provide the foundation for positive change. Quantitative vs. Qualitative Quantitative indicators are useful for summarizing large amounts of data and reaching generalizations based on statistical projections. Qualitative indicators can "tell the story" from the participant's viewpoint, providing the rich descriptive detail that sets quantitative results into their human context. One set of indicators is not better than the other; both are derived from multiple disciplines and can be used to address almost any research topic. Many researchers today argue for an integration of quantitative and qualitative methodologies.4 This document focuses on the meaning and selection of indicators that can be used to measure the impact of inter-sectoral partnering. The guide does not prescribe a specific set of indicators that must be measured in order to judge whether or not a partnership has produced successful results. Rather, the guide provides a framework with a menu of indicators from which to choose. The selection of indicators for a particular partnership should be done jointly by members of the partnership and the evaluators. This will ensure that the selection of indicators reflects what the partners think are important characteristics for the type of partnership and the results they seek. FOOTNOTES 2 - Definitions of performance monitoring, evaluation, and performance indicators can be found in USAID's Automated Directives System glossary http://www.usaid.gov/pubs/ads. 3 - GPRA web site: http://www.npr.gov. 4 - For a discussion of the "quantitative-qualitative debate," see http://trochim.human.cornell.edu/kb/qual.htm.

FOIA Requests : Privacy Policy : Email This Page Plug-ins : FAQs : Help Desk : Contact Us : Site Map : En Español