|
|
|
|
|
|
|
|
| >> This is USAID >> Intersectoral Partnerships >> ISP Handbooks | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This section describes the sixteen categories of ISPs (see Figure 3) and discusses the specific indicators that fit into each of the categories in the ISP Assessment Framework. The framework is designed to assist the development professional in selecting categories and indicators that best fit the objectives of a specific ISP. Collecting data for at least one indicator from each domain (values and capacity, process, and impact) will help to provide a comprehensive view of the results achieved. It should be noted that most of the indicators come from sources using them to measure results other than those produced by ISPs.7 The wording of the indicator, therefore, may need to be adapted to fit a particular ISP. Furthermore, the list of indicators is not comprehensive. New indicators that fit within the framework may need to be developed for specific ISPs. Finally, in some cases much of the data for the indicators already exists, and it is only a matter of collecting it from an existing source. Other proposed indicators may require more effort, such as conducting a survey or interviews. Values and Capacity The first aspect of an ISP to be assessed is the nature and identity of the partnership, in the form of its values and capacities. Three dimensions can be tracked: organizational capacity, organizational culture, and external environment. Organizational Capacity Three categories can be assessed over time when documenting the level of organizational capacity. These are the capacity of the organization to plan, administer, and gather resources. (See Table 1.) Planning: The Inter-American Foundation (IAF)8 defines an organization's capacity to plan as an application of a strategic planning approach to the design, follow-up, and evaluation of programs, taking into account institutional goals and objectives as well as contextual factors, and the capacity to develop, implement, and document effective development methodologies. Figure 3: The Sixteen Categories of ISPs
In assessing planning capacity, the IAF measures the degree to which planning and evaluation processes are systematic. The indicator refers to the overall planning process of the organization and not to the work plan for a specific project. IAF uses a five point scale ranging from "nothing systematic" to "completely systematic." IAF also measures the adaptability of an organization by the degree to which planning is adapted/adjusted (as needed) to the results of evaluation and the influence of contextual factors. This is measured on a five point scale ranging from "never" to "always." Administrative: The IAF defines the capacity of an organization to administer by its capacity to assemble and manage the human and financial resources necessary to carry out the institutional mission, and its capacity to manage a productive activity. It measures the management capacity of human resources development, communication, financial management, and accounting. The measures of management capacity are opinions ranging from "never" to "always" on a five point scale. Resources: The IAF defines the capacity of an organization to handle resources as the generation or mobilization of resources to finance the institution's operating expenses and to support grassroots programs and organizations. Resources can be financial, material, or human. Sources can be international, national, or internal. Profitability is measured by looking at whether the income produced by the activity achieves profits, covers its costs or does not cover its costs. IAF notes that this indicator only applies to income-generating projects. The IAF framework also examines the degree of economic sustainability achieved by the organization as it pertains to resource administration. This is measured by the percentage of the organization's total resources which are internally generated. USAID's NPI Resource Guide9 also suggests indicators to assess resource capacity. Specifically, it looks at the strength of the partnership by assessing achievements in terms of raising resources and notes that a greater diversity of funding sources indicates increased strength for an organization. The IAF uses a quantitative indicator to measure the mobilization of resources. Resources are monetary as well as material and human. It also identifies the resources negotiated by the organization and directed to groups. The NPI Resource Guide also mentions indicators related to the mobilization of resources and cites examples from missions in Bulgaria, the Philippines, and Sri Lanka. PACT's Discussion-Oriented Organizational Self-Assessment tool (DOSA)10 is another source of indicators that can be used to assess organizational capacity. PACT developed this tool to assist non-governmental organizations in strengthening their organizational capacity. Specifically, DOSA measures organizational capacities and consensus levels in six critical areas and assesses the impact, over time, of these activities on organizational capacity. The six critical areas are:
Table 1: Values & Capacity - Organizational Capacity
Organizational Culture In the second dimension, organizational culture, there are three categories that can be assessed: vision, presence of a common issue, and commitment. (See Table 2.) The presence of a common issue should be easy to identify in a goal or mission statement. The other categories are more difficult to capture in a results framework. While the existing indicators for this dimension have been used primarily to assess NGOs, they can be adapted for use in assessing ISPs. This does not imply, however, that ISPs are another type of NGO. ISPs represent a different set of relationships. Some ISPs may look like NGOs, while others may possess characteristics more similar to businesses. Many have a hybrid set of characteristics unlike more traditional organizational arrangements. Vision: The IAF defines vision as the ability to perceive reality in new and unconventional ways -- and the ability to see beyond the immediate project, the immediate institution, the immediate moment -- and to foresee change. The capacity to identify opportunities is measured by the degree to which the organization identifies new opportunities and situations that motivate its action on a five point scale ranging from "never" to "always." The IAF framework also measures the degree of perception of short- and long-term consequences resulting from present actions on a five point scale ranging from "never" to "always." Presence of a common issue: This category is easily assessed by asking whether partners agree on the existence of a common issue or goal that has brought the partnership together. Commitment to the ISP concept: Commitment is addressed through what the IAF calls solidarity, or the identification of common goals shared by organizations, and a willingness to join forces. Cooperation is measured by the degree of mutual support with other organizations based on affinity and mutual interests on a five point scale ranging from "none" to "total." In addition, the priority of the collective interests is measured as the frequency with which the individual organizations postpone their own goals when necessary to achieve goals agreed upon within the collective. Table 2: Values & Capacity - Organizational Culture
External Environment The final dimension of the values and capacity of an ISP is the external environment, which includes two categories: the capacity of each of the individual sectors (business, government, and civil society) that are collaborating on an ISP and the mechanisms that allow sectors to work together. (See Table 3.) Capacity of Individual Sectors: The capacities of individual sectors, while closely related, will need to be assessed independently in order to understand the environment that will influence the success of the ISP. Many resources exist to assist the development professional with this task. Private Sector A number of indices assess the capacity of private sector development. These include the Heritage Foundation's Economic Freedom Index and the Cato Institute's Freedom Index to assess private sector capacity.11 Many of the indicators can also be obtained through USAID's Economic and Social Data Service (ESDS). The Euromoney and Institutional Investor Indicators and World Bank World Development Indicators are other good sources with lists of indicators by which to assess the private sector. Government When assessing the capacity of the public sector, the work of USAID's Global Bureau Center for Democracy and Governance is particularly useful.12 This office has developed indicators to assess the fourth USAID democracy and governance objective -- more transparent and accountable government institutions. Achievement of this objective is measured through six intermediate results (IRs).13
Civil Society When assessing the capacity of the civil society sector, USAID's Handbook of Democracy and Governance Program Indicatorsalso offers useful indicators. USAID currently has five intermediate results relating to the increased development of a politically active civil society. While the Center has focused its attention on the role of politically active organizations, the following three IRs and corresponding indicators can be applied to the general external framework for all civil society organizations.14
Another source for indicators to assess the capacity of the civil society sector in a country will be the Civil Society Index currently under development by CIVICUS. The index will provide an assessment of the health of civil society at the country level and its impact as an effective public actor nationally, regionally, and globally.15 Mechanisms that allow sectors to work together: The second category to review when assessing the external environment is the existence of external mechanisms that make it possible for an ISP to form. The indicator reflecting the existence of these mechanisms will change depending on whether the ISP is functioning on the local or national level. USAID/Hungary has made progress toward defining the necessary factors that foster collaboration across sectors. While trying to achieve increased citizen participation in decision-making processes, it is working toward developing collaborative forums across the three sectors. One indicator it has developed is the number of municipalities with institutionalized mechanisms for collaboration with NGOs. One such mechanism is the existence of a civil ombudsman. ISPs could adapt this indicator to reflect their specific situation. The NPI Resource Guide notes that the following mechanisms are necessary for an enabling environment: political will with adequate legal and constitutional structures; fiscal independence, accountability, transparency; sufficient information and communication; a positive macro-policy environment; a minimal level of social accord; and, appropriate regulatory frameworks. Table 3: Values & Capacity - External Environment
Process The organizational mechanisms and practices of the ISP, or the processes involved in the formation and functioning of the partnership, are the second aspect to be assessed. This domain is often difficult to assess because the results may be intangible. However, the mechanisms that are created to facilitate collaboration within the partnership, as well as with outside actors, are crucial to the success of the ISP. Further research is needed into how to both create and evaluate these mechanisms. Two dimensions can be tracked: 1) communicating and collaborating within the ISP; and, 2) communicating and collaborating outside the ISP. Communicating and Collaborating Within the ISP In order to have an ISP that can successfully address the development issue at hand, certain mechanisms for operating within the ISP must exist. These governing mechanisms for joint, or cross-sectoral initiatives, do not have to be developed or regulated by external mechanisms (Ostrom, Schroeder, and Wynne 1993). To ensure commitment, however, each ISP should develop its own governance and operating structure based on the joint interests of its members. Any collaborative effort should have a locally designed set of rules that include an accountable and transparent monitoring mechanism. Sanctions, or punishments, for breaking these rules must be graduated from benign to severe, depending on the nature and the frequency of the infraction (Ostrom 1990). Organizational mechanisms that can lead to sustainable development must include: economic efficiency, fiscal equity, redistribution, accountability, and adaptability.16 The Inter-American Foundation's grassroots development framework offers two variables that apply to this dimension of the process involved in inter-sectoral partnering: 1) democratic practices; and, 2) attitudes and behaviors. (See Table 4.) Democratic Practices: Democratic practice, as defined in the IAF Grassroots Development Framework, refers to the institutional style and modus operandi which facilitate accountability to members and clients, broad participation in decision-making, availability of information about the management and allocation of resources, and the equitable distribution of benefits. The IAF assesses democratic practices by measuring the degree of access enjoyed by the organization's members, officials, and beneficiaries to key and clear information about the organization's finances, programs, and policies on a five point scale ranging from "none" to "total." The IAF framework also analyzes "participation" by measuring the degree of participation of members and officials of the organization in decision-making (regarding the distribution of benefits, management of the organization, selection of leadership, etc.) on a four point scale ranging from "none" to "excessive." Attitudes and Behaviors: Attitudes and behaviors refer to the ways in which people deal with or react to, given situations, groups, or individuals; the presence or absence of respect, tolerance, equality, etc.; and the breakdown of stereotypes. The IAF grassroots framework looks at behavior with regard to socio-cultural diversity. The indicator is defined as the degree to which groups express or manifest in their social behavior their attitudes toward: equality between genders; and interaction with minority or other groups traditionally discriminated against. This indicator is measured on a four point scale ranging from "rejection" to "mutual respect." A second indicator of behavior measures the degree of responsiveness within public and private entities. This analyzes the modification of the behavior of representatives of public and private entities toward initiatives of the target population on a five point scale ranging from "very negative change" to "positive change." Table 4: Process - Communicating and Collaborating Within the ISP
This dimension focuses on a partnership's internal mechanisms that have been created to facilitate interaction with external actors. (See Table 5.) Mechanisms for relating to other entities: Two indicators have been developed by the Inter-American Foundation that apply to external relations. The first indicator measures the transition from relations of domination and dependency (between individuals, groups, or sectors) to relations based on equity and interdependence. They measure the capacity of the organization to negotiate with other civil society organizations based on an opinion survey using a five point scale from "very low" to "very high." Using the same scale, they also measure the capacity of the organization to negotiate with public sector entities. Work is needed to develop an indicator of the organizational capacity to negotiate with private sector entities. USAID's GOLD project developed another way to measure these interactions.17 Adapted for NPI, the proposed indicator to measure partnering mechanisms is a percent increase in a development partnership score based on the following partnership scale. The scale consists of the following levels, with a total point range from 5 to 20:
Table 5: Process - Communicating and Collaborating Outside the ISP
Impact The final critical aspect of an ISP that merits assessment is that of impact. There are three dimensions along which impact can be measured: the impact of the ISP on the common development issue, on the partner members, and on society at large. Impact of the ISP on the Common Development Issue Assessment of the impact of the ISP on the development issue at hand will, of course, vary depending on the nature of the specific ISP. For example, if the ISP has formed to address the issue of transportation between two communities, the impact of the ISP on this issue could be measured by the number of buses contracted and running, the number of passengers served by new routes, etc. Another ISP may form to solve the issue of lack of credit for small business owners. Indicators to measure the effectiveness of this ISP may be number of loans approved, amount of capital made available, etc. Typically, these are quantitative, output indicators pertaining to the outcome of the activity. These indicators will differ according to the issue at hand and should address the sustainability of the result. Impact of the ISP on the Partner Members When assessing the impact of an ISP on partner members, one may ask "has the partnership changed the members?" Two categories are relevant here: the effect on partner members and the sustainability of partnership. (See Table 6.) Effect on partner members: One method of documenting impact on the members is to examine the level of satisfaction with the partnership. One indicator proposed by the USAID's New Partnerships Initiative is a scale to measure the increase in satisfaction with the partnership based on a rating by partnership participants. The partnership members would develop this scale to assess their satisfaction based on the diversity of membership; quality and frequency of interaction; and the short-, medium-, and long-term achievement of goals. The NPI Resource Guide proposes measuring the number or percentage of institutions that are at a certain level on an institutional capacity index or have achieved a defined rating. This index would be similar to PACT's DOSA or the upcoming CIVICUS Civil Society Index. USAID's Bureau for Humanitarian Response Office of Private and Voluntary Cooperation (BHR/PVC) used the DOSA methodology to measure change in the mean capacity score of private voluntary organizations on a PVC-developed capacity self-assessment instrument. Sustainability of partnership: USAID's Bureau for Humanitarian Response Office of Private and Voluntary Cooperation currently measures the change in the number of members of formal networks or associations of voluntary organizations to assess increased capability of non-governmental partners to achieve sustainable service delivery. The NPI Resource Guide suggests indicators to assess whether or not the partnership has been strengthened. One indicator is whether or not the partnership has been transformed into something more formal like a coalition or a business. Table 6: Impact - Impact on Partner Members
Impact of the ISP on Society An ISP can effect broader societal change in addition to change in the specific issues addressed and among the partner members. This dimension can be assessed by examining two categories: social capital and the enabling environment. (See Table 7.) Social Capital: Social capital, the features of social organization that facilitate coordination and cooperation for mutual benefit, is important in terms of measuring the effectiveness of ISPs because it is often an outcome of ISPs. Social capital can be examined at both the local and national level and change in social capital should be examined over the long term. In order to apply the concept of social capital to a results framework for an ISP, one must be familiar with the theory of social capital and the state of social capital prior to the formation of the ISP. The IAF identifies a variable called "practices" that is defined as the incorporation or the replication of new approaches to problems or new ways of relating to people, based on prior experience of NGOs (or an ISP). Replication may be demonstrated by other NGOs, the government or the private sector. It is measured by the number of organizations/agencies (outside the proposed beneficiary population) which replicated, adapted, or scaled-up a methodology demonstrated by the organization. A second aspect of practices is the dissemination of new approaches, methodologies or techniques in order to promote changes in practices is also part of this variable. Dissemination is measured by the number of distinct books, magazines, videos, or movies that document the experience of a program and that reach a public outside the immediate range of the project. USAID's Mozambique Mission uses a public survey to assess its objective of having government and civil society function as effective partners in democratic governance. It measures the perception of whether local questions of interest are better resolved through collaboration between local government, citizens, and the private sector. The NPI Resource Guide notes that the Guinea Mission measures the number of forums, or their results, to assess the expansion of public-private dialogues. The World Bank's Social Development department has launched a Social Capital Initiative19 comprised of ten research products, one of which will be a tool to assess social capital. The Social Capital Assessment Tool (SCAT) assesses the cognitive and structural aspects of social capital at the micro-level. The three components of the SCAT include a community profile, a household survey, and an organizational profile. The SCAT assesses indicators relating to values, attitudes, behavior, and social norms, as well as the horizontal organizational structure, collective/transparent decision-making process, accountability of leaders, and the practices of collective action and responsibility. Enabling Environment: An ISP can also affect the enabling environment at the local and national levels. USAID and the IAF have developed a number of indicators to reflect impact on the enabling environment. The first variable relates to laws, or the passage, amendment, or repeal of legal measures that affect the beneficiary population. This refers to laws, regulations, statutes, etc. at the municipal, regional, and national level; the implementation of existing laws; and, placing important legal issues on the public agenda (mobilizing public opinion). The IAF measures the "enactment, modification, or repeal of legal measures" by the number of legal measures influenced by the organization (enactment, modification, or repeal) at the municipal, regional, or national level. The framework also identifies "consultation and participation" as the degree of institutionalization of opportunities for consultation or participation of the target population in the formulation of legal measures. A second variable offered in the IAF grassroots development framework is policies, defined as the formulation, approval, or modification of policies -- public or private -- at the municipal, regional, or national level which result in: changes in the rules of the game for the beneficiary population; implementation of existing policies; or, getting a new topic on the public agenda regarding social policy or the mobilization of public opinion. The IAF defines "influence" on policy as the capacity to promote/put forth for public consideration themes important to the target population and "proposals and demands" as the number of proposals and/or demands presented by the organization that became policy. The NPI Resource Guide suggests a number of indicators to assess the enabling environment: policies, laws, or regulations changed; laws or regulations enforced; degree of freedom for community action to occur; reduction in enabling environment constraints; and, extent of decentralization.20 The NPI guide also suggests measuring improved advocacy based on the number of advocacy interventions or successes. USAID/Lithuania evaluates increased responsiveness to public participation among municipal government officials by measuring: increased interaction of NGOs in various local government debate fora; increased mechanisms that local governments have put in place to assure citizen access to public officials; increased public accessibility to city financial and budget records and minutes of public meetings; and, the training of local government officials, local trainers and NGO activists in citizen participation. Table 7: Impact - Impact on Society
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
FOOTNOTES 7 - See Annex A for more detailed information about the various sources. 8 - The Inter-American Foundation Grassroots Development Framework provides a comprehensive list of indicators to measure tangible and intangible development results. The framework can be found on the web at http://www.iaf.gov/results/menu01.htm. 9 - The NPI Resource Guide is available on the web through http://www.usaid.gov/pubs/isp/. 10 - See http://www.edc.org/INT/CapDev/dosafile/dosintr.htm. 11 - See the Heritage Foundation's Economic Freedom Index web page at http://www.heritage.org/index/ and the Cato Institute's web page at http://www.cato.org/. 12 - See USAID's Handbook of Democracy and Governance Program Indicators. 13 - See the handbook for a complete list of indicators that can be used to measure each result. 14 - See the handbook for a complete list of indicators that can be used to measure each result. 15 - See the CIVICUS web page at http://www.civicus.org/cindex.html. 16 - See Ostrom, p. 112-116. A more extensive analysis of Ostrom's work may yield appropriate indicators to assess this aspect of an ISP. 17 - For more information on the GOLD project, see http://www.ardgold.org. 18 - This indicator was adapted from the Philippines GOLD project. See Brilliant memo. 19 - See http//:www.worldbank.org/poverty/scapital/index.htm. 20 - See the NPI Resource Guide for specific indicators used in particular missions. |
 |
|||
|
