1996, Number 4


                   Performance Monitoring and Evaluation

                                   TIPS

          USAID Center for Development Information and Evaluation


                        USING DIRECT OBSERVATION TECHNIQUES


What is Direct Observation?


Most evaluation teams conduct some fieldwork, observing what's actually going on

at assistance activity sites.  Often, this is done informally, without much thought to

the quality of data collection.  Direct observation techniques allow for a more

systematic, structured process, using well-designed observation record forms. 


Advantages and Limitations


The main advantage of direct observation is that an event, institution, facility, or

process can be studied in its natural setting, thereby providing a richer

understanding of the subject. 


For example, an evaluation team that visits microenterprises is likely to better

understand their nature, problems, and successes after directly observing their

products, technologies, employees, and processes, than by relying solely on

documents or key informant interviews. Another advantage is that it may reveal

conditions, problems, or patterns many informants may be unaware of or unable to

describe adequately. 


On the negative side, direct observation is susceptible to observer bias. The very act

of observation also can affect the behavior being studied.


When Is Direct Observation Useful?


Direct observation may be useful:


    When performance monitoring data indicate results are not being

     accomplished as planned, and when implementation problems are

     suspected, but not understood.  Direct observation can help identify whether

     the process is poorly implemented or required inputs are absent. 


    When details of an activity's process need to be assessed, such as whether

     tasks are being implementing according to standards required for

     effectiveness.


    When an inventory of physical facilities and inputs is needed and not

     available from existing sources.  


    When interview methods are unlikely to elicit

     needed information accurately or reliably, either

     because the respondents don't know or may be

     reluctant to say. 


Steps in Using Direct Observation 


The quality of direct observation can be improved by

following these steps.


Step 1.  Determine the focus


Because of typical time and resource constraints, direct

observation has to be selective, looking at a few activities,

events, or phenomena that are central to the evaluation

questions.  


For example, suppose an evaluation team intends to study

a few health clinics providing immunization services for

children. Obviously, the team can assess a variety of

areas physical facilities and surroundings, immunization

activities of health workers, recordkeeping and managerial

services, and community interactions. The team should

narrow its focus to one or two areas likely to generate the

most useful information and insights.


Next, break down each activity, event, or phenomena into

subcomponents. For example, if the team decides to look

at immunization activities of health workers, prepare a list

of the tasks to observe, such as preparation of vaccine,

consultation with mothers, and vaccine administration. 


Each task may be further divided into subtasks; for

example,  administering vaccine likely includes preparing

the recommended doses, using the correct administration

technique, using sterile syringes, and protecting vaccine

from heat and light during use. 


If the team also wants to assess physical facilities and

surroundings, it will prepare an inventory of items to be

observed. 


Step 2.  Develop direct observation forms


The observation record form should list the items to be

observed and provide spaces to record observations.

These forms are similar to survey questionnaires, but

investigators  record their own observations, not

respondents' answers. 


Observation record forms help standardize the observation

process and ensure that all important items are covered.

They also facilitate better aggregation of data gathered

from various sites or by various investigators. An excerpt

from a direct observation form used in a study of primary

health care in the Philippines provides an illustration

below. 


When preparing direct observation forms, consider the

following:


1. Identify in advance the possible response categories for

each item, so that the observer can answer with a simple

yes or no, or by checking the appropriate answer.  Closed

response categories help minimize observer variation, and

therefore improve the quality of data. 2.  Limit the number of items in a form.   Forms should

normally not exceed 40 50 items. If nessary, it is better to

use two or more smaller forms than a single large one that

runs several pages. 


3.  Provide adequate space to record additional

observations for which response categories were not

determined. 


4.  Use of computer software designed to create forms can

be very helpful.  It facilitates a neat, unconfusing form

that can be easily completed.  


Step 3.  Select the sites


Once the forms are ready, the next step is to decide where

the observations will be carried out and whether it will be

based on one or more sites. 


A single site observation may be justified if a site can be

treated as a typical case or if it is unique.  Consider a

situation in which all five agricultural extension centers

established by an assistance activity have not been

performing well.  Here, observation at a single site may be

justified as a typical case.  A single site observation may

also be justified when the case is unique; for example, if

only one of five centers had been having major problems,

and the purpose of the evaluation is trying to discover

why.  However, single site observations should be avoided

generally, because cases the team assumes to be typical or

unique may not be. As a rule, several sites are necessary

to obtain a reasonable understanding of a situation.


In most cases, teams select sites based on experts' advice.

The investigator develops criteria for selecting sites, then

relies on the judgment of knowledgeable people. For

example, if a team evaluating a family planning project

decides to observe three clinics one highly successful,

one moderately successful, and one struggling clinic it

may request USAID staff, local experts, or other

informants to suggest a few clinics for each category. The

team will then choose three after examining their

recommendations. Using more than one expert reduces

individual bias in selection. 


Alternatively, sites can be selected based on data from

performance monitoring. For example, activity sites

(clinics, schools, credit institutions) can be ranked from

best to worst based on performance measures, and then a

sample drawn from them. 


Step 4.  Decide on the best timing


Timing is critical in direct observation, especially when

events are to be observed as they occur.  Wrong timing

can distort findings.  For example, rural credit

organizations receive most loan applications during the

planting season, when farmers wish to purchase

agricultural inputs. If credit institutions are observed

during the nonplanting season, an inaccurate picture of

loan processing may result.  


People and organizations follow daily routines associated

with set times. For example, credit institutions may accept

loan applications in the morning; farmers in tropical

climates may go to their fields early in the morning and

return home by noon. Observation periods should reflect

work rhythms.


Step 5.  Conduct the field observation


Establish rapport.  Before embarking on direct

observation, a certain level of rapport should be

established with the people, community, or organization to

be studied. The presence of outside observers, especially

if officials or experts, may generate some anxiety among

those being observed.  Often informal, fr