Section 3
The Nature of Programme Evaluation

- The purpose of programme evaluation
- Evaluation form, approach and methods
- Evaluation form
- Evaluation approach
- Evaluation methods
- Illustration of an evaluation model
- Conclusion

Programme evaluation can contribute to policy and programme development, implementation and decision-making in a number of ways. One important contribution is to provide a description of the specific context in which a programme operates; as well as the nature of its clients, physical and human resource inputs, and the intervention processes used in its implementation. This information documents what the programme actually is, the programme reality, in contrast to what was intended or might be assumed about it.

Evaluation can also identify the programme outputs (such as number of clients serviced) and outcomes (the results of the services provided), including both the intended and the unintended programme effects. In addition, an evaluation study can improve understanding of the underlying mechanism or causal processes by which the outcomes of a programme are achieved (Pawson and Tilley, 1994), i.e. it can contribute to an understanding of the 'why' of the outcomes as well as to knowledge of the 'what' and 'how' of a programme.

This section outlines the main stages of the evaluation process and two major orientations a study might adopt, and emphasises the need to clearly determine the nature of the information to be generated by an evaluation study. It also outlines the three elements of an evaluation model (the evaluation form or purpose, approach and method) and examines an example of an evaluation study indicating the evaluation model used.

The purpose of programme evaluation

To ensure that an evaluation study will yield useful information, the nature of the required information needs to be determined in the planning stages of the study. This determination depends upon who the audiences for the information are and the purposes for which they need the information, so a clear specification of the audiences is needed. Programme evaluation therefore is a process involving:

• deciding what information to gather
• obtaining that information
• providing the information to key audiences, including stakeholders
• use of the information by the audiences.

It may be defined more formally, as follows:

Programme evaluation is the process of delineating, obtaining and providing information which is of use in describing and understanding a programme and in making judgments and decisions related to the programme (Straton, 1981).

This process is outlined in Figure 3 and illustrated in more detail elsewhere (Straton, 1990).

Figure 3: Model of the evaluation process

Ensuring the usefulness of evaluation information

It is often assumed that evaluation information has not been used, or is not useful, unless specific overt decisions (and sometimes actions) have clearly been influenced by the information. However, this is a limited view. Frequently, such instrumental use of the information is not feasible because of various contextual, political, ethical or financial constraints, yet the information should nevertheless increase knowledge and understanding of the programme, i.e. there would be conceptual use. It may also contribute to the acceptance of a position already taken in relation to the programme, which has been called persuasive use (Shadish, Cook and Leviton, 1991). Patton (1997) has suggested that, in addition, stakeholders often change their views on a programme and their behaviour in relation to it as they reflect on the programme during the evaluation process, and calls this process use.

Clearly, the resources expended on evaluation must be justified by the value of the information provided. Hence, it is essential that evaluation studies be focused and conducted in ways that will enhance the likelihood that the information will be useful and used by the key audiences and stakeholders.

Determining audience needs
In order to maximise the usefulness of evaluation information, evaluators need to recognise the varying roles and responsibilities of the primary audiences of the information. Mayne and Hudson (1992) suggest that differences in these roles and responsibilities will lead to priority being given to information which is primarily useful either for programme management within a relatively short time frame, or for enhancing 'knowledge about a particular form of intervention in society' (1992: 10). Thus, they distinguish between 'action-oriented' evaluation and 'research-oriented' evaluation.

Research-oriented evaluation puts a high premium on methodological rigour since it is seen as a form of scientific inquiry. The emphasis is on longer-term use, rather than on the information being immediately useful for modifying programmes. The review by Sherman et al (1997) of what works and what does not work in crime prevention represents a research-oriented perspective on evaluation.

Action-oriented evaluation seeks to address the immediate information needs of those implementing, managing and modifying programmes. Mayne and Hudson (1992) point out that managers want to know how and why their programmes are working or not working, and so adopt an action-oriented perspective on evaluation. Improving their understanding of the programme puts them in a better position to make informed management decisions about how it can be improved, transferred to other settings and implemented with other target groups.

In considering these different orientations, Mayne and Hudson (1992) point out that each gives priority to different aims and so should be judged on that basis. Indeed, from their point of view, criticising research-oriented evaluation as not being effective in modifying programmes or criticising action-oriented evaluation as being weak methodologically and therefore of limited scientific value, is counterproductive. Clearly, both are important and complementary in most fields, and particularly in the evaluation of community crime prevention programmes.

Evaluation form, approach and methods

In choosing a model for an evaluation study it is important to first determine the main reasons for the study and consider the alternative evaluation approaches and methods that may be used. These considerations are outlined below and then illustrated in the identification of an appropriate model for the evaluation of a community crime prevention programme.

Evaluation form

An evaluation study may be undertaken for a number of reasons. The most common sets of reasons have been outlined by Owen (1993; Owen and Rogers, 1999) who has identified a particular form of evaluation for each. These reasons are to:

• determine the impact of an existing programme (impact evaluation)
• provide feedback information on a regular basis to facilitate programme management (evaluation for management or monitoring evaluation)
• obtain guidance on the modification of programme inputs and processes (process or interactive evaluation)
• clarify the underlying programme logic or rationale (design or clarificative evaluation)
• assist in programme development by identifying areas of client need and the resources that may be used in a new programme (evaluation for development or proactive evaluation).

This classification indicates that, in addition to the evaluation of programme impacts, programme development and implementation are also appropriate subjects for evaluation research. This is consistent with what Visher and Weisburd (1998: 230) call the 'new approach' to crime prevention research. Owen (1993, Owen and Rogers, 1999) also considers the current state of the programme (whether it is currently under development or 'settled'), the components of the programme likely to be of major interest, and the timing of an evaluation study in suggesting the evaluation approaches likely to be most appropriate2.

Impact evaluation
Impact evaluation establishes the effects of a programme once it has been implemented and settled for a period of time. This may involve determining the degree to which programme objectives have been met or documenting both intended and unintended outcomes. The main use of these data is to justify whether the programme should continue to be implemented or implemented in other settings and, if so, whether any modifications are required. Thus, it has a strong summative evaluation emphasis. Because of this, impact evaluation is usually conducted after some logical end point in the programme has been reached, such as where a neighbourhood watch programme has been fully operational for a year.

Evaluation for management
Evaluation for management or monitoring evaluation focuses on programme outcomes and delivery for management decision-making and accountability purposes. These data are used primarily to account for the expenditure of programme funds, including the extent to which key accountabilities have been met by programme managers. This type of evaluation 'is appropriate when a programme is well established and ongoing' (Owen, 1993: 24). It frequently involves keeping track of how the programme is progressing. Real time feedback to managers is an important feature of this type of evaluation.

Process evaluation
Process or interactive evaluation examines programme implementation including the extent to which a programme is being delivered in the way it was intended to be delivered. The information gained is used to determine how the implementation of the programme could be improved and so has a strong formative evaluation emphasis. Consequently, this form of evaluation is conducted as the programme is being delivered within its various settings. The information is of particular use to those implementing the programme.

Design evaluation
Design or clarificative evaluation is concerned to clarify the underlying rationale of a programme. Programme developers use this information to think through, and make explicit, the logic that supports the programme, including assumptions about how its components link together to produce the desired outcomes. While design evaluation would usually occur before the implementation of a programme, it may also be carried out while a programme is operating if it is not clear how it was intended that the programme was to be delivered. As such, it has a formative evaluation orientation.

Evaluation for development
Evaluation for development or proactive evaluation focuses on the actual need for a programme. The main use of these data is to help planners determine what type of programme would meet the identified social need or problem. This type of evaluation is carried out before a programme is developed.

Ekblom and Pease (1995) have proposed an alternative framework for distinguishing between various evaluation purposes, which addresses some of the issues inherent in determining the evaluation form. In this framework a major distinction is made between evaluations that address implementation issues (for example 'What were the practical difficulties in implementing the programme?') and those which seek to determine programme impact (for example 'Was there a real change in crime as a result of the implementation of the programme?'). While a distinction between implementation and impact evaluation captures some of the variation reflected in the five forms of evaluation suggested by Owen (1993; Owen and Rogers, 1999), it does not capture all of it. As noted above, evaluation can also be used for clarifying the underlying programme rationale, as well as for documenting current unmet needs in the community.

Evaluation approach

Numerous suggestions have been made about how evaluation studies might be conducted. Several authors have analysed and classified these into a few distinct broad approaches to evaluation, providing a description of their rationale and main features (House, 1980; Madaus, Scriven and Stufflebeam, 1983; Straton, 1985; Stufflebeam, 2001; Stufflebeam and Shinkfield, 1985; Taylor, 1976). There are five evaluation approaches that are most likely to be of use in the evaluation of community crime prevention programs:

• goal-based evaluation
• decision-oriented evaluation
• systems analysis
• professional review
• illuminative/responsive evaluation.

The nature of each approach is outlined below together with an indication of its rationale.

Goal-based evaluation
Goal-based evaluation focuses on obtaining information on the extent to which the objectives of a programme have been attained. It assumes that programme goals represent the most important criteria in judging the worth of the programme. However, the results of the study may prove to be inadequate or even misleading where the goals are inappropriate or have been superseded. There also is a risk of ignoring significant unintended effects of the programme, effects which may be positive or negative.

Decision-oriented evaluation
Decision-oriented evaluation identifies the key decisions to be made about the future of the programme and seeks to obtain information relevant to those decisions. Information about the attainment of existing programme goals may not be of high priority, particularly where client need or programme context will be different in the future. Rather, the major concern is the extent to which the programme is likely to be successful in the future in the locations and contexts in which it might be implemented, and how it might be adapted to achieve the required degree of success. This approach may, however, yield information too narrow in scope if the decisions to be faced are not well anticipated or the information needs are not appropriately specified.

Systems analysis
Systems analysis aims to determine programme efficiency, providing information on a few key indicators of programme effects (including the extent to which goals have been attained) and programme costs. The indicators typically provide only highly aggregated information on the programme effects and little or no information on how the programme might be made more effective or efficient. In essence, the information may be useful for broadly based decisions at the highest levels, but be of little use to those directly responsible for improving programme delivery and management or adapting it to changing circumstances.

Professional review
Professional review relies on the judgment of experts from outside the programme for determining the key information to gather, the suitability of programme objectives and processes, the degree to which it is successful and changes that should be made to the programme. It assumes that experts in relevant fields are best placed to determine the criteria and the information to be used in the evaluation of the programme. The usefulness of the information and the recommendations will be determined by the extent to which the judgments of these professional experts are appropriate to local circumstances and needs, and reflect the values and priorities of the key stakeholders.

Illuminative/responsive evaluation
Illuminative/responsive evaluation aims to provide a fine-grained depiction of a programme, focusing on 'thick' description, an understanding of the complexity of the programme and a portrayal of the experiences of programme participants including clients, staff and others affected by the programme. This approach assumes that a detailed and intimate view of a programme is required to understand its operation and accomplishments - well enough to provide a sound basis for making appropriate changes to the programme or judgments about its success. The potentially high authenticity of these descriptions and their utility for those delivering the programme on a day-to-day basis might be obtained, however, at significant risk of bias and reduced credibility among those further removed from the programme.

Every approach to evaluation involves trade-offs, so that in any particular evaluation study it may be necessary to combine the characteristics of more than one approach to achieve the major purposes of the study. This may mean developing a specific evaluation approach tailored for a particular study rather than simply selecting one from those identified.

Evaluation methods

The full range of social research methods may usefully be applied in undertaking evaluation studies, including the various alternative research designs and techniques of data collection and analysis. Nevertheless, particular methods are often found to be useful for achieving certain evaluation purposes as reflected in the evaluation form and approach. In particular cases, however, unique programme-specific considerations may make those 'usual' methods inappropriate and other methods more appropriate.

In determining the effect of a programme on the incidence of particular types of crime, for example, obtaining quantitative data using an experimental or strong quasi-experimental design frequently might be seen to be the most fruitful method to use. However, ethical, access or other constraints may make it impossible or inappropriate to use such methods. Similarly, the use of naturalistic methods to obtain qualitative data may often be useful for process evaluation, but resource constraints and the extensive nature of the programme may mean that an ex post facto design and quantitative data may be the most feasible method to use in a particular case.

Alternative evaluation methods

The very broad division of research methodology into qualitative and quantitative is a useful distinction even though it masks the wide array of different types of data that might be collected and ways of doing so. In this section, five broad evaluation methods have been distinguished which reflect more the alternative research design frameworks which might be used to conduct an evaluation study than the various data collection techniques. This has been done on the grounds that the choice of design typically is a broader decision, and that either qualitative or quantitative data, or both, might be obtained within any of these design frameworks in any particular evaluation study. These five groups of evaluation methods are identified as:

• experimental methods
• quasi-experimental methods
• survey research methods
• naturalistic research methods
• ex post facto research methods.

Experimental methods
A high degree of control over where, when, how and to whom an intervention is administered is required in a true experiment so that programme effects can be determined by comparing initially equivalent 'treatment' and 'control' groups. However, this level of control is difficult or impossible to achieve in most evaluation settings and so it is rare to find experimental methods fully and appropriately applied in programme evaluation. In addition, the random allocation requirements of these designs, which typically need large or homogeneous populations of cases or programme settings to work effectively, frequently cannot be met, even approximately.

Quasi-experimental methods
Quasi-experimental methods share with experimental methods a major concern for group comparisons using quantitative data, but adjust to the constraints of the programme setting in a variety of ways. These include the comparison of non-equivalent intact groups that have been subjected to different interventions and examining change over time for the programme group on programme-related variables. Because of the lack of strictly equivalent comparison groups, quasi-experimental methods may allow a variety of alternative explanations for any observed differences (or lack of them). This makes the interpretation of the findings, including conclusions about the efficacy of the programme, somewhat uncertain. Nevertheless, quasi-experimental methods can often usefully be adopted in evaluation studies, although the conclusions drawn are more clearly reliant on the logic of the evaluative argument which is developed, based on the available evidence.

Survey research methods
Survey and naturalistic methods tend to be more descriptive, relying on the reports of participants and other stakeholders. This often makes these methods particularly appropriate for obtaining information on perceptions of a programme's context, processes and outcomes.

Survey research methods are characterised by a strict adherence to formal sampling designs and a commitment to obtaining high response rates so as to ensure a representative sample of respondents. They can be particularly useful in identifying the various perspectives held on a programme and its effects. Surveys may be difficult to implement well in an evaluation context, however, due to problems in developing an appropriate sampling frame and differential access to sub-groups within the sample - such as current and previous programme participants and non-participants. There is also a tendency towards high refusal rates and 'sanitised' responses among those who think that their access to services or their jobs might be adversely affected by providing negative comments on the programme.

Naturalistic research methods
Naturalistic research methods can provide useful in-depth information about a programme through the use of extended interviews with open-ended questions and participant and non-participant observation, allowing the detailed exploration of significant issues. These methods may, however, have problems similar to those encountered in surveys.

The use of deliberative rather than formal sampling procedures may make sampling easier, but can raise serious questions about the representativeness of the information, particularly as resource constraints will usually severely limit the range and number of sources of information which can be used. This can lead to limited credibility of the information obtained, a shortcoming which may be overcome to the extent that the information clearly contributes to and is compatible with a well-argued case about the nature and achievements of the programme.

Ex post facto research methods
Many evaluation studies focus on a programme that has been in operation for some time, and are initiated and conducted over a relatively brief period. These studies are essentially retrospective rather than prospective. Thus, the use of ex post facto methods may be required, such as case control studies where those who participated in the programme are compared, after the fact, to those who did not. Clearly, in these situations it may not be possible to observe all significant programme processes and stages or to follow up a representative sample of programme clients during and after their participation to determine outcomes. Useful information about a programme can be a product of ex post facto studies, but the range of possible alternative descriptions and explanations for the apparent programme context, inputs, processes and outcomes means that plausible conclusions about the programme will depend on the strength of the evaluative argument based on this information.

Choosing an evaluation method

It is important to note that no evaluation method is necessarily superior to the other methods per se. The extent to which the various methods will be appropriate and optimal will depend upon the context of a particular evaluation study with its specific purposes and constraints, and taking account of the various trade-offs which must be made.

In deciding on the evaluation methods to use, careful consideration should always be given to the likely threats to internal and external validity and the measurement validity of the data the study will yield. In addition, the costs involved, access to information sources, the sampling designs and procedures which might realistically be used, the ability to maintain the integrity of alternative interventions, the availability of required expertise, and time constraints all are factors to be considered in examining these trade-offs. Where a comprehensive evaluation study is to be undertaken, a combination of methods is likely to be required (National Crime Prevention, 1999).

The three main evaluation considerations in selecting a model for an evaluation study are summarised in Figure 4. This figure indicates that the evaluation form needs to be determined first, as this will reflect the main purposes of the study. Next, an appropriate evaluation approach is specified, followed by a decision on the methods to be used. However, the decisions about the evaluation form, approach and methods are made repeatedly so that the particular constraints of the study, such as the time and other resources available, can be accommodated in an 'optimal' way.

Figure 4: Evaluation considerations in selecting the model for an evaluation study

Illustration of an evaluation model

The evaluation study of the neighbourhood watch programme implemented in Britain from 1983 and described by Bennett (1989) is an example of the application of an evaluation model frequently used in this field. A major focus of the study was the programme outcomes, particularly the extent to which the programme had achieved its objective of reducing victimisations one year after its implementation. Such a focus is characteristic of impact evaluation using a goal-based approach, i.e. there is a concern with outcomes directly related to programme objectives, in a programme which essentially is 'settled', to determine whether its continuation is justified (Owen and Rogers, 1999).

To determine the programme's effectiveness, a quasi-experimental design was employed comprising four areas: two 'experimental' areas; a control area; and a displacement area (Bennett, 1989). The choice of a displacement area is given particular attention by Bennett who draws on the theory (mechanism) underlying neighbourhood watch in his discussion. Thus, in general terms, the model used in the planning and conduct of the evaluation study comprised an impact evaluation form, conducted through a goal-based approach, using quasi-experimental methods.

Conclusion

The extent to which the model used in the neighbourhood watch evaluation was 'optimal', given the information needs of the key stakeholder groups and the various constraints within which the study was conducted, cannot be determined from the description provided. Nevertheless, the choice of model for an evaluation study needs to be fully justified in each particular case, taking into account a range of considerations, including the nature of the programme and its context, the information needs and priorities of various stakeholders, the resources available for the study and other constraints, and various methodological considerations. In essence, a number of trade-offs need to be made so that the extent to which a study yields relevant and important information is maximised within the programme, and evaluation constraints which apply.