Section 5
Summary and Conclusions

- Conceptual framework
- Conclusions

Evaluation studies have an important role to play in providing information on the nature and performance of community crime prevention programmes. This information can be of particular use in improving programme effectiveness and efficiency where the information is valid, relevant, broad in scope and important for making significant programme judgments and decisions. Obtaining such information depends on using an appropriate evaluation model to guide the planning and conduct of an evaluation study.

This report is concerned with the conceptual foundations of evaluation models. Based on a review of evaluation frameworks and strategies, it has identified the key dimensions of evaluation models that can be translated into effective evaluation guidelines for the use of practitioners, administrators, community groups and evaluation consultants and researchers.

The main considerations in the selection of an evaluation model have been identified as the following:

• the characteristics of the programme
• the purposes of the evaluation study
• the generic criteria that apply to evaluation studies generally
• the specific criteria which apply to a particular study.

Paying only limited attention to any one of these considerations will make it less likely that the information provided by an evaluation study will meet the information needs of key programme stakeholders and other audiences.

Conceptual framework

A conceptual framework developed for identifying alternative evaluation models, representing various combinations of evaluation form, evaluation approach and evaluation methods, is described in the report. Key characteristics of community crime prevention programmes which are significant in the choice of an evaluation model are also described. The main elements of the framework are outlined below.

Programme characteristics

Three characteristics of community crime prevention programmes of particular concern in their evaluation are the:

• prevention approach
• programme type
• programme specification.

The prevention approach of a programme is usually classified broadly as being either an opportunity reduction approach or a social/developmental approach (O'Malley and Sutton, 1997).

The two main programme types identified here are those seeking to influence behaviour and those providing a product or service (Funnell and Lenne, 1990).

The programme specification is seen to comprise three aspects of a programme significant for evaluation: the programme setting; the target group; and the particular need or problem being addressed.

In addition to these characteristics of programmes, a further consideration important for evaluation and the choice of an appropriate and viable evaluation model is the social, cultural and political context within which a programme operates (Weiss, 1993).

Evaluation form, approach and methods

The conceptual framework identifies five evaluation forms and five generic evaluation approaches, which in combination represent the essence of alternative evaluation models. However, an evaluation model is only fully specified when linked with particular evaluation methods.

The different evaluation forms reflect the major purposes for which an evaluation study may be undertaken, for example to determine programme impact or to guide the design of a programme. The evaluation approaches, on the other hand, provide alternative bases for determining the high priority information to be gathered in an evaluation study and used in making judgments about the merit and worth of the programme. These judgments are concerned with determining programme quality as assessed against relevant standards, and with the value of the programme for achieving certain purposes.

The various combinations of these evaluation forms and approaches are illustrated in matrix form in Figure 5. For each cell of the matrix, there is a variety of methods which might be used. While certain research designs will tend to be more compatible with particular evaluation forms and approaches, there is a range of alternatives worthy of consideration. Similarly, a variety of data collection techniques might be used in association with each design, but some combinations will be more compatible than others in most evaluation situations.

Figure 5: Matrix of evaluation forms and approaches

In identifying the specific evaluation models that might be appropriate for an evaluation study, consideration first needs to be given to the main purposes of the study. These purposes should reflect the information needs of the key stakeholders, given the nature of the decisions and judgments they face. However, determining appropriate models should also be influenced by both the programme characteristics and the various generic evaluation criteria and standards, as outlined in the Programme Evaluation Standards (Joint Committee on Standards for Educational Evaluation, 1994).

Finally, the study-specific constraints and criteria also need to be taken into account in the selection of the evaluation model. These will include the resources available for the evaluation study and the accessibility of alternative sources of information.

Conclusions

From a review of recent evaluation studies of community crime prevention programmes eight studies were selected and categorised in terms of their programme characteristics and evaluation model to illustrate the nature of the models typically used. The results of this analysis are shown in Table 1.

A number of conclusions can be drawn from the analysis of these particular cases and the wider review which was undertaken. These conclusions are concerned with evaluation models in community crime prevention, including the evaluation form, approach and methods typically used.

Evaluation form
Overwhelmingly the empirical evaluation studies identified involved impact or process evaluation. There are also several examples of conceptual studies related to evaluation for development, such as literature reviews conducted to inform the development of social/developmental community crime prevention programmes. Nevertheless, few examples were found of empirical evaluation studies primarily concerned with the evaluation for development, design evaluation or evaluation for management forms of evaluation.

Evaluation approach
Goal-based evaluation approaches were found to dominate the evaluation of community crime prevention initiatives. In the literature search, it was rare to find examples of evaluation studies that are illuminative/responsive, in spite of the contribution such studies can make to how programmes work in practice; and the issues that need to be addressed in steering programme delivery as it was intended, or currently is thought to be most productive.

Evaluation methods
Experimental or quasi-experimental methods are very commonly adopted in evaluation studies of community crime prevention programmes, with quantitative data being of primary concern. In fact, one commentator has noted that evaluation appears to be 'method driven', i.e. by a desire to implement strong designs. This appears to reflect a research orientation to evaluation in this area, as opposed to an action orientation (Mayne and Hudson, 1992) which usually is more responsive to the information needs of a range of decision makers, including programme stakeholders and other audiences.

Selection of evaluation models
The choice of an evaluation model in a particular case should, from the point of view of best practice, emerge from the interplay of:

• an understanding of the nature of the programme
• the judgments and decisions of importance to key stakeholders
• the various forms, approaches and methods available to conduct an evaluation study
• the extent to which commonly accepted standards of practice can be met.

It would appear, however, that programme evaluation in community crime prevention is not taking full advantage of the wide range of alternative models that can result from this interplay. The likely consequence is that the potential benefit of the evaluative efforts, in terms of useful information that is relevant and important to the needs of key stakeholders and audiences, is not being realised at present.

Given these conclusions, it is clear that the evaluation of community crime prevention programmes would be better served if a wider range of models was used to meet the information needs of the stakeholders in community crime prevention. It is important that the evaluation model used to guide any particular evaluation study is 'optimal' given the specific relevant considerations applying in that case. An evaluation model is more likely to be optimal if careful and systematic consideration is given to the full range of models, and the programme characteristics and contexts, when the choice of model is being made.