Executive Summary
Programme information
Programme characteristics
Evaluation models
Conclusion
Community crime prevention programmes are used throughout Australia to reduce the incidence of crime and encourage community participation in crime prevention. These programmes frequently are not evaluated, but when they are, the evaluations typically are quite limited and contribute far less to decision-making about the programme than they might. Yet, an evaluation of a community crime prevention programme can provide valuable information about the way it was conducted and its context, outcomes and efficiency, helping managers and others to plan programme changes.
This report is one product of the Evaluation Models for Community Crime Prevention Project. The project was initiated by the Commonwealth Attorney-General's Department through its National Crime Prevention Programme and the Australian and New Zealand Crime Prevention Ministerial Forum. It was carried out in partnership with Commonwealth, State, Territory and local governments.
The main objective of the report is to develop an awareness of the models of evaluation which can be used to provide information to guide the practice and strategic direction of crime prevention. In the report a conceptual framework is developed which uses the characteristics of a community crime prevention programme and why information on it is needed as the basis for choosing an appropriate evaluation model. The various models which might be chosen are a combination of the particular form of evaluation (Owen, 1993) and the evaluation approach and methods which could be used in a study. Several recent evaluation studies are examined to illustrate a range of the models which have been used in evaluating community crime prevention programmes.
Programme information
Before beginning an evaluation study, evaluators need to ensure that the information they gather will be relevant and useful for making the required judgments and decisions about the programme. The planning stages of the study must determine what information is needed, which in turn depends upon who the audiences for the information are and the purposes for which they need the information. Programme evaluation therefore is a process involving:
deciding what information to gather
obtaining that information
providing the information to key audiences, including stakeholders
the stakeholders and other audiences using the information.
It will be clear that evaluation is an 'audience driven' process, providing information which is sufficiently relevant, broad in its scope, important and meaningful to meet the needs of key audiences.
Programme characteristics
Different approaches to community crime prevention embody different assumptions about the causal mechanisms (Pawson and Tilley, 1997) underlying the programme rationale. Two main approaches are usually identified:
opportunity reduction/situational, approaches which assume that crime can be reduced by 'changing the immediate situation in which offences may occur'
social/developmental approaches, which assume that crime is reduced by 'changing potential offenders generally in terms of the dispositions, motives, knowledge, and skills they bring to situations', and so emphasise the centrality of the individual and their personal development to make it less likely that they will engage in criminal activity.
The underlying rationale of a programme is crucial to evaluation because it is essential for understanding the 'why' of the programme outcomes. It should be critically examined in an evaluation study to determine whether its logic is flawed and to understand why any of the outcomes observed are not as intended.
Two examples of each type of approach are given in the report. Opportunity reduction/situational approaches are being used to deter prospective robbers of Victorian TABs (betting shops) by limiting access to cash; and reducing motor vehicle theft in New South Wales through a range of police-initiated measures relying on cooperation between government, private agencies and the community. Two Queensland examples are given of social/developmental approaches. One is the PeaceBuilders programme, which aimed at reducing anti-social behaviour through increasing children's resilience and reinforcing their positive behaviour. The other involved establishment of community justice groups in remote Indigenous communities to facilitate conflict resolution, sanctioning and liaison between criminal justice agencies and the community, in order to reduce personal and property crime and juvenile criminal activity.
Characterising prevention programmes
A scheme is outlined in the report which can help evaluators characterise community crime prevention programmes and, using that characterisation, choose an appropriate evaluation model. The three programme characteristics of concern are the prevention approach being used, the programme type and the programme specification, each having various sub-categories. Using this classification scheme a neighbourhood watch programme might be characterised in the following way:
prevention approach involves opportunity reduction embodying a situational-oriented causal mechanism
programme type is essentially advisory, being a programme that seeks to influence behaviour via the information and advice given by police, insurance companies, local government and security firms
programme specifications could include a range of geographical areas, with the target group being householders and burglary being the problem addressed.
Choosing an appropriate and viable evaluation model also depends upon taking account of the social and political context within which the programme operates, with each context providing unique challenges for evaluation.
Evaluation models
There are several different models that can be used to guide the planning and conduct of an evaluation study of a community crime prevention programme. Each model reflects a different combination of the evaluation form or purpose, evaluation approach and evaluation methods which might be used. In choosing the model to be used it is important to determine the reasons why the evaluation study is to be undertaken and then consider the alternative evaluation approaches and methods that may be used.
Evaluation form
The major purposes or reasons for conducting an evaluation study are reflected in the five evaluation forms specified by Owen (1993; Owen and Rogers, 1999):
impact evaluation, which establishes the effects of a programme once it has been implemented and settled for a period of time
evaluation for management or monitoring evaluation, which focuses on programme outcomes and delivery for management, decision-making and accountability purposes
process or interactive evaluation, which examines programme implementation including the extent to which a programme is being delivered in the way intended
design or clarificative evaluation, which is oriented to clarifying the underlying rationale of a programme
evaluation for development or proactive evaluation, which focuses on the extent and nature of the need for a programme.
Clearly, the state of a programme is an important consideration in determining the appropriate evaluation form, for example whether the programme is in place and in a 'settled' state; or whether it is a new programme being developed or 'bedded down' as it is implemented for the first time.
Evaluation approaches
There is a wide range of evaluation approaches. Five are most likely to be useful in evaluating community crime prevention programmes. While determining the extent to which programme goals have been achieved is often considered to be essential in an evaluation study. This will not necessarily be the most useful approach. The original goals may become less important over time, or inappropriate with recent or anticipated demographic changes, or for the implementation of the programme at new sites or with different sub-groups in society. The major approaches for evaluating community crime prevention programmes are the following:
goal-based, which focuses on obtaining information on the extent to which the objectives of the programme have been attained
decision-oriented, which identifies the key decisions to be made about the future of the programme and seeks to obtain information relevant to these decisions
systems analysis, which determines programme efficiency and programme costs
professional review, which relies on the judgments of experts from outside the programme about its appropriateness and efficacy
illuminative/responsive, which provides an understanding of the complexity of the programme and a portrayal of the experiences of programme participants including clients, staff and others affected by the programme.
Every approach to evaluation involves trade-offs. In any particular evaluation study it may be necessary to combine the characteristics of more than one approach to achieve the major purposes of the study, rather than simply selecting one from a list.
Evaluation methods
While the full range of social research methods might be used in evaluation studies, there are various programme specific considerations which may make certain methods inappropriate for a particular study. These considerations include ethical, access, cost and other constraints which might apply; such as those related to gathering information from minors. In this report five broad evaluation methods have been distinguished which reflect more the alternative research design frameworks that might be used to conduct an evaluation study than particular data collection techniques. These five evaluation methods are the following:
experimental methods, which determine programme effects by comparing initially equivalent 'treatment' and 'control' groups on quantitative data gathered on specifically defined variables
quasi-experimental methods, which are constrained to using non-equivalent comparison groups or time series to indicate programme effects on specific quantitative variables
survey methods, which tend to be more descriptive, relying on the reports of participants and other stakeholders who have been selected using formal sampling designs to ensure a representative sample of respondents
naturalistic methods, which typically use extended informal interviews, participant and non-participant observation, case studies and documentary analysis to obtain 'in-depth' information about programme processes and outcomes and the contextual and social influences on its operation
ex post facto methods, which are essentially retrospective in orientation, relying on 'after the fact' comparisons of programme participants and non-participants; and on informants' memories of programme processes, outcomes and the significant factors influencing them.
Most evaluation studies would benefit from using a mixture of evaluation methods. Information gathered by a range of methods is most likely to provide a broad understanding of programme processes and the factors which influenced its operation. In addition, using both qualitative and quantitative indicators should assist the identification of both the intended and unintended outcomes of the programme.
Examples of evaluation studies
No single evaluation model is superior to the other models across the board. The choice of model should be influenced by the context of a particular evaluation study and its specific purposes and constraints, and take account of various trade-offs. An appropriate evaluation model is one that should yield the information needed by the audiences through an optimal choice of evaluation form, approach and methods for the study. It should also facilitate meeting currently accepted standards of evaluation practice, such as those outlined in the Programme Evaluation Standards (Joint Committee on Standards for Educational Evaluation, 1994).
In this report a number of evaluation studies of community crime prevention programs are reviewed to illustrate the various combinations of evaluation form, approach and methods which represent the particular evaluation model used. Both Australian and overseas examples are examined.
Conclusion
The report concludes that evaluation studies are critical in providing information on the nature and performance of community crime prevention programs - information that can be instrumental in improving programme effectiveness and efficiency. However, the extensive literature review undertaken indicates that it is clear that while there are many different evaluation models available, only a limited range of models is being used. Overwhelmingly, the studies undertaken have:
involved impact or process forms of evaluation
used goal-based evaluation approaches
adopted experimental or quasi-experimental methods.
In many of the studies reviewed it was clear that the model adopted had significantly restricted the usefulness of the information provided and that other models would have been more productive. In fact, it seemed that a number of studies were method driven (Pawson and Tilley, 1994) and reflected a narrow view of evaluation rather than being audience driven, i.e. the choice of what methods to use had been dominant to the information needs of the key audiences.
The choice of an evaluation model for a particular study should, from the point of view of best practice, be based on a number of considerations:
the nature of the programme
the judgments and decisions of importance to key stakeholders and audiences
the various evaluation forms, approaches and methods which might be adopted
the extent to which commonly accepted standards of practice can be met by using a particular model.
It is apparent, however, that programme evaluation in community crime prevention is not taking full advantage of the wide range of alternative models that might be adopted. Thus, it is likely that the full benefit of the evaluative efforts (in terms of useful information that is relevant and important to the needs of stakeholders and other audiences) is not being realised. Given this conclusion, it is likely that community crime prevention programs would be better served if a wider range of evaluation models was used.