Section 4

Selected Evaluation Studies

- Preventing alcohol-related crime through community action
- Preventing crime through citizen patrols
- Community policing of neighbourhoods
- Transferring crime prevention initiatives
- Creating safer cities
- Reducing the risk of leaving school early and developing offending behaviour
- Reducing recidivism for illegal vehicle use
- Violence prevention through informal socialisation
- Conclusion

The choice of evaluation model will have a significant influence on the information obtained in an evaluation study, and on the way a programme is described and its operation and outcomes are understood. In this section eight evaluation studies of community crime prevention programs are examined to illustrate the various combinations of evaluation form, approach and methods which defined the particular evaluation models used. The examples have been chosen from recently published reports of evaluation studies using the following criteria.

• The crime prevention programs are community-based, being preventive efforts either for, with or through the community.
• The studies are examples of programme evaluation, with both research-oriented and action-oriented evaluation being included.
• The studies were completed within the past 15 years.
• The examples represent a range of programs and evaluation models.
• Both Australian and overseas examples are included.

Table 1 presents a summary of the characteristics of each of the selected evaluation studies.

Preventing alcohol-related crime through community action

Form: Impact evaluation
Approach: Goal-based
Methods: Survey, naturalistic

The evaluation model for this study combined impact evaluation, a goal-based approach and a range of methods, particularly survey and naturalistic methods, to provide the evaluation data (Homel et al, 1997). The purpose of the study was to examine the extent to which key programme activities had reduced alcohol-related crime, violence and disorder in and around licensed premises in a major tourist location in Queensland. The study also examined the extent to which these effects were maintained over the longer-term.

Homel et al (1997) point out that previous research into the nature of the problem had revealed that inappropriate drink promotions that encourage mass intoxication are a major risk factor for violence. The clear implication is that if the level of intoxication decreases, then the level of associated violence should also decrease.

Several strategies based on the association between levels of intoxication and levels of violence were implemented with the aim of decreasing the level of intoxication of patrons in and around the licensed premises. These included the formation of a community forum and several community-based task groups to oversee and direct the programme, and the development of model house policies and associated codes of practice for the responsible serving of alcohol. There were also improvements in the external regulation of licensed premises by the police and licensing inspections.

Study characteristics

During the evaluation study, data were collected before implementing any changes, as well as throughout and following the implementation period. This enabled changes over time to be monitored and used to assess the effects of the various intervention strategies. A comparison area was considered but not included because of budget constraints. The information collection procedures included community surveys, interviews with licensees, direct observation of serving practices at licensed premises, and the analysis of incident reports.

Findings

The findings of the evaluation study suggest that there had been major changes towards the responsible serving of alcohol and more responsible promotions. The rates of physical violence also dropped by 52 per cent, from 9.8 per 100 hours to 4.7 per 100 hours over the year the programme was implemented. Security and police data also revealed evidence of a decline in violence and street offences. However, the study found that two years after the programme had ceased its implementation phase, the rates of physical violence had increased again to 8.3 per 100 hours. This finding was interpreted by the evaluators as indicating the importance of maintaining community monitoring in programs where communities seek change and are empowered to bring about the changes. The evaluation team also underlined the importance of participatory approaches to programme design, implementation and evaluation in this area.

Preventing crime through citizen patrols

Form: Impact evaluation
Approach: Goal-based
Methods: Survey, quasi-experimental

Funded by the National Institute of Justice of the United States Department of Justice, the purpose of this study was to assess the impact of the Guardian Angels on deterring crime and reducing citizens' fear of crime. The study, by Pennell, Curtis, Henderson and Tayman (1989), illustrates a goal-based, impact evaluation, which employed survey and quasi-experimental methods.

The Guardian Angels organisation consists of 'unarmed, red bereted, peace seeking, racially mixed youth who "dare to care" by patrolling the streets as a visual deterrence to crime' (Pennell et al, 1989: 379). It began operating in 1977 and by 1983 had 148 chapters throughout the United States and Canada. Apart from acting as a visual deterrent to crime and reducing citizens' fear of crime, the Guardian Angels also sought to provide a positive role model for young people.

The authors report that the Guardian Angels tend to differ from traditional community crime prevention efforts in terms of leadership, membership, longevity and methods for reducing crime. They note that, traditionally, collective responses to crime prevention have adopted one of two main approaches. The victimisation approach focuses on increasing the community's awareness of risk and educating them about ways to reduce opportunities for victimisation. The social order approach assumes that the community's fear of crime indicates a lack of social control. In this latter approach, prevention programs are designed to increase the community's capacity for social control as well as to enhance social integration. The Guardian Angels, with its small, cohesive group membership, similarity in beliefs and social characteristics, and frequent contacts, appears to act as an informal social control mechanism.

Study characteristics

The evaluation study itself had a wide range of objectives. These were to:

• document the organisation and activities of the Guardian Angels
• evaluate the effectiveness of the Guardian Angels in preventing crime
• assess the effectiveness of the Guardian Angels in reducing fear of crime and increasing feelings of safety
• examine the perceptions of law enforcement and local government personnel about the Guardian Angels
• compare the Guardian Angels with other volunteer crime prevention groups with respect to leadership, organisational structure and membership.

Several data collection strategies were used including non-participant observation of the Guardian Angels' activities, record analysis of the Guardian Angels' patrol log books, the collection of crime statistics over a three-year period, qualitative interviews with a range of people including members and leaders of the Guardian Angels, law enforcement personnel and local government officials, and mail surveys and interviews, both telephone and face-to-face, with law enforcement personnel, Guardian Angels members and the public.

Findings

Overall, the findings indicate that, while the Guardian Angels may have had some impact in reducing the fear of crime in some segments of the population, it is unlikely that they had been successful in preventing or deterring crime. For example, the results revealed that 60 per cent of the citizens who knew that the Guardian Angels patrolled in their area reported feeling safer, but the number of citizens who knew of the Guardian Angels patrols ranged from nearly all those surveyed to less than half. Also, while a drop in the number of violent crimes reported was observed from pre- to post-introduction of the Guardian Angels patrols in an experimental area, an even greater drop was found in the control area where no Guardian Angels patrols occurred. Similarly, even though a significantly greater drop in the number of property crimes occurred in the experimental area than in the control area, the presence of other factors prevented any firm conclusions being drawn about the extent to which this could be attributed to the Guardian Angels patrols.

The findings also suggested that law enforcement and local government agencies appear neutral towards the Guardian Angels. The authors believe that these agencies may be unclear about the Guardian Angels organisation and purpose, and uncomfortable about its membership recruitment and screening practices. Indeed, discussion of the findings tends to focus on the relationship between the Guardian Angels, the police and local officials, and why these authorities generally appear reluctant to support groups such as the Guardian Angels.

While the evaluation's strength lies in the wide range of information collection procedures and sources used, it largely ignores the implications of the findings for further refining the programme and guiding its implementation in new areas. This may be because the evaluation team adopted a research orientation to the evaluation study rather than an action orientation.

Community policing of neighbourhoods

Form: Impact evaluation
Approach: Goal-based
Methods: Quasi-experimental

In evaluating this programme Kessler and Duncan (1996) conducted a goal-based, impact evaluation, which employed a quasi-experimental design. The purpose of the evaluation study was to assess the impact of community policing in reducing crime in four neighbourhoods of Houston, Texas. The programmes conducted in two of the neighbourhoods were initiated in response to citizens' complaints about the widespread use and availability of drugs and the associated crime, and were referred to as the 'War on Drugs' and the 'Link Valley Drug Sweep' campaigns. These programs involved the mobilisation of the community and the police to board up abandoned properties, close 'crack houses' and arrest drug dealers. Volunteer local citizens also cleaned up neighbourhoods by removing trash and hypodermic needles. One of the neighbourhoods was 'swept clean' and access was restricted to residents only.

In contrast, the Blocks Organising Neighbourhood Defence (BOND) programs in the other two neighbourhoods were neighbourhood watch programs. These programs focused on creating processes whereby citizens could work continually with police to improve safety and security. This involved:

• identifying residents who were willing to participate in the BOND programs
• training residents in crime prevention techniques
• teaching residents proper crime and suspicious behaviour reporting techniques
• establishing an organisation through which to communicate neighbourhood problems to police and residents
• identifying support activities.

As crime patterns developed, the use of networks and newsletters allowed residents to make informed decisions about how to deal with problems as they occurred, such as instituting foot patrols during certain hours and in particular areas where burglaries were occurring.

Study characteristics

The evaluation study used statistical data collected over the course of the implementation of the programme to assess its effectiveness. Data such as calls for service (calls made by citizens to police), reported crime (crime reported by citizens) and narcotics crime (arrests by police) was collected for the four neighbourhoods. Time series analysis was used to track the changes in these variables over a number of years before the programs were implemented through to approximately 12 months after their commencement, to allow lagged effects to be examined.

Findings

The results of the evaluation study did not allow any clear generalisations to be made about the effectiveness of the programs. None of the four programs produced results that conformed to the hypothesis that community policing programs would bring about a temporary increase in calls for service and reported crime, followed by a long-term decrease. For example, while the War on Drugs programme appears to have been successful in motivating citizen involvement and cooperation with police in that there were increases in calls for service and reported crime, the authors report no evidence for an eventual reduction in this neighbourhood's problems.

Similarly, while the citizens of the Link Valley Drug Sweep programme were highly motivated and worked closely with the police, there was no evidence of increased reporting to police, or indeed of any significant impact on the measures examined. In one of the neighbourhood watch programs, only calls for service showed any significant change by temporarily decreasing, the opposite of what was hypothesised for this type of community policing programme. In the second neighbourhood watch programme, reported crime increased gradually. The authors stated that overall 'the best conclusion may be that the programs had no impact' (Kessler and Duncan, 1996: 657).

The authors remark that police departments appear to not yet know the solutions to providing an effective community policing service. The authors noted limitations to the evaluation study, including difficulties associated with the quality of the records and documents about the programs, and concern over the reliability and validity of the measures used. Other considerations could include the use of a wider range of information collection procedures and sources to address a wider range of questions, such as changes in the type of crime, the seriousness of the crimes, who are the perpetrators of crimes and so on, so as to fully appreciate the effects of this type of programme.

Transferring crime prevention initiatives

Form: Process evaluation
Approach: Decision-oriented
Methods: Naturalistic

This study illustrates the use of a decision-oriented, process evaluation employing naturalistic methods (Crawford and Jones, 1996). The purpose of the study was to provide information that would assist in the successful transfer of the Kirkholt Anti-Burglary Project to Tenmouth, a city in the south-east of England.

The Kirkholt Project had three main aims, which were also adopted in the Tenmouth Project:

• reduction of burglary in the targeted area
• delivery of the crime reduction mechanisms through a multi-agency approach
• eventual local community ownership of the project.

The authors summed up the aims of the project as an attempt 'To devise a joint plan between the statutory and voluntary agencies, commercial sector and the community towards frustrating the activities of burglars and the fear they provoke - and make it work!' (Crawford and Jones, 1996: 25).

Study characteristics

The report asserts that evaluation in the area of community crime prevention is preoccupied with determining whether the programme objectives have been achieved, using before and after surveys. In the view of the authors, this preoccupation prevails despite wide recognition that any outcome in this area will be determined by its policy context and implementation processes. Accepting this means accepting that the transference of community crime prevention initiatives (for example from one location to another) will need to be informed by a thorough understanding of the 'mechanisms', 'contexts' and 'outcomes' of these preventive efforts and their interrelationships through process, or interactive, evaluation (see Pawson and Tilley, 1994).

The process evaluation of the Tenmouth Project included the use of observational methods and semi-structured interviews. This dual data collection strategy enabled the evaluation team to assess the gap between what was said would be done and what was actually done, an important consideration given the purpose of the evaluation study.

The key issues explored in the study were informed not only by investigating the similarities and differences between Kirkholt and Tenmouth (for example in terms of social and demographic factors), and what is meant by success (for example 'Is inter-agency collaboration an end in itself or a means to achieving an end?'), but also by the evaluation team's knowledge of the social processes that constitute crime prevention efforts. These key issues included the:

• importance of understanding the structure and decision-making processes of community crime prevention initiatives
• nature of community participation
• extent of implementation
• nature and extent of inter-agency involvement.

Findings

Among the main findings related to the process of transferring the Kirkholt Project to Tenmouth was an assumption that communities, including the commercial sector, would be intimately involved in the Project (Crawford and Jones, 1996). The authors point out that 'there was, however, little community involvement during the life of the project, largely because the technical parameters of the project had already been set by Kirkholt' (Crawford and Jones, 1996: 34). Moreover, communities with significant crime problems are typically those which lack the necessary social structures to support community ownership, yet these same communities are being asked to help themselves in the prevention of crime. In summing up the results of this evaluation study, Crawford and Jones (1996) argue that in the quest for a quick fix, community crime prevention technologies, including evaluation, have lost sight of the complex social relations in which programs become embedded.

Creating safer cities

Form: Impact evaluation
Approach: Decision-oriented
Methods: Quasi-experimental

In this example a decision-oriented, impact evaluation study, using quasi-experimental methods is illustrated. The purpose of the study was to contribute to the development of a wide range of projects established in 20 local areas in England, and funded through the Safer Cities Programme (SCP). Ekblom states that 'the function of the SCP evaluation is to contribute to the development of SCP... in particular through the decision to continue with the programme and the direction of that continuance' (1992: 50).

In describing the nature of the SCP, Ekblom (1992) underlines how it drew on existing experience in crime prevention initiatives to identify three main objectives. They were to:

• reduce crime
• lessen the fear of crime
• create safer cities within which economic enterprise and community life could flourish.

A wide range of projects had been funded to achieve these objectives incorporating both situational (such as target-hardening) and offender-oriented (such as educational initiatives) preventive action.

Study characteristics

In the evaluation study, two principal types of data were collected, one focusing on the extent to which the SCP projects were implemented according to their blueprint (process evaluation); and the other on the extent to which the three main objectives were being met (impact evaluation). Ekblom notes that in determining the extent to which the SCP objectives were being achieved, the evaluation study faced

'extremely difficult conditions which centre on having to detect the effects of a set of preventative schemes which are very diverse in size and nature (some being extremely modest), which may or may not be successfully implemented, which start up at different times, which possibly overlap, and whose locations are not merely scattered but unknown in advance... (and) these effects have to be detected against a background of non-SCP preventative activity in the SCP areas themselves' (1992: 37).

Consequently, two complementary types of comparison were adopted, involving comparisons with a number of areas (cities and boroughs) which were not participating in the SCP, and comparisons within the SCP project areas. Both types of comparison included investigating before and after changes within the constraints faced by most quasi-experimental methods.

Findings

Ekblom (1992) is concerned with the rationale for focusing and conducting the SCP evaluation rather than the findings of the evaluation study. The focus is on a detailed explanation of one approach to overcoming some of the difficulties faced when evaluation studies are conducted to inform 'real time' policy level decisions about big 'P' programs such as the SCP. In this study the difficulties included what counted as SCP action (ranging from mere provision of advice about a project through to core schemes fully funded by the SCP) and how to design out, as far as possible, the influence of other preventive initiatives that are funded to address the same problems. Ekblom concludes by stating

We have also attempted to be honest regarding the uncertainty that will inevitably surround the final results, and frank about the risk of measurement failure which we have striven to keep to a minimum within the resources available to us (1992: 50).

Reducing the risk of leaving school early and developing offending behaviour

Form: Process evaluation
Approach: Illuminative/responsive
Methods: Naturalistic

This evaluation study (Iuliano, 1995) represents a process evaluation form, utilising an illuminative/responsive approach and naturalistic methods, embedded within a participatory action research design3. The purpose of the evaluation study was to bring to light the experiences of all the participants in this school-based programme in order to develop new directions, as well as to provide information to other schools that might be interested in implementing a similar project.

The original project concept of recruiting unemployed young people to act as mediators or supervisors to reduce fighting and bullying in schools was developed in France (Iuliano, 1995). It was further developed in South Australia where young unemployed adults acted as mentors to young truants who were at risk of leaving school early and developing offending behaviour. Iuliano states that:

A mentor's role will be to support school attendance, to coach students to be successful at school (improved learning outcomes), to act as a behaviour coach and a link and advocate between the family and school (1995: 22).

Study characteristics

The specific illuminative/responsive approach adopted for the evaluation study emerged from a view about sustainable change in how programs are designed and implemented. In essence, this approach emerged from understanding the experiences the participants brought to light and the new directions created from these experiences. Evaluation data were obtained from three primary sources:

• workshops for mentors and other programme participants (such as family members)
• interviews with the young people experiencing difficulty in attending school
• document and records analysis.

Interview data were initially recorded in 'mindmaps' and later translated into text for the review process. The uniqueness of adopting a participatory action research design to obtain these data is also reflected in the interview process where interviewees (referred to as a speaker) described their experiences in terms of 'reflection' about them, 'interpretation' where the person is invited to make sense of these experiences, and 'review' with the focus being on endorsing the authenticity of the data.

Findings

The findings of the evaluation study brought to light several issues that are not usually addressed by other approaches, particularly those that restrict their focus to the extent to which programme goals have been achieved. The findings included:

• some mentors had been working with students in schools which appeared to have no knowledge of the programme or that mentors were working in the school
• a second group of mentors recruited for the second round of the programme were not subjected to the formal processes of selection, induction and training required in the first round, which may explain changes in satisfaction with the programme.

Examining the extent to which programme activities are implemented as intended is particularly important when the effects of programs are being evaluated - so as to establish that the observed effects are a result of the programme intended to be implemented, not some other version of it.

The participatory action research design employed in this evaluation study starts with the assumption that the individuals directly affected by social interventions are in the best position to reflect on their experiences and develop new ways of improving the processes and outcomes of value to them. This involves assisting these key stakeholders to develop the evaluation questions, and it ends with a process of negotiated change that addresses their interests and expectations.

Reducing recidivism for illegal vehicle use

Form: Process evaluation
Approach: Illuminative/responsive
Methods: Naturalistic

The evaluation model which guided this study combined a process evaluation form, illuminative/responsive approach and naturalistic methods, but differed from the Iuliano (1995) study in not adopting a participatory action research design. The purpose of the evaluation study was to improve a series of youth camps for young offenders focusing on learning new skills and increasing self-esteem and self-discipline. This, in turn, was expected to encourage lawful and constructive behaviour, and so some preliminary impact data also were obtained (Wundersitz, 1994).

The camps, held in South Australia, recruited 10 Indigenous youths, seven of whom were serving a custodial order and three community-based orders. All 10 youths had a history of illegal vehicle use and had been involved in high-speed car chases. From the outset, there was general recognition that the goals of the programme could not be achieved through a single camp or even a series of camps. The camps were seen to be only one part of a broader strategy aimed at decreasing the likelihood of recidivism for illegal use of motor vehicles and associated offences. Secondary objectives of the programme included establishing closer working relationships between the police and Department of Family and Community Services personnel to facilitate improved inter-agency cooperation and understanding of youth offending.

Study characteristics

The evaluation study focused on the experiences of the youth, police and various other participants in the camps with the aim of improving how the camps were run and their perceived impact. The evaluation data primarily were obtained through interviews before and after the camps. The rates of offending before and after the camps were also obtained from records held by Family and Community Services and the police, although the report notes some unexplained discrepancies in these data.

Findings

The interview data suggested that the camps were regarded positively by the youths and by the police officers and the staff at the centre where most of the youths were serving a custodial order. Indeed, the police were seen as 'good blokes'. Moreover, experiences such as abseiling in the first camp, having family members to talk to in the second camp, and an opportunity to see how traditional Indigenous people lived in the third camp, were all viewed as positive experiences by the young people. The police officers, on the other hand, underlined the importance of each young person being given an opportunity to accept responsibility for themselves and others. The police also felt that they and the young people had increased their understanding of each other.

One of the strengths of evaluation studies which adopt an illuminative/responsive approach is that suggestions for improvement are closely tied to the perceptions of the people directly affected by the programme. In this case, the suggestions included ensuring that there was continuity of both individual police and youth participation in the three camps, a greater involvement of Indigenous adults who were related to the young people concerned, and the involvement of the young people who participated in the pre-planning stages of the camps. Indeed, in view of the purpose of the evaluation study, namely to improve how future camps would be run, the views of the key stakeholders clearly were an essential component.

Violence prevention through informal socialisation

Form: Impact evaluation
Approach: Goal-based
Methods: Quasi-experimental

In this evaluation study an impact evaluation form, goal-based evaluation approach and quasi-experimental methods were employed. The purpose of the study was to test the effectiveness of the South Baltimore Youth Center programme for prevention of violence, delinquency and substance abuse, and refine it where appropriate. South Baltimore is described as a city where joblessness is a way of life, and where people get through the day by addiction, alcohol and TV sports (Baker, Pollack and Kohn, 1995).

Study characteristics

In the report the authors discuss the use of 'proactive' evaluation, involving:

• reviewing the programme evaluation literature to see what types of programs work in these circumstances
• abstracting the common features of these programs
• continuous testing and refining of the programme model.

They point out the importance of understanding the programme rationale (or theory of action, cf Stake, 1967), seeing this as an important aspect of any evaluation study that seeks to improve the efficacy of a social programme. In this study, the major components of the programme, including the provision of a safe haven, empowering the young people concerned in decision-making and the provision of mentors, were assumed to achieve their impact through increasing feelings of self-efficacy, self-esteem, and peer, family and community bonding. The programme therefore is offender-oriented.

The evaluation study gathered both process and outcome data using a range of tests, questionnaires and interview schedules over a three-year period. Programme implementation data focused on how the project involved youth, the nature of project activities, how often each participant engaged in the activities, and the problems encountered in implementation. A quasi-experimental design with three matched, untreated comparison groups was used to obtain programme outcome data. Programme success in reducing violence, delinquency and substance abuse was investigated in terms of the causal linkages articulated in the programme rationale; for example, that increased peer bonding is an important intervening variable in reducing this behaviour.

Findings

The evaluation study findings are presented as clearly indicating the success of the programme in achieving both its implementation and outcome objectives. For the process evaluation, the authors conclude that the programme provided a place of sanctuary or escape from high-risk home environments as well as excelling in the extent to which staff were able to create a family-like atmosphere. Similarly, from the pre-test, post-test comparisons for the intervention and comparison groups, the authors report that participants' scores improved relative to the comparison sample on 86 per cent of the 15 outcome measures, thereby demonstrating a positive programme impact.

As with many evaluation studies employing a longitudinal design, the attrition rate was substantial, with comparisons over time being based on only 73 of the original 200 individuals included in the study; namely, 38 from the programme and 35 from the comparison groups. In addition, although the authors report collecting a wide range of process data, none of these data is provided to support the conclusions about programme success.

Conclusion

The eight evaluation studies outlined here illustrate the use of a variety of evaluation models. Nevertheless, they represent a limited range of distinct evaluation models, with only two of the five evaluation forms and three evaluation approaches being used. This reflects the limited number of studies and the difficulty of access to reports of studies using other evaluation models.

For each evaluation study a judgment about the appropriateness and utility of the evaluation model used should depend upon the extent to which it was compatible with the nature of the programme and its social, cultural and political context, and the extent to which the information needs and priorities of the key stakeholders and other audiences were met by the information provided. Thus, these should be the primary considerations in the determination of the model to be used in the planning and conduct of an evaluation study.