Guidelines for Evaluating Community Crime Prevention Projects
STAGE 6 - MANAGING THE EVALUATION
What you will do in Stage 6
- Brief the Evaluator
- Finalise the evaluation method, timetable and budget
- Monitor the evaluation
- Keep stakeholders informed
- Review any preliminary findings
- Revise the Action Plan
- Check drafts of the final report
BRIEFING THE EVALUATOR
1. Let the Project Committee, staff and all relevant stakeholders know:
- the date the Evaluator will begin
- the date they will meet the Evaluator.
This will help assure stakeholders that their views will be heard by the Evaluator and that they will be fully involved in the evaluation.
2. Make a copy of every document the Evaluator needs in order to form a complete understanding of the project and the evaluation.
3. Arrange a meeting with the Evaluator to brief them on the project and the evaluation:
- introduce the Evaluator to the people they will work with
- let stakeholders tell the Evaluator first-hand about the information they need and how they will use it
- let the Evaluator know constraints he or she will have to work within
- let the Evaluator hear about people's perceptions and expectations
- discuss with the Evaluator the best way of doing the evaluation and reporting the findings
It is important that the Evaluator meets all stakeholders and that no-one feels left out. Follow the guidelines for managing stakeholders at the end of Stage 2.
Getting the Evaluator to meet all of the stakeholders will help the Evaluator see the project from all angles, not just the views of those responsible for administering the project. - take the Evaluator to visit the project site(s).
4. Brief the Evaluator on the project and the evaluation.
This briefing should include:
- the reasons the project was set up
- the political context of the project and the evaluation
- stakeholders and their involvement in the project
- the organisation and administration of the project
- services the project provides and how it provides them
- why the evaluation is needed
- what the various stakeholders hope to learn from the evaluation
- what action the stakeholders want to take using the information from the evaluation.
5. Give the Evaluator all the documents you copied in Step 2. Explain what each contains and put the information into context.
6. Introduce the Evaluator to the Project Committee, key stakeholders and any project staff.
This is a crucial step. Establishing good relations between the Evaluator and the stakeholders from the outset will greatly enhance the access stakeholders get to information and experience.
It will also do much to ensure the findings are seen as credible and the Evaluator's recommendations are acted on.
7. Get the Evaluator to prepare a report on their discussions with stakeholders.
Use this to check that the Evaluator understands all of the issues affecting the evaluation. Correct any misunderstandings.
WRITING THE EVALUATION METHOD, TIMETABLE AND BUDGET
8. Once the Evaluator has met all the stakeholders and relevant staff, they should write the evaluation method, timetable and budget.
The evaluation method is a description of what the Evaluator will do to evaluate the project and the materials they will produce. It includes:
- the research method-for instance, if people are to be interviewed, the Evaluator may use focus groups, or one-to-one interviews
- who will be interviewed or surveyed
- what documents will be analysed
- if a survey is to be undertaken, the sample size, acceptable error rates, reliability, and how participants will be recruited
- how participants will be recruited, interviewed and debriefed
- interview and survey questions
- data sheets or interview protocols
- whether any confidential information will be needed and, if so, how the Evaluator will ensure the information remains confidential
- arrangements and approvals required for interviewing children, young people, people affected by trauma or the effects of crime, or people suffering from a mental illness
- a description of ethical standards and how the Evaluator will observe them-a good starting point is the Australasian Evaluation Society's Ethical Guidelines
- how the information gathered will be analysed
- how recommendations will be developed and reported
- how the Evaluator will report their findings and recommendations.
The evaluation timetable outlines when each activity in the evaluation will take place. It will be based on PLANNING SHEET 4B. It should include all project milestones and the dates that material will be delivered, including dates on which:
- the Evaluator will be briefed on the project
- the Evaluator will summarise the initial discussions, so the project can check that the Evaluator understands what is required
- the Evaluator will start and finish developing the evaluation method
- the Project Committee will check the evaluation method
- project participants and others will be recruited for interviews, surveys or other activities
- the Evaluator will begin and end observations or data collection
- the Evaluator will visit sites where the project operates
- the Evaluator will report any preliminary findings
- the Evaluator will report on progress to the Project Committee
- the Evaluator will deliver written reports
- the Evaluator will deliver a draft of the final report and any presentation materials
- findings and recommendations will be announced, the final report released, and presentation materials used.
The evaluation budget will be based on PLANNING SHEET 1B. It lists the costs the Evaluator anticipates such as:
- staff time
- office and support costs
- travel and accommodation, meal and living expenses
- equipment purchases
- recruitment costs for interviews and surveys
- data analysis costs
- venue hire
- production of reports and presentation materials.
The Evaluator will need to develop the method, timetable and budget with input from you. In longer evaluations, the timetable and budget may evolve as the evaluation progresses. Once the method, timetable and budget have been developed, they should be documented so the Project Committee can check and approve them before the evaluation begins.
Because there are many ways of evaluating community crime prevention projects there is no one way of checking the methods, timetable and budget. The following is a guide to the main items only.
9. Check:
- the method and timetable are sensitive to stakeholders' needs
Compare the Evaluator's proposal with PLANNING SHEET 2C. - the method will give all stakeholders the opportunity to contribute to the evaluation
- the evaluation questions will produce information that stakeholders need and want, and enable them to make decisions and act
Compare the Evaluator's proposal with PLANNING SHEET 2C. - whether the Evaluator will require access to confidential information and, if so, how they will keep the information secure
- that the evaluation can realistically be completed with the time and resources available
- that the Evaluator will obtain authority to interview children, young people, people affected by trauma or the effects of crime, or people suffering from a mental illness
- regular contact is scheduled between the Evaluator and project manager
- the budget is within the resources allocated by the project and meets the project's accounting requirements
- the way the Evaluator intends to report the findings and recommendations will meet stakeholders' needs.
Compare the Evaluator's proposal with PLANNING SHEET 2C.
10. Discuss any changes with the Evaluator and get them to make any necessary modifications.
11. Ask the Project Committee to approve the method, timetable and budget.
MONITORING THE EVALUATION
Once the Project Committee has approved the evaluation method, timetable and budget, the Evaluator will begin the evaluation.
12. Have regular meetings with the Evaluator during the evaluation to:
- check resources are being used and accounted for properly
- check the timetable is being met
- resolve any disputes between stakeholders and the Evaluator
- help the Evaluator resolve any ethical dilemmas
- help the Evaluator access information, stakeholders or participants
- help the project learn from the Evaluator and evaluation process
- gain insights into how the project is running
- give the project early warning of unexpected findings so you can inform stakeholders, revise the Action Plan, refocus the evaluation or renegotiate the budget.
13. Use PLANNING SHEET 6 to keep a record of all meetings with the Evaluator. Include a summary of:
- any decisions made and the reasons for them
- any disputes or problems that arose and how they were resolved
- any key findings the Evaluator presents along with any consequences for the project.
KEEPING STAKEHOLDERS INFORMED
14. Keep stakeholders informed, as set out in the evaluation timetable recorded on PLANNING SHEET 4B.
Depending on the project, you may need to keep stakeholders informed about progress of the evaluation to help maintain their commitment to it and be ready to act when the findings become available.
REVIEWING PRELIMINARY FINDINGS
Whether there are preliminary findings will depend on the evaluation.
15. Check the findings:
- will provide the information stakeholders need and want
- will answer the questions stakeholders have about the project
- will be in a form that will allow stakeholders to make decisions and to act
- are consistent with the type of information the Evaluator has been commissioned to produce.
If they are not in an acceptable form, you must tell the Evaluator and the Project Committee immediately. If necessary, you may have to suspend the evaluation. It is important that funds are not spent unnecessarily and the goodwill you have built up with stakeholders is not lost.
You must make sure the findings will be credible and useful to stakeholders, otherwise the entire evaluation process will be undermined.
16. Use PLANNING SHEET 6 to record any decisions you made concerning preliminary findings.
REVISING THE ACTION PLAN
17. Get all of the completed PLANNING SHEET 4A.
These record the draft Action Plan.
18. Review the Action Plan with input from:
- the Evaluator
- the Project Committee
- any preliminary findings
- reports supplied by the Evaluator.
19. Make any necessary changes to the Action Plan.
20. Check any changes with affected stakeholders.
If the changes mean stakeholders will no longer get information they expected, discuss with them what information you do expect will be available, and whether this will be adequate for their needs. You may need to negotiate with the Evaluator to ensure all the information required will be produced.
21. If you make a change to the Action Plan, make a copy of the relevant PLANNING SHEET 4A, and write the changes on them.
CHECKING DRAFTS OF THE FINAL REPORT
The final report will include all the materials the Evaluator will provide to make the findings public and help the project act on the recommendations. These materials are listed in PLANNING SHEET 2C.
In all but the smallest evaluations, the Evaluator should prepare a draft of their final report. You and the Project Committee should check the draft report and any presentation materials to make sure they will meet the needs of the project and stakeholders. You may also need to help the Evaluator manage any controversial topics.
22. Check the draft final report and all presentation materials meet the needs of stakeholders listed in PLANNING SHEET 2C.
23. Check the draft final report contains:
- an overview of the project, the reasons the project was set up, and how it was expected to prevent or reduce crime
- the reasons the project needed to be evaluated
- the method used to evaluate the project
- what data the Evaluator collected
- how the Evaluator interpreted or analysed the data
- the action the Evaluator recommended the project and stakeholders take
The Evaluator may also suggest alternative courses of action. This part of the report should explain any recommendations that are not obvious.
- political issues that would need to be managed when implementing the recommendations
- the finances, resources, time and activities required to implement the recommendations
- the outcomes the project could expect from following or rejecting the recommendations.
24. Check that the recommendations are realistic.
Specifically, check that:
- the project has the staff, resources, funds and skills to implement the Evaluator's recommendations
- the stakeholder politics can be managed.
25. If there are any changes required, ask the Evaluator to make them before the project accepts the final report.
Changes that may need to be made include:
- making the recommendations more practical
- supplying more or less of the findings from the evaluation
- explaining the reasons behind the recommendations
- expressing politically sensitive information or issues in ways that will not unnecessarily upset particular stakeholders or interested parties
- keeping the identity of people confidential
- ensuring confidential information is not made public.
It is unethical to ask the Evaluator to delete particular findings or recommendations. If you feel the Evaluator has made a mistake, you must discuss this with them as soon as possible so potential problems can be addressed.