The purpose of this section is not to deal with the technical aspects of evaluation design and implementation. You may be able to locate a professional evaluator with the experience and skills necessary to accomplish the evaluation objectives contained in your proposal and negotiated with your funding agency. What we share with readers in this section are specifics about evaluating outdoor/experiential programs, some history about NIYLP's evaluation efforts and approaches and some tools that we have found useful. Please feel free to contact NIYLP's Evaluation Coordinator, Dr. Susan Carter, to discuss specific evaluation issues for your project. susancarter@hughes.net
As you begin your program evaluation, consider the following:
· Project Venture is very much a process-oriented, interactive program rather than a content-oriented, classroom type of program. Since so much of what happens must be observed in order to be understood, it is important that your evaluator be available in the field in order to provide you with objective feedback as the program is implemented. Simply stated, we do not feel that this program can be evaluated from behind a desk!
· Evaluators of this program should be knowledgeable about the guidelines and safety considerations provided by groups like the Association for Experiential Education in order to provide informed feedback from the field regarding the efficacy and safety of program implementation.
· As a friendly outsider or “critical friend”, the program evaluator must develop relationships with stakeholders that reflect trust and respect. This is especially important in Native American and other indigenous communities where, historically, researchers have not always acted in the best interests of the people they were hired to serve.
· Think of your evaluator as one of the “story tellers” for your project. Make sure they have both quantitative and qualitative research skills. Our experience has shown us that it is often the wonderful qualitative descriptions and analysis of project activities that finally win over community members who may be reluctant to participate fully in evaluation requirements.
The following list provides a summary of the key components and ideas that should be part of your evaluation planning and your discussions with your evaluator.
Procecss evaluation answers the question, “Is the program being implemented as planned?” This may include regular site visits of the program in action to observe and talk with staff and youth. It may also include tracking the number of youth enrolled and descriptions of activities delivered by staff. You may also want to ask for feedback from stakeholders (participants, staff, parents, teachers, principal, etc.) about the program. This is most useful if you don't wait until the program is over! NIYLP has created a Session Evaluation form which you might want to adapt.
Outcome evaluation answers the question, “How have participants changed as a result of the program?” This almost always includes collecting information at regular intervals about participants. You might administer a paper and pencil survey at the beginnning and end of the program to see how participants have changed their attitudes, skills, knowledge, and behavior. You might also conduct interviews with individual or small groups of youth participants at the beginning and end of the program year. NIYLP can suggest a variety of survey and other instruments to support your outcome evaluation.
Use data to improve program efforts. Share good news, as well as areas in need of improvement.
Send regular reports (oral and/or written) to funding agencies, schools, community agencies, parents and youth to ensure their continued support and interest. Brag about achievements. Enlist their help where needed.
Present data in relatively “raw” form and ask staff and other stakeholders what it means.
Your evaluator should attend regular staff meetings, conduct participant observations and informal interviews, set up a listserv, etc.
The evaluator works for the program, not the funding agency. She should let program staff know when and what she’s communicating to the funding agency and others.