PEEC Home Page | Research | Tools | Resources | Reports | Forum | Subscribe | Search 75 Eval Tools from: Bay Area Environmental Education Evaluation Learning Communit Below is a matrix of 75 evaluation tools and resources compiled in July of 2004 by the Bay Area Environmental Education Evaluation Learning Community (BEEC). Nearly all the resources can be accessed directly from this page either by clicking the link on the title of the tool, or by downloading the document from the very bottom of this page. First, an introduction from the people who compiled this resource.... The Bay Area Environmental Education Evaluation Learning Community July 2004 Dear Colleague: Better together. These two words convey what participants in the Bay Area Environmental Education Evaluation Learning Community have demonstrated through their collaborative activities since June 2003. More than 25 environmental education program providers, evaluation specialists and foundation officials have engaged in a series of learning experiences to: • Build a common base of knowledge about environmental education research and evaluation • Develop a logic model to explain how environmental knowledge and awareness can result in pro-environmental behavior • Create a plan, based on the logic model, to increase capacity for evaluation of environmental education in the San Francisco Bay region. Congratulations to all the learning community participants for this important contribution to advancing environmental education in the Bay Area and beyond. Special thanks to the learning community facilitator, Jack Chin of Blueprint Research & Design, for guiding the learning process. We are also indebted to the many evaluators who have shared their expertise and tools with us. Use of this CD-ROM’s contents is encouraged by the sponsors of the learning community to spread the knowledge that has been collected and synthesized. However, as you do so, we ask that you follow these guidelines: Evaluation Tools Database. When using any of the tools contained in the database, please provide proper credit to the author(s) of the particular tool. In most cases, the appropriate information can be found in the author field of the database and/or on the pdf file or web-site to which the database is linked. For further information, please feel free to contact any of the sponsors or the learning community facilitator: Carol Welsh Gray Executive Director Center for Venture Philanthropy Peninsula Community Foundation 2744 Sand Hill Road Menlo Park, CA 94025 P: (650) 854-5566 x251 E: carolg@cvp.pcf.org  Don Jen Program Officer Marin Community Foundation 5 Hamilton Landing, Suite 200 Novato, CA 94949 P: (415) 464-2525 E: djen@marincf.org  Arlene Rodriguez Environment Program Officer San Francisco Foundation 225 Bush Street, Suite 500 San Francisco, CA 94014 P: (415) 733-8500 E: amr@sff.org  Bob Uyeki Program Officer East Bay Community Foundation 200 Frank Ogawa Plaza Oakland, CA 94612 P: (510) 208-0809 E: buyeki@eastbaycf.org  Jack Chin Senior Analyst Blueprint Research & Design, Inc. 720 Market Street, Suite 900 San Francisco, CA 94102 P: 415.677.9700 E: jack@blueprintrd.com  We hope that you will find this knowledge useful for improving environmental education practice and results. Learning Community Sponsors East Bay Community Foundation Marin Community Foundation Peninsula Community Foundation The San Francisco Foundation Evalution Tools Database Tool Name Publisher/Author Brief Description Notes Matched against criteria Tool Type "A Scale for Measuring the Impact of a Primary School Environmental Education Programme on Parents' Attitudes" Reid, I. A 1999 article in the International Journal of Environmental Education and Information 31(2):1-12.  Article must be obtained through the journal. This tool has not been reviewed by Blueprint, but it may be useful. 2003 EETAP Leadership Clinic Goals, Measurable Objectives, Outcomes, Outputs and Measurement Tools National EE Advancement Project (NEEAP) A logic model for a leadership clinic training program, with the tools to be used to measure each of the objectives.   Could be used as a sample for others engaged in EE related  training programs. Criteria not applicable. Not applicable Logic model A checklist for effective communications North, Rick/McKenzie-Mohr and Associates A checklist for effective communications.  Online resource providing a variety of methods to enhance the effectiveness of the communications EE programs produce. Audience-centered Clear objectives Research-driven message development Multiple levels of engagement Checklist A checklist for the critical evaluation of informal environmental learning experiences Ballantyne, R. R. & Uzzell, D. L. An article in the 1994 International Journal of Environmental Education and Information 13(2):111-124.  The article must be obtained through the journal. This tool has not been reviewed by Blueprint, but it may be useful. Applied Behavior Change Framework William A. Smith, Academy for Educational Development (AED) Booklet describes the ABC Framework as a tool intended to help program managers design, develop, and evaluate interventions leading to large-scale behavior change. It integrates lessons learned from communication planning, social marketing, and behavioral science. Includes three charts: Integrated Planning Network; Behavioral Constellation; and Applied Behavior Change Framework This tool has not been reviewed by Blueprint, but it may be useful.  This framework must be ordered through AED. Other Assessing Your Collaboration: A Self Evaluation Tool Lynne M. Borden and Daniel F. Perkins, in the Journal of Extension A collaboration checklist of 12 factors Open communication Self-governance Checklist Attitude Transfer Scale, Intermediate Form (Grades 2-6) National Association for Humane and Environmental Education and Western Institute for Research and Evaluation Group administered test "designed to assess attitudes toward people that might be affected by animal-focused humane education programs.  In other words, this test can help you determine if children's positive attitudes toward animals transfer or generalize to other people."  It uses seven photos depicting elementary-school experiences, to which students respond with varying degrees of kindness.  For grades 2-6. Survey Attitude Transfer Scale, Primary Form (Grades K-1) National Association for Humane and Environmental Education and Western Institute for Research and Evaluation Individually administered test "designed to assess attitudes toward people that might be affected by animal-focused humane education programs.  In other words, this test can help you determine if children's positive attitudes toward animals transfer or generalize to other people."  It uses seven photos depicting elementary-school experiences, to which students respond with varying degrees of kindness.  For grades K-1. Survey Bay Area Indicators:  Measuring Progress Toward Sustainability Bay Area Alliance for Sustainable Communities Regional report with data on Bay Area indicators on Natural Assets (Protected Land, Brownfields, Water Use per Capita, Ecological Health of the Bay, Ecological Footprint) and Resource Use (Energy Use, Carbon Emissions, Ozone, Particulate Matter, Waste Disposal and Diversion).  The other categories of indicators are:  Sustainable Economy, Housing Supply, Transportation, Neighborhood Integrity, Educational System, Community Health and Safety, and Local Government Finance. Criteria not applicable Not applicable Other Benchmarking Best Practices in Environmental Education LaFrance Associates LLC and Peninsula Community Foundation Organizational self-assessment tool with benchmarks for the following types of programs:  programs designed to affect policy change, designed to change business practices, engage volunteers in environmental action, for children and youth, and to train teachers. Action-oriented Learner-oriented Connects with place/community Self-assessment tool Checklist for Program Evaluation Planning in "Measuring the Success of Environmental Education Programs" Gareth Thomson, Canadian Parks and Wilderness Society;  and Jean Hoffman, Sierra Club of Canada, BC Chapter; adapted from Carter McNamara's Checklist for Program Evaluation Planning Short checklists for evaluation purpose, audience, evaluation type, types of information needed.  The publication has an appendix with guidance on the pros and cons of different evaluation instruments, as well as some instrument design tips. Clear purpose Feasibility Checklist Checklist to Facilitate the Development of Culturally and Linguistically Competent Primary Care Policies and Structures The National Center for Cultural Competence (Georgetown University Center for Child and Human Development) A self assessment tool from an organizational capacity angle.  Much of it is health and human service provider-oriented, but some of the elements are transferable. Capacity for Self-Assessment Consciousness of cultural dynamics Adaptability to Diversity Checklist Children’s Attitude Toward the Environment Scale Malkus, A. J. & Musser, L. M. A 1994 article in the Journal of Environmental Education, 25(3), 22-26.  Article must be obtained through the journal. This tool has not been reviewed by Blueprint, but it may be useful. Children's environmental attitude and knowledge scale:  construction and validation Leeming, F. C. & Dwyer, W.O., and Bracken, B. A. A 1995 article in The Journal of Environmental Education, 26 (3), 22-31.  Article must be obtained through the journal. This tool has not been reviewed by Blueprint, but it may be useful. Community Mapping Program Educator Survey Place-based Education Evaluation Collaborative (PEEC) A survey on a community mapping program's curriculum, for the participants who are adult educators.  For more information:   http://www.PEECworks.org Connects with place/community Survey Community Mapping Program: Community Partner Survey Place-based Education Evaluation Collaborative  (PEEC) A survey for a community mapping program's community partners, which are organizations part of a wider collaborative.  For more information:   http://www.PEECworks.org Connects with place/community Survey Conducting a Cultural Competency Self Assessment Dr. Dennis Andrulis Developed for health care delivery organizations, but contains some relevant questions.  Questionnaire in three sections:  ethnic/cultural characteristics of the staff and board; organizational approach to accommodating diversity needs; and links to communities.  Contains guide to conducting the self assessment. Value diversity Capacity for self-assessment Consciousness of cultural dynamics Self-assessment tool CO-SEED Community Member Survey Place-based Education Evaluation Collaborative (PEEC) A survey for a community's impression of their public schools and the Community Based School Environmental Education (CO-SEED) program within the schools.  For more information:   http://www.PEECworks.org Connects with place/community Takes a systems approach Survey CO-SEED Project Student Survey Place-based Education Evaluation Collaborative (PEEC) A survey used by the Community Based School Environmental Education (CO-SEED) program for students.  The survey asks about their schools and communities.  For grades 4-12.  For more information:   http://www.PEECworks.org Connects with place/community Promotes wonder Action-oriented Survey CO-SEED Project Teacher/Staff Survey Place-based Education Evaluation Collaborative (PEEC) A survey for adult educator participants that asks for feedback on curriculum and a few program outcomes for the Community Based School Environmental Education (CO-SEED) program.  For more information:   http://www.PEECworks.org Connects with place/community Survey Cultural Competency Self Assessment Tool Ministry for Children and Families, British Columbia Tool for measuring organizational level capacity. Institutionalization of cultural knowledge Adaptability to diversity Self-assessment tool Curriculum and Compendium Project Compendia Evaluation Tool California Regional Environmental Educaion Community (CREEC) Developed by CREEC for their Curriculum and Compendium Project.  Open-ended questions that the project's reviewers used to evaluate the curricula, which is focused on communities, specifically the interplay of natural and built environments.  Can be used across different age levels. Learner-oriented Action-oriented Promotes wonder Takes a systems approach Other Development of an Evaluation Checklist for Communicating about Environmental Education Dark, Melissa J and Holsman, Robert A scholarly paper that contains a checklist for communicating about EE. Audience-centered Research-driven message development Clear objectives Checklist EETAP States Program Workplan and Evaluation plan National EE Advancement Project (NEEAP) A workplan for a capacity-building program that strengthens EE educator programs, including an evaluation plan.  This tool could be useful for those who are evaluation planning for their capacity building efforts.  For adults. Criteria not applicaable Not applicable Evaluation plan Environmental Education Materials: Guidelines for Excellence and Workbook North American Association for Environmental Education (NAAEE) Guidelines for EE programs can help educators link their practices to the guidelines and provide some indicators to explore and measure.  Workbook leads educators through the process of using the Guidelines for Excellence to assess EE materials. Action-oriented Takes a systems approach Learner-oriented Connects with place/community Guidelines Environmental Protection Indicators for California California Environmental Protection Agency Approximately 90 statewide indicators in the following categories:  air quality, water, waste management, pesticides, transboundary issues, human health and ecosystem health.  Some indicators are Bay Area specific. Other Evaluating Collaboratives:  Reaching the Potential University of Wisconsin Extension Manual (200 pages) on evaluating collaborations, segmented into different types of evaluation:   self-interest, feasibility, process, and outcomes.  Includes checklists, collaboration milestones, and examples of logic models, evaluation questions, indicators and a "collaborative functioning scale" throughout the guide. Open communication Self-governance Mutual respect, understanding and trust Shared purpose Appropriate composition Synergistic Climate Other Evaluation Capacity Self Assessment Tool Applied Survey Research on behalf of Family Resource Center of Santa Cruz County Short survey that asks questions about the evaluation data collection methods of an organization. Feasibility Survey Evaluation System for Experiential Education (ESEE) UC Berkeley Service-Learning Center A series of pre-post surveys for measuring civic responsibility, across the following student development domains: academic achievement, career development, social development, personal development, civic participation, and ethical development..   Some surveys are in Spanish.   The tools can be useful for EE experiential programs, and/or for programs that have increased civic responsibility, academic achievement, personal development, etc as outcomes.  "The UC Berkeley Service-Learning Center has developed the Evaluation System for Experiential Education (ESEE), a comprehensive system to assess the impact of service-learning on students, faculty, educational institutions, and communities, as a means to systematically assess the impact of service-learning, school-to-work, and community service programs on students' educational development"  There is a K-12 system and a higher education system a well as some individual tools. This tool measures civic responsibility, so the behavioral outcomes criteria is not applicable to this tool. This tool measures civic responsibility so the behavioral outcomes criteria is not applicable to this tool. Survey Excellence in Environmental Education:  Guidelines for Learning (K-12) North American Association for Environmental Education (NAAEE) Guidelines and benchmarks for EE learning K-12 EE program delivery, including a self assessment tool.  Users can develop age-appropriate indicators. Takes a systems approach Connects with place/community Action-oriented Promotes wonder Guidelines Fostering Sustainable Behavior North, Rick/McKenzie-Mohr and Associates Online resource with chapters written on commitment, prompts, norms, communication, incentives and convenience, with a checklist supplied at the end of each chapter.  Look under "guide" then "tools." "Rick North of Portland, Oregon has been using community-based social marketing (cbsm) to reduce C02 emissions and alter food choices " Core message: Audience-centered Clear objectives Research-driven message development Multiple levels of engagement Behavioral Outcomes: Clearly addresses barriers to change Incentives for Change General resource Grizzly Bears Forever Student Survey, in "Measuring the Success of Environmental Education Programs" Gareth Thomson, Canadian Parks and Wilderness Society;  and Jean Hoffman, Sierra Club of Canada, BC Chapter A three part pre and post survey designed to measure knowledge, attitudes and behaviors from a school-based presentation.  The questions related to behaviors are self-reported.  The publication has an appendix with guidance on the pros and cons of different evaluation instruments, as well as some instrument design tips.  Protocol is not age specific, but probably is more appropriate for older youth. This tool measures civic responsibility, so the behavioral outcomes criteria is not applicable to this tool. This tool measures behavior change and does not focus on the strategies (program content) which is what the criteria are about.  In this case, it is not relevant to match the instrument to the criteria. Survey Guide to Planning and Implementing Cultural Competence Organizational Self-Assessment The National Center for Cultural Competence (Georgetown University Center for Child and Human Development) A self assessment tool from an organizational capacity angle.  Much of it is health and human service provider-oriented, but some of the elements are transferable. Value diversity Capacity for self-assessment Consciousness of cultural dynamics Self-assessment tool Guidelines for Culturally Competent Organizations Minnesota Department of Human Services This assessment tool helps an organization to identify its strengths and areas where it may want to enhance its ability to serve culturally diverse populations.  The tool focuses on:  service delivery and quality management ; human resources practices; governance, community relations, and marketing; administration and policy; and organizational culture. Consciousness of cultural dynamics Adaptability to diversity Self-assessment tool Guidelines for Developing and Evaluating Communications Tools/Efforts EETAP Includes lots of tips in developing communications, and also provides indicators of communications goals and a clear communications objectives.  Includes a checklist of components that should be present in communications-related materials for EE issues.   Audience-centered Multiple levels of engagement Clear objectives Research-driven message development Guidelines Guidelines for Excellence in Nonformal Environmental Education North American Association for Environmental Education (NAAEE) "...comprises a set of recommendations for developing and administering high quality nonformal environmental education programs."  The guidelines can be used as a tool to measure against. Guidelines Guidelines for the Initial Preparation of Environmental Educators North American Association for Environmental Education (NAAEE) A set of recommendations about the basic knowledge and abilities educators need to provide high-quality environmental education.  Used to apply to instructor/educator preparation. EE Practice: Learner-oriented Connects with place/community Takes a systems approach Promotes wonder Guidelines Headlands Institute Field Evaluation Duffy Ross and Linn Jensen for the Headlands Institute An observation protocol for an outdoors program with the following components:  learning environment, curriculum, teaching strategies, and professionalism. Learner-oriented Promotes wonder Connects with place/community Action-oriented Other Heating Up Society to Take Environmental Action: A Guide to Effective Environmental Education and Communication AED's Center for Environmental Strategies (part of GreenCOM project) This guide is not a tool per se, but it gives some guidance on elements to include in environmental ed communications and programs.   "This booklet summarizes the lessons learned from eight years of experience in over thirty countries. It provides environmental protection and natural resource managers, planners, and decision makers with insight on how Environmental Education & Communication can help them achieve their goals and objectives." Criteria is not applicable. Not applicable General resource Inclusiveness Assessment Tool Rainbow Research The tool must be ordered from Rainbow Research.  "This Inclusiveness Assessment Tool (Tool) can be used to help organizations make progress in becoming more racially and ethnically inclusive and more competent in dealing with diverse cultures.   The Tool is packaged in three parts: 1. Overview of the tool: its rationale and a description of five organizational practice areas critical to an organization's level of cultural competence 2.  A checklist of organizational practices (including an inclusiveness planning worksheet), to be completed as part of an initial planning process 3.  A progress checklist of organizational practices (including an inclusiveness progress report form) designed to measure subsequent progress in the five organizational practice areas." This tool has not been reviewed by Blueprint, but it may be useful. Self-assessment tool Intermediate Attitude Scale (Grades 3-6) National Association for Humane and Environmental Education and Western Institute for Research and Evaluation Group administered test designed to assess children's attitudes toward the humane treatment of animals.  Thirty-six questions related to the care and treatment of pets, farm animals and wild animals.  For grades 2-6. Learner-oriented Promotes wonder Survey Learning as We Go:  Making Evaluation Work for Everyone Peter York, TCC Group Contains a short check list of elements that need to be in place to make evaluative learning work Clear purpose Culture of inquiry Collaborative design Checklist Marguerite Casey Foundation Organizational Capacity Self Assessment Tool Blueprint Research & Design Inc for Marguerite Casey Foundation Adaptive Capacity module has some questions on evaluation capacity. Culture of inquiry Feasibility Self-assessment tool Measuring Results Schneider, Barbara and Nicole Cheslock Not a tool per se, but a report to the Coevolution Institute, April 2003.  Can be used as a reference guide.   "Gaining insight on behavior change strategies and evaluation methods from environmental education, museum, health and social marketing campaigns."  A synthesis of behavior change theories, evaluation methodologies and findings.  Brings together research on a range of outcomes - knowledge, attitudes and behavior change-- as tracked by museum, health and social marketing campaigns.  Provides different behavioral change frameworks that could be useful to those engaging in an evaluation.  Provides some examples of institutions and the types of evaluation data collected and their data collection methods. Criteria not applicable. Not applicable General resource National Survey on Biodiversity Belden, Russonello and Stewart for the Biodiversity Project public opinion survey:  gauging awareness of biodiversity, and appreciation for the importance of saving species and habitat diversity.  For adults 18 years or older. This survey instrument measures public opinion, so the criteria for core message development is not applicable. This survey instrument measures public opinion, so the criteria for core message development is not applicable. Survey NEETF / Roper Report Card on Energy and Environment, Attitudes and Knowledge Survey 2001 National Environmental Education & Training Foundation (NEETF) The 10th annual NEETF/Roper Starch Worldwide survey of adult Americans around energy.  At the end of the report is the survey questionnaire. Tool measures public opinion. This survey instrument measures public opinion, so the criteria for core message development is not applicable. This survey instrument measures public opinion, so the criteria for core message development is not applicable Survey NEETF / Roper Report Card on Environmental Knowledge, Attitudes and Behaviors