Below is a text-only version of the Executive Summary (i.e. no tables, figures, graphics). A web compatible version (i.e. low resolution graphics) of the complete report can be downloaded from the bottom of this page. Contact the CMP directly for a high resolution version of the report.
An Evaluation of
The Community Mapping Program
2003-2004
Prepared for
The Orton Family Foundation
and
The
Place-based Education Evaluation Collaborative
Prepared by
Program Evaluation & Educational Research (PEER) Associates
September 8, 2004
Executive Summary
This report on CMP evaluation activities for 2003-2004 consists of three strands: tool development (logic model and Toolkit), case studies (Richmond, VT, and Hayden, CO), and surveys (western region and northeastern region). The conclusions and implications from each of these strands, and thus the key conclusions from this report as a whole, are embedded in each of those sections of the report. This layout was employed for two reasons: first, to allow for each section to better stand alone as its own coherent piece; and second, to account for the very distinct process and content of each of these types of evaluation work. It should be noted that another report, the Evaluation Toolkit for the Community Mapping Program, was also completed this year and is provided under separate
cover.
Introduction
The Community Mapping Program (CMP) is a grassroots, community development initiative that strives to connect students, educators, and community partners to their home place through local inquiry. With resources provided by The Orton Family Foundation and others, CMP supports school projects in Vermont, New Hampshire, Colorado, Texas, Wyoming, Montana, and Utah through teacher training, on-site project support, and expert guest-lecturing.
Now completing its fifth year, the Community Mapping Program provides educators and community members with skills and ongoing assistance to design place-based curricula for youth that use global positioning systems (GPS) tools, geographic information systems (GIS) technology, and other mapping processes. Many CM projects take the form of service-learning projects that utilize student-generated information about the local community and landscape to address identified needs in the community.
Context for 2003-2004 CMP Evaluation
In the summer of 2002, the Orton Family Foundation and the Vermont Institute of Natural Science (VINS), embarked on a comprehensive, year-long evaluation of the Northeast Community Mapping Program. Both quantitative and qualitative methods were utilized to measure program impacts on educators, students and, to a lesser degree, community partners. A report was written to convey the stories, facts, and figures representing the northeast CMP. Meanwhile, the western region of CMP had also developed some quantitative measures of student learning and begun putting them together in an organized package of evaluation resources.
Program staff and evaluators together determined that 2003-2004 would be a transition year, bringing together the evaluation efforts of the two CM programs in order to streamline data collection efforts and bring program learning to a higher level. This strategy was particularly important in light of recent and increasing success with expanding the CMP into a nationally replicable model. A key mandate for this year was to create a set of tools that could be used to gather data across the country, and across the program model variations. These tools would then position program staff and evaluators to analyze data centrally so that conclusions could be drawn about the program as a whole. Furthermore, since the northeast and western versions of CMP vary to some degree in format, content, and philosophy, it became
eminently clear that the existing logic model, which had been tailored to the northeast program, needed revision to serve as a guide for the nationwide evaluation effort.
Thus, the four primary components of the 2003-2004 CMP evaluation plan were:
1. facilitating the refinement of a national CMP logic model;
2. completing two case studies of CM projects in action, one in the Northeast, and one in Colorado;
3. refining existing survey instruments, including pilot tests in both the Northeast and in selected western states;
4. consolidating the first two years of learning about the CMP evaluation process into a toolkit of evaluation instruments and methods for use across all CMP sites and regions as the program expands nationally.
This document reports the results for items 1-3 in the list above, as well as the process for item 4. The Evaluation Toolkit for the Community Mapping Program (PEER Associates, 2004), which represents the content of item 4, is presented as a separate, stand alone document since the purpose, audience, and use of that content dictated a different presentation format.
Evaluation Methods
Evaluation activities for this year can be described in terms of three distinct but highly interwoven processes: tool development, case studies, and surveys. The following sections describe each of these strands in turn. However, due to the prominent role and high priority of evaluation tool development, many of the “findings” of the evaluation are more accurately presented under the “methods” heading in the flow of the report. The methods and tools are the findings in large part. Thus, the content and length of the following sections may appear to be atypical for a standard evaluation report.
Logic Model
In order to encourage and support a core set of coherent goals and practices among CM projects across the nation, the CMP evaluation plans for 2003-2004 called for substantial investment in developing and refining the conceptual tools that describe and operationalize the program’s philosophy. A key product of this effort was a revised CMP Logic Model (see Appendix A) which is intended to represent the most crucial elements of all CM projects, even as the program adapts to local conditions in its national expansion. A simplified version of the logic model is shown here.
Toolkit
The evaluation team engaged program staff and other stakeholders in a series of activities that guided the development of the various components of the Evaluation Toolkit for the Community Mapping Program:
· Utilizing the newly developed comprehensive CMP logic model as a platform for identifying the assorted tools that would need to be developed in order to support the various incarnations of CMP.
· Revising and refining Educator and Student survey instruments to encompass program variations and account for CMP’s participation in PEEC’s cross-program evaluation efforts.
· Developing a Community Partner survey.
· Translating the essential elements of a CM Institute model into a survey format that provides guidance to CM Institute trainers and is also a vehicle for direct feedback from participants.
· Developing a system for on-line administration of the Institute survey, which then enabled all other CMP and PEEC surveys to be administered through internet browsers. This was sparked by the need to efficiently administer Institute surveys in a wide range of locations.
· Conducted two case studies, one in Vermont and one in Colorado to highlight the workings of exemplary projects and explore useful themes.
· Creating a case study template so that the protocol can be replicated in other locations.
All of these evaluation activities ultimately resulted in the creation of a toolkit of evaluation instruments and processes for use in CM programs nationwide. This Evaluation Toolkit for the Community Mapping Program is published as a separate stand alone product, and the contents of the Toolkit are not presented in this report.
Surveys
The survey portion of this year’s evaluation is an example of the way that different strands of the Community Mapping Program needed to be brought together under the single umbrella of CMP’s national expansion effort. The results from the western and northeastern survey pilots are presented here in a single report, but data were collected and analyzed in two separate but parallel processes.
The results of the western region survey pilot as a whole provide preliminary evidence that the CM program impacts students’ awareness of how to use research to solve problems, and monitor their own progress. Additionally, pilot data suggests that students felt like they had knowledge and skills to share with the community and with younger children, and a responsibility to do so. Results were obtained from two-tailed significance tests on 63 matched sets of pre-post student surveys, and two sets of teacher survey instruments designed to triangulate with the student survey results.
The primary purpose of the northeastern region survey pilot was to test a “dose-response” measurement strategy, adapted from concepts in the field of behavioral psychology (Strosahl et al.,1998). From this frame, the question becomes: Do participants exposed to a higher “dose” of a program systematically report higher levels of behaviors and attitudes that the program is trying to impact? To the extent that relevant outcomes vary with dose, assertion of program impact becomes increasingly credible.
Overall, the pilot of the dose-response measurement strategy was deemed successful, though several opportunities for refinement of design and analysis remain, particularly with regards to student and community partner surveys. However, the pilot generated several defensible inferences showing positive impacts of CMP on educators. Statistically significant positive correlations of moderate to large size were found between educator exposure to CMP and:
· Overall teacher practice
· Use of local places
· Service learning
· Improving teacher craft
· Teacher engagement
· Educator reports of student performance
· Student engagement
· Student stewardship behavior.
Additionally, the dose-response measurement strategy effectively sidestepped many logistical and conceptual challenges associated with a pre-post measurement strategy.
Case Study of CMP in Action: Students Bring the Rivershore Preserve to the Community
RSP Case Study Overview
During the 2003-04 school year, students, teachers, and community partners in Richmond, Vermont embarked on a project to bring public awareness to the rich ecological, cultural, and historical resources provided by the Rivershore Preserve adjacent to the Winooski River. Using the framework set forth by the Community Mapping Program (CMP), sixty seventh and eighth graders from Camel’s Hump Middle School joined forces with thirty community members to conduct an analysis of the 35-acre parcel, owned in part by the Richmond Land Trust. Each working with a community expert, student teams assumed delegated roles to complete the comprehensive survey. As the project took form, the community partners came to include: the Richmond Land Trust, Richmond Planning Commission, Richmond Conservation Commission, and
the Richmond Recreation Path Committee.
RSP Case Study Findings
Five primary areas of findings for the RSP Case Study are discussed briefly below.
Community Partners Achieve Goals through CMP Project Collaboration
The primary product of the RSP project (at the conclusion of this case study) was a series of student presentations to the public highlighting the ecological, cultural, and historical significance of the Rivershore Preserve. Student PowerPoint Presentations were given to several community and school groups of adult decision makers. Other products-in-progress include a website displaying the project’s findings to be used as a community resource, and an interpretive trail map to be displayed at central town kiosk as a resource for visitors.
CMP Project Promotes Community-School Connections
Thirty community members actively participated in the weekly project sessions. This community presence in the school facilitated a greater awareness of the school and its students as genuine resources for improving the health and vitality of the community. Further, students gained exposure to the human resources present in their community.
Students Learn Valuable Lessons from Real World Project
Teachers, students, and community members saw value in the “real world” nature of the RSP project. Participants had to account for the challenges, obstacles, and rewards present in real life projects. Consequently, students were exposed to the types of work they might find at a future job. The size and ambitious nature of the project encouraged cooperation amongst the student teams, and also challenged them to take responsibility for the finished product.
Preserve Adopted as Teaching, Learning, and Community Resource
Students participating in the RSP project were able to develop an intrinsic attachment to the wonder of the place. Some student teams visited the Rivershore Preserve almost every Friday in the company of community experts and their teachers. Teachers noted that the project helped students to develop important personal qualities such as confidence, a “sense of themselves and their classmates” and a connection to their community, explaining that, “You can’t teach them about that without doing something real.”
RSP Project Challenges
Two primary challenges were noted within this project:
Ø The RSP project was described by many as overly ambitious in terms of its size and scope. This prevented timely completion of some products for partners.
Ø Because the RSP Project was so ambitious, teachers reported having to trim the curriculum in other areas. The teachers did mention, however, that the RSP project was designed to indirectly incorporate these missing components through activities like the writing of press releases.
RSP Case Study Conclusions
This case study shows that:
Ø The associated community partners were able to achieve their goals by being involved in the RSP project. Involving community members with school projects strengthened community-school connections and provided students with different perspectives on types of organizations and job roles in their community.
Ø Students received “real world” lessons through project challenges and students demonstrated increased interest in learning traditional content in the context of their home place.
Ø The number of participants (students and community members) created challenges in project coordination.
RSP Case Study Implications for Practice
Three recommendations hailed from this case study. These are described in greater detail within the main report.
Ø To maximize the likelihood that products will be useful to and used by community partners, CMP staff could encourage community partners to be actively engaged in the project work during all phases of planning, implementation and follow-up.
Ø To improve the chances that students will identify an adult partner as a mentor or role model, CMP projects could encourage the inclusion of a variety of local community experts assisting with various phases of a mapping project.
Ø Because CMP is necessarily a flexible professional development model, projects are vulnerable to becoming large and complex. CMP staff could more closely monitor project size and scope to help keep projects more manageable.
Case Study of CMP in Action: Young Students Monitor Wildlife Patterns Along Roads
Students, teachers, and community partners in Hayden, Colorado, and nearby Steamboat Springs, recently completed year two of a community mapping project called Critter Control. Elementary students involved in the project collect and provide roadkill data to the Colorado Division of Wildlife (DOW) to help the agency determine how best to reduce the number of animals killed by motorists. Using the framework set forth by the Community Mapping Program (CMP), forty-four second- and fourth-grade students from Hayden Elementary School, along with their two teachers, began the project in the fall of 2002. In the fall of 2003, 40 fifth graders and their two teachers at Strawberry Park Elementary School in Steamboat Springs joined the project.
Students have presented their information to DOW in the form of maps, data sheets, PowerPoint presentations, and detailed written information, all of which is being added to a multiple-year DOW study intended to help the agency make informed decisions about appropriate mitigation methods such as where to place signage or install fencing.
DOW lends great support to the Critter Control project in the form of classroom teaching, field work supervision, and technical support.
A new community partner, The Colorado Department of Transportation (CDOT), is looking forward to helping students hone their technical skills and better understand CDOT’s interest and role in wildlife management.
Critter Control Case Study Findings
Three primary areas of findings for the Critter Control Case Study are discussed briefly below.
True Collaboration is Critical
Collaborative involvement among community partners, teachers, students, and parents involved in the project is critical to its success. Five types of relationships were highlighted:
Ø Teachers repeatedly stressed the importance of working with a community partner who truly has the time and enthusiasm to commit to the project.
Ø DOW expressed the value of having established a prior relationship with a school and community.
Ø Teachers reported on the power of teaming with another teacher.
Ø Teachers explained the value of viewing students as fellow team members, and the importance of honoring their input and ideas.
Ø Teachers highlighted the critical role played by parent volunteers.
Community Partner Achieves Goals
Two important goals of the community partner are clearly being met by the project: students are providing valuable data to DOW, and DOW’s educational and public relations goals are also being met.
DOW referred to the long-term nature of the study. “We’ve stressed with the students and teachers that we need much more data before we can make decisions about mitigation methods such as signage. The students’ data will help greatly with those decisions….” The DOW Wildlife Manager also noted, “We’re honing a model of data collection and reporting that works. Getting that model in place takes a bit more time because it’s being created by such young students.” DOW also noted that their goals for the project extend far beyond that of data collection and into the realm of important public relations benefits.
Students Motivated, Engaged, and Achieving
Ø Students demonstrated and teachers observed increased academic engagement and achievement through work on the project. In particular, the “real world” nature of project motivated students.
Ø Students commented repeatedly on the value of hands-on learning.
Ø Students and teachers reported growth in writing skills and critical thinking.
Ø The project afforded teachers many opportunities to authentically assess student academic achievement.
Critter Control Case Study Conclusions
As evidenced by this case study, teachers, students and community partners all benefited from participation in the Critter Control project.
Ø Active and direct involvement by the community partner was critical to the success of the project and the participating community partner was able to achieve its goals by being actively involved.
Ø Teamwork among teachers, along with extensive parental involvement, allowed teachers to work effectively with a multi-age group of young students on an ambitious project. The project energized teachers and validated their belief in the efficacy of project-based learning.
Ø Students demonstrated growth in areas such as writing, critical thinking, public speaking, and cooperative learning.
Ø Students gained a better understanding of and deeper connection to their local place and its natural resources.
Critter Control Case Study Implications for Practice
The following recommendations are described in greater detail within the main report.
Ø CMP staff could encourage community partners to be actively engaged in the project work and to provide timely feedback and direction to students throughout the project.
Ø CMP staff could encourage teachers and community partners to adopt a collaborative approach to the project wherein all participants -- including students -- have a voice in project design and implementation.
Ø In order to increase the likelihood that student work is both academically relevant and useful to the community, CMP staff could facilitate the clear articulation of academic and community needs being addressed, including the goals of the project and expectations of the students.
Ø CMP staff could support teachers and community partners in designing projects that allow students to draw on a number of disciplines and learning styles and that engage students outside the classroom in addressing a real-world problem.
CMP Evaluation Report Conclusions
Stepping back and looking at the whole set of evaluation findings for 2003-2004, it is clear that the program is actively providing an exciting model for delivery of place-based education, and is contributing in many positive ways to improving community-school connections in many locations around the country.
Participants are reporting some challenges and much satisfaction and accomplishment. From the case studies we find that community partners are receiving products of value from the students with whom they work, teachers are finding new ways to effectively teach and assess their students, and students are developing relationships with important people and places in their communities. Quantitative evaluation efforts are making steady headway on the formidable measurement challenges inherent in this research context, but are already revealing several statistically significant correlations between participating in CMP and improved teacher engagement, use of local places, service learning, and student stewardship behavior. This is despite the fact that the primary focus of this year’s evaluation was to pilot specific research
methods for use in future evaluations.
By committing the time and energy to articulate a clear, mutually agreed upon logic model for the national program, CMP staff are modeling the continuous learning philosophy that is embedded in the CM model itself. Further, the creation of the Evaluation Toolkit can be expected to greatly leverage limited program resources by helping to embed evaluation efforts right into the fabric of the program delivery, rather than relying exclusively on outside evaluation consultants for every aspect of program research. With the development of tools that are able to capture program outcomes across the country, CMP is ever closer to being able to monitor projects with precision and accuracy. And, with a growing body of evidence gathered in its first two years of comprehensive evaluation, there is a clear
pattern indicating that CMP is providing many, many benefits to students, their teachers, and partners in the local community.
| 