An Evaluation of
The Northeast Community Mapping Program
Prepared for
The Orton Family Foundation
The Vermont Institute of Natural Science
and the Place-based Education Evaluation Collaborative
By
Sharon Plumb
The University of Vermont
Amy L. Powers
Program Evaluation and Educational Research Associates
With
Sally Wither
July 1, 2003
(What follows is the Executive Summary. The full report is attached as a pdf file.)
Executive Summary
Now completing its fourth year, the Northeast Community Mapping Program (CMP) is a place-based education professional development program of the Orton Family Foundation and the Vermont Institute of Natural Sciences (VINS). CMP provides educators and community members training and ongoing assistance to design and implement place-based curricula for youth. Staff members help educators and community members design service-learning projects that utilize student-generated information about the local community and landscape.
The Orton Family Foundation and VINS share a vision of creating sustainable communities. The Orton Family Foundation works with rural communities as they face challenges brought about by rapid environmental, economic, and social growth. While the Foundation provides much of the funding and direction for the Community Mapping Program, Northeast CMP staff members are based out of VINS.
CMP program goals:
If we implement school-community investigations of a natural or cultural resource of local concern using spatial tools and field experience, participants will gain:
· A deeper understanding of community systems (ecological, social, economic, cultural systems).
· Increased capacity to use visual tools for problem-solving.
· Products of real community value.
· New opportunities for connecting learning to real world uses.
· Expanded capacity for school-community collaboration.
Elements of the CMP model:
· Flexible curricula
· Training institutes
· Visual and spatial tools
· Service-learning projects
· Ongoing support in the areas of
o Project Development
o Classroom Support
o Technical support
o Materials and Resources
In the summer of 2002, CMP embarked on a comprehensive evaluation of its professional development model. This report is the synthesis of evaluation efforts conducted during the 2002-2003 school year with the vast majority of these evaluation efforts taking place in Vermont and New Hampshire. Several sections of the report contain reference to Colorado CMP evaluation efforts and are referenced as such. Findings disclosed in this evaluation report are a reflection of the Northeast Community Mapping Program unless otherwise explicitly noted.
Evaluation Methods
This was a mixed-methods evaluation, relying on both qualitative and quantitative data. The evaluation process was participatory and utilization-focused, in the hope that it will serve stakeholders as a tool for program documentation, development and refinement.
Evaluation questions
Evaluation questions were divided amongst four areas of inquiry:
· Teacher outcomes
· Community partner outcomes
· Student outcomes
· Process effectiveness
The evaluation focused largely upon how a teacher’s pedagogy is affected by his or her involvement with the Community Mapping Program. Because program staff expects the teacher/community partner relationship to play a large role in the teacher’s implementation of a CMP project, the evaluation also focuses upon the relationship between teachers and community partners. In this first year of evaluation, questions regarding student outcomes center largely upon students’ relationship with their school and community. Questions regarding academic performance were part of the Colorado CMP program’s evaluation and are also referenced in this report.
Data Collection
Evaluation instruments and types of participants are listed in the following table:
Instrument Quantity
Teacher interviews 23
Student interviews 68
Community partner interviews 12
Summer Institute participant observations 2 Institutes (8 days)
Classroom observations 10
Staff interviews 4
Staff focus group 1
Examples of student and teacher work Assortment
Photos Assortment
Historical data (Year End surveys 2000, 2001, 2002) 47
Student pre and post surveys 160 pre-surveys
168 post-surveys
Teacher pre and post surveys 6 pre-surveys; 7 post surveys
Pilot: Teacher Matrix (Colorado project) 1 5-8 grade teacher
2 high school teachers
Pilot: NAEP-based test (Colorado project) 12 students, 5-6 grade private
10 students, 7-8 grade private
12 students, 9-12 grade private
15 students, 9-12 grade public
Pilot: Adapted Student Survey (Colorado project) 12 students, 5-6 grade private
10 students, 7-8 grade private
12 students, 9-12 grade private
Data Analysis
Qualitative data were analyzed using Miles and Huberman’s (1994) approach to sorting data. When fieldwork was complete, descriptive observation data and transcribed interviews were coded to illuminate key emergent issues. Later, a table was created to further organize the themes and identify their significance. This last level of analysis allowed for an intimate knowledge of the data, and provided a concise, efficient method to quantify some of the qualitative results. Likert scale data were entered into an MS Excel spreadsheet and analyzed with basic statistics to determine if the means had changed in meaningful ways. Open-ended survey questions were analyzed using coding, as above.
Teacher outcomes
Evaluation questions addressed in this section:
· How does involvement with the CMP affect a teacher’s utilization of the local community ands ecological resources in their curriculum?
· Did the teachers continue to use CMP concepts/tools after year one, and/or do they intend to do so in future years?
· How does participation in CMP change teachers’ teaching practices?
Key findings
· Teacher response to the Community Mapping Program is overwhelmingly positive. The majority of teachers feel that their participation in trainings was time well spent, and CMP contributed significantly to their teaching practice.
· Over 90% of the teachers interviewed said that CMP stimulated them to use local and community resources more than they have in the past.
· Most classes went into the community or investigated the landscape between 2-10 times throughout the life of the project.
· Teachers felt that CMP was a practical and meaningful way to incorporate service-learning into their curriculum.
· Teachers universally agreed that their use of maps and technology increased as a result of involvement with CMP. They used maps as a teaching tool for the first time, or in ways they never had before.
· Most teachers used Global Positioning Systems (GPS), often with CMP assistance.
· Most teachers introduced students to Geographic Information Systems (GIS) and close to one-quarter of the educators used GIS in the classroom.
· Teachers had an impressive “spread of effect” upon their school and the community, enticing other teachers to use community and school resources, and sometimes GPS and GIS.
· Over three quarters of the teachers commented on, or in their curricula showed evidence of interdisciplinary CMP projects.
· Close to one half of the teachers felt that CMP projects gave them the confidence to teach “citizenship skills.”
· All of the educators are interested in continuing to incorporate elements of the program in the future, either in a formal relationship with CMP or on their own.
· Educators feel that CMP helps boost the school’s image and attract positive community attention.
· Pre- post-survey data revealed that educators’ practice changed significantly in three areas: collaboration with other teachers (increase); crossing disciplines (decrease); and using maps and other spatial tools in their curriculum (increase). Surveys also showed that after implementing CMP project, some educators’ definitions of place-based education and the importance of maps and spatial tools were expanded to include a more active community engagement focus. Surveys also confirmed that educators’ use of local resources had expanded considerably.
· (Colorado) A survey of two teachers in Colorado indicated that the teacher who uses a CMP project as a part of a science class covered more of the science areas while the computer instructor emphasized the technology and the students’ results on the student learning tests reflected this emphasis.
Community partner outcomes
Evaluation questions addressed in this section include:
· What types of relationships develop between community partners, schools, educators and students as a result of the CMP collaboration?
· In what ways were community mapping projects useful to community partners?
· Did the community or the community partner take action based on the information generated by the project? (preliminary data gathered)
Key findings
· The community partner is an extremely powerful element of the program model. The types of relationships varied greatly, from community partners being integrally involved in the development of the project to community partners having minimal direct involvement with students.
· Community partners come from a wide variety of backgrounds, and work or volunteer for organizations such as nature centers, planning or conservation commissions and watershed councils.
· All of the community partners expressed an interest in working with CMP projects in the future.
· Over one-half of the community partners felt that working with the school was more important than the final product.
· Mapping projects are often useful, and are often of high quality. Close to three-quarters of the community partners who were interviewed were pleased by the students’ final product, including creating brochures about their community for local tourists, building hiking trails, creating orienteering maps, protecting vernal pools, and designing web pages for the local farmer’s market or their community.
· The most useful projects arose when community partners and teachers worked together to identify a real community need and when community partners stayed involved throughout the process.
· Several organizations were involved with the projects because they felt that CMP was an affordable way for them to gain technological capacity.
· Community partners were not involved in curriculum development or project planning at the level that CMP staff or educators would like to see.
· Only two community partners involved in this study had attended a CMP summer institute. Several community partners were not aware of who CMP is or how they are involved with the school.
· At nearly every site, community partners agreed that a rewarding and mutually beneficial relationship arose between the partner, teachers and students. However, nearly three-quarters of the community partners did not feel they had enough contact time with the students. In most cases this was attributable to the community partners’ feelings of over-commitment in their current jobs.
· All of the community partners who did work with the classrooms felt that the relationships gave students opportunities to interact with adults in a genuine fashion.
Student outcomes
Evaluation questions addressed in this section include:
· How does involvement with the CMP change students’ perception of and relationship to their local community?
· How does participation in CMP affect students’ level of civic engagement?
· What long-term effects does involvement with the CMP have on students in terms of their academic or career choices, and their relationship to and involvement in their home place? (preliminary data gathered)
· (Colorado) Do students participating in CMP projects improve their abilities to understand and apply knowledge about geography and science?
Key findings
· Students were overwhelmingly positive about their involvement with the project. Students were excited to get outside, and most preferred conducting local research over lectures.
· Nearly all students said that doing hands-on activities was more valuable than learning from books or lectures and wished that more classes were like what they did with CMP. They felt that active learning helped them to retain and understand difficult concepts and factual knowledge and also helped them understand how the topics they studied in school related to the rest of their lives.
· Two-thirds of the educators stated that participation in CMP helped students gain a stronger sense of place.
· Close to half of the students stated that the project helped them gain a deeper appreciation for their community.
· It is not conclusive that CMP projects helped students to feel more empowered to engage as active citizens outside of the project. While many students said they “felt good” about their projects, numerous students did not immediately extrapolate this experience into feelings of empowerment in other parts of their lives.
· While students mentioned that they were exposed to spatial tools (maps, GPS and GIS) during the development of their project, only a few thought they were the most exciting aspects of the project, felt they knew how to use them, or were inspired to use them for other applications.
· Most students were not involved in creating the “final map based product”. They collect much of the relevant data, while a teacher or CMP staff member frequently produces the final product. Students do, however, regularly participate in a culminating activity such as presenting data to town conservation commission officials, parents, cooperating organizations, or other students.
· Quantitative analysis of student survey data revealed that students made gains in all five indexed areas: Sense of responsibility to community; Interest in local place; Feeling of connection to adults/mentors; Motivation and Empowerment to act; and Perceptions about the importance of maps.
· (Colorado) Student pilot survey responses indicated that the students were aware of wildlife issues and the Division of Wildlife’s work. They also remembered using GIS and the field work as activities that helped them learn competencies including using resources well, working with others, acquiring and using information, understanding complex inter-relationships, and using technology.
Process effectiveness
Evaluation questions specific to the program model include:
· Which aspects of the CMP professional development model are most effective at accomplishing CMP goal? (e.g. community partner relationship, teacher/student-responsive curriculum, field support, provision of tools, inquiry-based learning, summer workshops)?
· Are CMP program goals and participant goals and expectations aligned?
· What barriers have existed for teachers, community partners, or project coordinators?
· What types of CMP staff interventions best support the collaborations between community partners and educators?
Key findings: Process strengths
· 100% of the teachers felt that the two strongest elements of the CMP model are the flexibility and ongoing support provided by the staff.
· Teachers and community partners felt that program flexibility allowed for creativity, responsiveness to curricular needs, and diverse community needs.
· The majority of teachers utilized CMP staff for classroom and field support; staff members were asked most frequently to teach GPS and GPS and act as chaperones on field trips.
· All teachers felt having a partnership with a community organization was one of the most critical components of the CMP model, and several mentioned that CMP helped them to find community partner and other resources in the community.
· All of the teachers felt that CMP convinced them of the wide application maps could have in their classroom, and were thrilled with their increased awareness of maps, GPS, and GIS.
· Most teachers felt that the strongest elements of the summer institutes were the mini-mapping project and the introduction to spatial tools.
· The majority of educators mentioned that learning from fellow and past CMP educators was valuable.
· Nearly all of the teachers interviewed had used some form of CMP (beyond the Institute) and recognized that the supportive roles they played were significant benefits that came at little or no cost.
· Most staff members agreed that educator/community partner relationships are a critical component of the program but are not as strong as they would like them to be.
Key findings: Process challenges
· Nearly half of the educators felt that CMP would benefit from more clearly articulating program goals.
· Close to one half of the community partners and educators felt they would like to have had a stronger relationship with one another.
· While teachers appreciated the program’s flexibility, they felt that supplemental resources would help them to design a stronger program, create a stronger dynamic with their community partner and help them to be more effective transferring what they had learned at the institute to their students.
· CMP staff members believe that their program is appealing because it is flexible; this strength, they feel, also makes it more challenging for them to develop the model, and for educators to use.
· Teachers felt that the technological component of the model presented challenges, sometimes at the cost of developing the rest of their place-based project.
· Many teachers felt they left the institute needing greater clarity about what kind of support (technological and otherwise) they could expect from CMP once they left the institute.
· Teachers face numerous challenges doing place-based education within the prevailing school structure, including: finding curricular time, length of class periods, planning time, and knowing how to manage students outside.
Conclusions
Northeast CMP is succeeding at many of its goals. In its first four years, it has inspired numerous teachers to do locally based investigations of their community and the landscape with their students. Students were engaged in hands-on projects, and used maps and other spatial tools more than they had in previous classes. They learned details about local cultural and natural history, and gained a deeper appreciation for their community. Teachers worked with community partners to design valuable learning experiences for students that often result in usable map-based products for the community partner.
CMP’s approach to professional development is admirable. The model’s flexibility and staff support encourage teachers to use the program to meet their individual curricular needs. This level of support is one of the program’s strongest assets and will help teachers to overcome barriers that are often found in environmental education programs. Furthermore, CMP is making great strides towards encouraging teachers to use local community and ecological resources.
The community partner relationships, and the service-learning component, are one of the most rewarding, and one of the most challenging aspects of the model. They bring focus to the curricula, and often raise the quality of work students produce. The level to which community partners are involved varies tremendously, and is largely influenced by the teacher’s motivation and the community partner availability. The service-learning/community partner relationship holds incredible potential. In cases where strong relationships have developed, they have proven to be invaluable to teachers in meeting their curricular goals and the partners who receive valuable student products and connections with youth.
The program’s outcomes depict a remarkable blend of the missions of two distinct organizations. Clearly, CMP projects engage individuals and communities in the care of their environment, the mission of VINS. And, while the Orton Family Foundation’s mission addresses the more specific proposition of responsible rural land use planning, there is ample evidence that CMP fits well within its mission. Many student projects contribute information toward land use decision-making, and by presenting data to and interacting with local officials or town committees, students have the opportunity to influence local policy. Students’ growing feeling of connection to adult mentors, sense of responsibility to their community and interest in their local place are promising steps toward more integrated, cooperative communities.
As CMP continues to train teachers, it will benefit from continuing efforts by staff, teachers and partners to adapt and refine strategies for working with each element of the program model that serve as guideposts and indicators of success: place-based education, ongoing support, flexibility, the community partner relationship and service learning, and the use of visual and spatial tools. The continuation of this exemplary program is an important step toward connecting communities with their schools, growing citizens who have a sense of place and civic responsibility, and ultimately building more sustainable communities.
Recommendations derived from the analysis of evaluation data can be found in the Final Report.
| 