Below is a text-only version of the Executive Summary (i.e. no tables, figures, graphics). A web compatible version (i.e. low resolution graphics) of the complete report can be downloaded from the bottom of this page. Contact the SSP directly for a high resolution version of the report.
An Evaluation of
The Sustainable Schools Project
2003-2004
Prepared for
Shelburne Farms
The Vermont Education for Sustainability Project
and
The Place-based Education Evaluation Collaborative
Prepared by:
Amy L. Powers, Michael Duffin,
& Program Evaluation & Educational Research (PEER)
Associates
September 8, 2004
Executive Summary
Introduction
The Sustainable Schools Project (SSP) is a partnership program of Shelburne Farms and Vermont Education for Sustainability (VT EFS). The project uses sustainability as the integrating context on a school-wide level, seeking to connect science, literacy, ecology, and community. SSP focused its first school year (2002-2003) piloting its program in one urban elementary school in Burlington, Vermont. In this, its second year, SSP continued its intensive work with the same elementary school while beginning to make inroads into its next elementary school site on the other side of Burlington. Full-fledged involvement with the second site will take place in the third year of programming.
The primary goal of SSP is to use sustainability as an integrating context for strengthening a whole school’s curriculum improvement efforts, teacher leadership, community partnerships, and campus ecology.
SSP strives to help a school become more connected and collaborative internally, as well as more connected with the school’s local community and natural environment. SSP staff describe the desired outcome as creating a more connected, coherent learning experience that inspires teachers to collaborate, and inspires students to engage with their school, community, and environment.
The SSP Logic Model (see Appendix A) further describes the program’s short and long term outcomes. The evaluation team worked with key program stakeholders to develop this year’s evaluation plans by using the SSP Logic Model as the anchor.
SSP’s sponsoring organizations are among the founding members of the Place-based Education Evaluation Collaborative (PEEC), recently formed to evaluate the individual programs of its members and also to lay the groundwork for broader research into the effectiveness of place-based education.
Methods
The first cycle of program evaluation sought to understand the successes and challenges of SSP’s process of program development and implementation, and to measure the degree to which its short-term projected outcomes were attained. The second year of evaluation continued to monitor progress at Champlain Elementary School, observing program impacts on faculty and students, investigating the integration of literacy and sustainability, and assessing the program’s staying power. The current evaluation cycle also involved gathering baseline data for the newer SSP school, Barnes Elementary.
While this is primarily a formative evaluation, the report does provide evidence of positive outcomes as well as suggestions for program improvements. The evaluation is intended to be useful to program managers, funders and interested stakeholders, and to contribute to the field of place-based education.
Based on the SSP Logic Model, and the objectives of the program stakeholders, the evaluation team developed the questions listed in the Figure M1. After SSP staff reviewed and approved the questions, the evaluators designed appropriate research instruments to explore questions.
The evaluation utilized multiple methods. Semi-structured interviews and focus groups were conducted with teachers, administrators, parents, community members, and SSP staff members. Students were interviewed informally during site visits. Classroom and school observations, internal reflection instruments, student academic achievement data and educator and student surveys served to triangulate the interview data.
The evaluation team collected both qualitative and quantitative data about the integration of literacy and sustainability at Champlain Elementary. Qualitative interview data were gathered as part of semi-structured interviews that also sought data about other aspects of SSP. Quantitative data was gathered using a series of four locally designed and implemented assessments:
1) A compilation of two lists: one that showed all the major activities within Champlain elementary that integrated sustainability and literacy during the 2002-03 school year; and a second list that showed similar activities that occurred during the 2003-04 school year
2) A simple, one page questionnaire about the use of non-fiction resources administered to the entire teaching staff.
3) A two page questionnaire consisting of open-ended questions about literacy teaching practices.
4) A quasi-experimental assessment of student literacy skills was designed, piloted, and administered to a representative random sample of students.
Findings
This year’s evaluation of SSP focused on both process and outcome elements. In particular, we monitored the impacts of the program on teachers and students at Champlain Elementary School, now in its second year with SSP.
Teacher Outcomes
One of SSP’s primary strategies is to help teachers plan and implement sustainability-related curricula through the summer institute and ongoing coaching and support during the school year. There has been ample evidence this year that such practices are flourishing and becoming more embedded in the school’s operations. In addition to an overall sense that teachers were thinking differently about the curriculum planning process as a whole, there were three specific areas in which teachers described changes in their teaching practice. Teachers are increasingly:
· Using essential questions to plan and teach their students throughout the year
· Creating interdisciplinary units of study that weave together various disciplines, often using concepts of sustainability as an integrating theme
· Connecting their curricula with that of other classrooms, including special subjects such as art, physical education, and music, and across the grade levels.
Student Outcomes
Through SSP work, students were exposed to diverse new places in the community, and--equally importantly--they began to develop a deeper understanding of places they already knew. This exposure came on various levels: a greater awareness of schoolyard flora and fauna, increased knowledge of the different groups that make up a neighborhood, and heightened insight into the ways that a business fits into the local economy. Three key areas are highlighted in the report to demonstrate the impact of SSP on student awareness of and engagement in their community:
· The grades 4/5 team immersed itself in comprehensive citizenship projects that had beneficial effects on the community and for the young learners.
· Younger grades also explored the community, including forming relationships with elders and developing a more fine-tuned sense of local geography.
· Teachers reported that students’ sensitivity was growing as they engaged more deeply with the community and developed a sense of involvement and feelings of responsibility.
Furthermore, three other areas emerged as outcomes related to student involvement in SSP:
· Through their school work, students demonstrated that they were linking issues of personal and community health to the topic of sustainability.
· Teachers reported that because of the level of student engagement in sustainability-related work, students are increasingly making connections across units of study, and are increasingly directing their own learning processes.
· Students are making an increased personal commitment to sustainable practices at school, and they are taking sustainability messages, such as composting food waste, into their homes.
Process Strengths
An overwhelming portion of the interview data and document review comprising this evaluation pointed toward program strengths. The most salient of these are:
· SSP reaches a schoolwide audience
· Composting takes hold
· Skilled and effective SSP staff promote change
· Providing resources and teaching classes
· Second summer workshop embeds changes in practice
· Individualization and flexibility of the SSP model
· SSP has staying power
A crucial measure of the success of a program such as SSP is the likelihood that the program will be continued over time, or have “staying power,” within the site. Three key areas emerged as clear evidence of the staying power of SSP, defined more specifically as its potential to continue at Champlain School beyond the first two program implementation years:
· A wide network of support among the school administration, parents, members of the community, and the teachers.
· A commitment to continuing relationships that were established through SSP, such as those fostered between community members and teachers.
· An effort to document SSP-related projects, such as the living machine, on site.
Process Challenges
Unequivocal praise for the SSP experience at Champlain Elementary School and clear evidence of positive outcomes strongly indicate that the model is a success. Nonetheless, as can be expected with any program, a pattern of challenges emerged in the data. The challenges are broken down into the following three areas:
· Time and scheduling can be barriers
· The language of sustainability can be a deterrent
· Some subjects seen as outliers to sustainability work
Integrating Literacy and Sustainability
This section of the report explores data collected from four different assessments of various aspects of SSP’s efforts to integrate literacy and sustainability, and then synthesizes that data with relevant evaluation interview data.
A comparison between the past two school years reveals a marked increase in the amount of effort put into literacy at Champlain as a result of SSP. Not only did the amount of activity increase from 2002-03 to 2003-04 (especially in terms of more work with specific grade levels), but the type of activities seems to have taken on a new dimension as well. Clearly, sustainability concepts are being successfully integrated with literacy. This presents SSP with an opportunity to articulate just how much integration needs to happen in order to call their efforts a success, and how that might be accurately measured.
Champlain Literacy Assessment Questionnaire
One way that SSP’s strategic focus on the use of informational (non-fiction) texts showed up was through a simple, one-page, pre-post questionnaire given to all teachers at Champlain elementary school. The most prominent pattern in the findings is a seeming increase for every one of the (loosely) scaled items. This provides one tangible level of affirmation that SSP’s efforts to integrate sustainability and literacy are having positive results.
Literacy/Sustainability Study Group Reflections
The literacy/sustainability study group was an intentional component of the SSP at Champlain School. Five teachers participated in this study group that met 11 times for a total of 30 hours from September through December 2003. The purpose of the group was to build skills in integrating literacy into classroom work and to develop units to support such integration.
In order to capture more detailed evaluation data from this study group, SSP staff designed a brief assessment, and had them write out their responses at the beginning of the school year and then discuss the questions as a group at the end of the school year. Participants echoed the clear intention and focus of the SSP staff and district literacy specialist in their various presentations of the importance and utility of non-fiction, informational texts. Participants also expressed the poignant tension between the additional demands and commitment required by SSP-type efforts and the benefits that can accrue as a result.
Quasi-experiment
SSP piloted a direct assessment of student literacy skills during the 2003-04 school year. The most striking element of the data is that all students demonstrated improvement from pre- to post-, except those who had achieved at or near the highest possible score on the pre-assessment. Assuming the integrity of the scoring process, it can be confidently asserted that the integrated sustainability/literacy lessons that took place between the pre- and post-assessments led to improved student skills in the first two out of the three questions/skill areas covered by the assessment rubric. This pilot quasi-experimental investigation could well become a replicable model for similar investigations of other PEEC or place-based programs if some of the research design and administration procedures were to be slightly
more refined and systematized.
Presentation of Educator and Student Survey Data: Barnes and Champlain Schools
The main thrust of the survey-related sections of this report is to present the findings of the effort to pilot a dose-response measurement strategy. Thus, this report takes an educative approach to presenting statistical findings. We omit some of the more descriptive statistical representations (e.g. simple bar graphs) in favor of putting more focused effort into helping the reader understand the meaning and power of the inferential statistics that undergird this dose-response measurement strategy.
Several potentially useful findings have emerged. A very strong correlation was found between amount of exposure to SSP (i.e. “dose”) and responses to survey items about teacher practice. Other positive, substantial, and statistically significant results were found for student time spent outdoors and student attachment to place. Several other outcomes showed little relationship to SSP exposure or were not statistically significant. Since the primary focus of the quantitative part of this evaluation was to pilot test the dose-response measurement strategy itself, there is still more to be learned from analyzing the data more exhaustively.
Conclusions
The SSP does not require that a specific topic, method, or activity be taught in a school. Rather, SSP encourages people to think about connections and to see the web that weaves together topics (e.g. art and writing), places (e.g. classrooms and neighborhoods), and people (e.g. students and local workers).
There is a parallel conceptual and structural evolution of the program taking place. As SSP’s sustainability-related content takes hold and begins to create a curricular web, so too the students, teachers, staff, and others involved in the process find themselves in an evolving framework of interconnectivity that has many benefits for the school and community alike.
A specific example of this is the impact of SSP’s literacy efforts. By focusing on integrating sustainability broadly into as many aspects of the literacy curriculum as possible, SSP itself is becoming more broadly integrated into the school and district.
In the broadest sense, what is happening with SSP is an expansion of perspective as
participants begin to see the interconnectivity of all things. More specifically, at Champlain Elementary School, the SSP model’s “testing grounds,” teachers are learning how all elements of their curriculum are or can be related, and are gaining awareness of how the curriculum is linked between grades and between special subjects, like art and music, and classrooms. They are refining their skills in curriculum planning built upon a new foundation of essential questions and sustainability. Moreover, students are seeing that their learning process extends well beyond the classroom, to their neighborhoods, and to other students, the schoolyard, their parents, and other community adults.
Implications for Practice
Evaluation provides the opportunity for program growth, reflection and refinement. Interpretation of the data used in this year’s evaluation points to a number of recommendations for program development, also called implications for practice. The following list offers a snapshot of the more detailed and extensive implications for practice provided in the report.
· Acknowledge from the start that the process of change itself does indeed require an investment of time, but that there is ample evidence that it is worth it in the longer term.
· Allow participants to explore and define the language pertinent to the project—particularly “sustainability” and “sustainable,”--and to select the best way to represent the concept for their school and community.
· Funding should be sought to allow for expansion of the summer institutes to include a greater number of days and greater variety of participants.
· Include all teachers equally from the beginning of program implementation.
· It would be valuable to have a school liaison who can assist a school in its third year as the site transitions away from heavy involvement of SSP staff.
· Assure that the social equity element of education for sustainability is addressed in as much depth as the environmental integrity and economic prosperity elements.
· Assure that all program staff feel adequately prepared to implement the area of focus, such as coaching teachers on literacy work or diversity issues.
· Consider building upon the quasi-experimental assessment exercise as a way to eventually provide a solid body of evidence about SSP’s direct influence on student achievement.
· The first round of analysis of the survey data collected in 2003-04 reveals some compelling confirming evidence of program success. Further exploration of the data is warranted.
· Continue to explore the benefits of a deeper collaboration and sharing of responsibility between program staff and evaluators.
| 