Place-Based Education Evaluation Collaborative
PEEC Home Page | Research | Tools | Resources | Reports | Search
Cross Program Eval Report 02-03

Place-based Education Evaluation Collaborative
Cross-Program Evaluation Report
2002-2003


kids-and-ranger-web.jpgPresented to:
Community
Mapping Project:  
Vermont Institute of
Natural Science and
The Orton Family Foundation

Sustainable Schools Project:
Shelburne Farms and
The Vermont Education for Sustainability Project

CO-SEED:
Antioch New England Institute of Antioch New England Graduate School,
in partnership with the Appalachian Mountain Club, Prescott Farm of the Audubon Society of New Hampshire, Hulbert Outdoor Center, and
Zoo New England

Forest for Every Classroom:
Conservation Study Institute of the National Park Service, Shelburne Farms, The Marsh Billings Rockefeller Historical Park, Green Mountain National Forest Service, and the National Wildlife Federation

The Upper Valley Community Foundation



Prepared by:
Amy L. Powers
Program Evaluation & Educational Research Associates

September 1, 2003

(What follows is the Executive Summary. The full report is attached as a pdf file.)



EXECUTIVE SUMMARY

Introduction    
Place-based Education Evaluation Collaborative
In 2002, several New England foundations and educational organizations came together to form the Place-based Education Evaluation Collaborative (PEEC) with the intention of evaluating their individual programs and laying the groundwork for broader research into the effectiveness of these models in attaining mutual objectives. PEEC has three main purpose:

To serves as a learning organization for program developers, fueling internal growth and program development for the individual organizations;
To develop, identify and disseminate evaluation techniques, tools and approaches that can be applied elsewhere; and
To contribute to the research base underlying the field of place-based education and school change.

The goals of the four collaborating programs vary somewhat but common themes are:
·       enhanced community/school connections
·       increased understanding of and connection to the local place
·       increased understanding of ecological concepts
·       enhanced stewardship behavior
·       increased academic performance in students
·       improvement of the local environment
·       improvement of school yard habitat and use as teaching space
·       increased civic participation
Collaborating organizations
PEEC is a unique partnership of organizations whose aim is to strengthen and deepen the practice and evaluation of place-based education initiatives. PEEC programs (and organizations) include the CO-SEED Project (Antioch New England Institute); the Community Mapping Program (the Orton Family Foundation, Vermont Institute of Natural Science,); the Sustainable Schools Project (Shelburne Farms, Vermont Education for Sustainability Project); and A Forest for Every Classroom Project (Shelburne Farms, National Wildlife Federation, the Conservation Study Institute, Marsh-Billings-Rockefeller National Historical Park, Green Mountain National Forest). In addition, the Upper Valley Community Foundation provides funding and support for several of these programs through its Wellborn Ecology Fund, as well as financial, administrative and staff support for collaborative evaluation and research efforts.
Brief overview of the four programs being evaluated
The following is a brief overview of each of the four programs evaluated during the 2002-2003 school year as part of PEEC.
The Community Mapping Program (CMP) is a partnership of the Orton Family Foundation and the Vermont Institute of Natural Science. The Community Mapping Program supports students, educators, and community groups in a process of local inquiry. Middle and high school students work with community groups, conduct fieldwork, and use tools including Geographic Information Systems (GIS) and Global Positioning Systems (GPS) to explore their communities and address specific local needs.
A Forest for Every Classroom (FFEC) has worked with 30 Vermont teachers over the past two years. Shelburne Farms, in conjunction with the Marsh-Billings-Rockefeller National Park, the Green Mountain National Forest, the National Wildlife Federation and many other local partners, provides sophisticated forestry education to many collaborating teachers and schools.
The Sustainable Schools Project (SSP) is a partnership program of Shelburne Farms and Vermont Education for Sustainability (VT EFS). The project uses sustainability as the integrating context on a school-wide level, seeking to connect science and literacy, ecology and community, and beginning inquiry and practice to improve the quality of life for all. SSP focused its first school year (2002-2003) piloting its program in one urban elementary school in Burlington, VT.
The CO-SEED (Community-based School Environmental Education Project has been working collaboratively with a dozen schools, six environmental learning centers and a diverse array of rural and urban New England communities for the last five years. CO-SEED aspires to creating a synergistic relationship between school improvement, community development and the preservation of environmental quality. This requires extensive collaborative engagement of town officials, teachers, administrators, scientists, community officials and environmental educators.
Theories of Change in Place-based Education
Increased connection to place results in increased civic engagement and greater “social capital.”  
Place-based education rooted in the research of Piaget—children are more successful with active learning and interaction.  
Students who are engaged in real-world learning are more likely to succeed than those who learn the same type of material from more abstract text books.

Figure 1 is offered as a working model for the theory of change behind place-based education.




Developing a Methodology
Evaluating complex programs
The evaluations of these four programs attempt to understand whether and how the models provide opportunities for educators, students, schools, and communities change their practices in the short, intermediate and long term.  The following table attempts to distill common outcomes shared by two or more of the programs:

Common projected outcomes       
Short term:     
·       Connections are forged and partnerships are formed between the school and the community         
·       Student learning occurs out in the community OR teachers use community in their teaching        
·       Teachers understand that the program is related to standards and serves as a tool for teaching, not an add-on curriculum        
·       Students are more engaged in learning through service OR teachers encourage service learning    
·       Schoolyard/school sustainability improves (CO-SEED, SSP)        
Intermediate/Long Term  
·       Students gain the knowledge, motivation and/or skills for stewardship/citizenship behavior      
·       Teachers and/or schools implement or adopt place-based education beyond the program intervention        
·       Social capital increases in the community (FFEC, CMP, CO-SEED)  
Long Term       
·       Civic society is strengthened. Communities have enhanced sustainability, vitality, and/or civic participation.  
·       Schools engage in systematic improvement efforts (SSP, CO-SEED) 
·       Students develop a greater attachment to place (FFEC, CMP)      
Note: Where no program(s) are indicated, ALL share this projected outcome.
Developing a cross-program evaluation strategy
PEEC developed the following cross-program questions for 2002-2003, using various instruments to measure change in teachers, students, and community members:

1.      Evaluating process strengths and challenges—What are the greatest strengths and challenges of each program model? How can these programs learn and grow from one another?

2.      Measuring teacher practice change—How does participation in one of these place-based education programs change teachers’ teaching practices. 3.

3.      Measuring student civic engagement—How does participation in one of these place-based education programs affect students’ level of civic engagement? (Possible sub-questions: Are students more engaged in service-learning projects in school? Are students more engaged in service-learning projects outside of school time? What types of activities are these? To what degree do students participate on town boards, commissions, committees?)


Findings and Discussion

The intention of hiring one evaluator to work with all four programs on a collaborative basis was to heighten the level of shared learning that could transpire from comparative analysis in the following areas:
1. Process findings: Greatest strengths and challenges of PEEC programs
The tables below provide highlights of process strengths and challenges for each of the four PEEC programs. These findings are based on a 12 month evaluation of each of the four programs.                     
The CO-SEED Project     
Strengths       
SEED Team allows for in-depth relationships to develop between community and school     
Higher ed partner provides credibility, expertise, access to interns at sites   
Early attention to community involvement encourages long-term community-school connections with long term community change more likely  
Three-year (minimum) commitment to a site               
Visible examples of success at each site promote buy-in, inspiration    
Strong support from outside the school building in most sites (towns, ELCs, etc.)       In some sites, only limited portion of whole school is involved with CO-SEED    
Whole school model increases likelihood of school administration’s support for teacher practice changes 
Challenges      
Environmental Learning Center (ELC) representatives bring different skills to each site; training is inconsistent
Could focus more on public relations and program/project recognition    
Emphasis on curriculum planning skills inconsistent; professional development focuses more on exposing teachers to resources, content, place-based ed. strategies       
ELC capacity-building not consistently emphasized
Lack of time, curricular pressure: CO-SEED sometimes seen as “add on layer”     
Greater attention to follow-up support with “emeritus” sites may be warranted for greater long-term sustainability      

The Sustainable Schools Project 
Strengths       
Great effort and success at involving the whole school  
Continual presence of predictable, experienced personnel as school-organization liaison 
Emphasis on curriculum building as a basic step toward long-term change in teacher practice and school wide curricular integration      
Good balance of building capacity and providing support, process facilitation   
Rapport and trust established – consistency, flexibility, knowledge, skills, responsiveness, sensitivity        
Provision of resources and contacts     
Role-modeling teaching practices and providing one-on-one coaching      
Challenges      
Community advisory board could be more integral to program/site develop’t       
Limited age-appropriate resources for younger grade (e.g. sustainability literature for K-2)
Communication challenges: inter-grade, some community partners, parents could be better apprised of program
Teachers not required to document emerging curriculum                   
Relationships with university students could be strengthened through clearer terms and guidelines       
Time limitations: summer training too short, program start-up too quick 
Lack of time, curricular pressure: SSP sometimes seen as “add on layer” 

A Forest for Every Classroom Program    
Strengths               
Nurturing environment and respect for participants engenders their commitment and dedication    
Diverse sponsoring partners provide balanced viewpoints, extended access to resources   
Year-long contact with partners and amongst teachers creates relationship and community 
Unique link between schools, non-profits and public sector/public lands 
A well-crafted, skillfully-executed program             
Role-modeling sound teaching practices throughout institutes    Greater clarity of partner roles        

Challenges
Due to location, primary community partners not easily accessible to all teachers; leads to under-utilized follow-up visits and prohibits access to national public lands for some teachers     
Service-learning component, with tangible skill-building, less emphasized       
Some teacher participants isolated in schools without administrative or partner-teacher support
Lack of time, curricular pressure: FFEC sometimes seen as “add on layer”        
Need more help on “the how” – implementing into existing structure
        
The Community Mapping Program   
Strengths               
Links with an individual community partner increases likelihood that tangible, useful projects emerge   
Tangible skills and resources provided to educators encourages true service- learning projects to develop with students 
Staff offers flexibility and extensive on-going field/classroom tech. support   
Strong summer institutes in which completing a service-learning project is actually modeled     
Explicit emphasis on partnerships with community organizations
Growing efforts at networking between current and past CMP teachers at institutes and roundtables       
Financial assistance provided to participants   
Challenges
Limited administrative buy-in and single-teacher participation creates unstable base of operation, limited internal support for teachers, less spread of effect.        
More institute time needed to develop project and learn GIS; acquisition of technological skills can be daunting for teachers   
Program expectations could be articulated more clearly to participants  
Teachers are interested in being provided with more project examples and sample activities
Relationships between teachers and community partners and students and community partners could be stronger             
Some teacher participants isolated in schools without administrative or partner-teacher support 
Lack of time, curricular pressure: CMP sometimes seen as “add on layer” 

2. Highlighting strengths across the programs
Four particular areas emerged as consistent process strengths across the four programs evaluated:
Use of community partners
High quality program staff
Sustained program intervention
The programs’ summer institutes
3. Examining challenges evident across the programs
Three particular areas emerged as consistent process challenges across the four programs evaluated (both internal and external to the programs):
Lacking time for teachers to integrate new material
Helping teachers acquire curriculum planning skills
Communication (with parents, community members, other teachers, administrators)

Figure 2 suggests a model for incorporating both teacher professional development and whole-school change in order to focus on strengths of the four programs and minimize challenges.
4. Cross-program outcomes: Changes in teacher practice
Six impacts on teacher practice were found to be most salient across the four programs. These include:
·       Use of local places/ resources
·       Interdisciplinary teaching
·       Collaboration with other teachers
·       Personal growth/Leadership
·       Service-learning
·       Stronger curriculum planning skills
5. Emergent findings: other roads to follow
Two particular areas emerged as potentially useful to pursue as widespread evidence of the usefulness and effectiveness of place-based education:
·       The importance of community-based learning for special needs students
·       The impact of place-based education on student motivation toward learning and engagement in school


Recommendations

Five significant themes emerged across all four programs:
Effective start-up approaches
·       Assure that terminology is clearly defined
·       Clearly define the program’s goals and theory of change
·       Provide educators or sites with a checklist of the skills, resources, etc. that are available and from whom (menu of options, resource list, etc.)
·       Document success early on.
·       Communicate about, and engage participants in defining, the role of program evaluation up front and provide incentives for on-going participant contribution to the process.
Creating teacher, administrator and community buy-in
Teachers
·       Offer high-quality, “high-touch” (e.g. respect, communication, comfort, support, rapport) professional development opportunities.
·       Provide tangible resources such as money, publications, examples of curricula, and project models that teachers may begin using immediately.
·       Provide training in tangible skills.
·       Offer skill-building in curriculum development
·       Involve alumni or emeritus participants.
·       Provide help “on the ground”
·       Work to reduce the burden of implementing a new program
Administrators
·       Educate administrators early on in the process
·       Ensure understanding of terminology such as place-based education.

Community
·       Establish consistent public relations efforts
Partnerships and collaboration
Encourage (or require) teachers to participate in teams.
Provide help in locating community partners
Help build capacity of community partners
Develop recognition of community partners, volunteers and parents
Communication: the key to lasting partnerships
Facilitate networking between schools, teachers, volunteers, administrators and program staff.
Link past and present sites, and past and present program participants
Other Recommendations
Involve teachers, administrators, community members in program evaluation from the start.
Lead teachers in learning new assessment techniques for community-based, place-based, service-learning curriculum.
Plan for diffusion of program concepts beyond directly targeted audience.
Suggestions for future PEEC research
1.      Develop annotated bibliography of relevant research
2.      Refine individual programs’ logic models to show the flow of logic for each audience or outcome: the theory of change for teachers; theory of change for students; theory of change for community.
3.      Develop a cross-program logic model.
4.      Build understanding and evidence of place-based education’s intrinsic workings through qualitative inquiry across the four programs.  
5.      Identify quantitative measures for all four programs.

Conclusions

In some senses PEEC is both a microcosm of the larger place-based education initiative and a mirror of the individual place-based projects it promotes. At their various levels of scale, each of these entities promotes greater collaboration, a greater attention to interdisciplinarity, and an attention to the strength inherent in diversifying one’s base of support. Examining the PEEC programs together builds the credibility of each of these programs as well as offering broader-based knowledge to the field.

The first year of collaboratively evaluating PEEC’s programs has revealed several conclusions:

·       The four programs are clearly strong vehicles for enhancing education. In particular, all four programs demonstrate commendable success at promoting teacher practice change.

·       With some variability, teacher practice is affected in consistent ways by the place-based education programs.

·       By highlighting outcomes salient to all four programs, we begin to suggest the power of place-based education as a broader educational approach.

·       By highlighting process strengths, challenges and opportunities, the four programs have the opportunity to learn both from their own efforts and from those of other programs. Quite consistently, recommendations specific to one or two programs have powerful implications for all four.

·       All four programs are received very positively and are highly valued by a range of stakeholders, from teachers to students to administrators to community individuals and organizations.

·       Patterns in the data gathered this year for the individual program evaluations suggest that there were positive outcomes emerging for all of the intended audiences, beyond teachers: students, schools, organizations and communities.

·       There are ample opportunities for growth and refinement within the program models, with some challenges being internal and others external to the programs.

·       It is worth examining how program offerings might be enhanced not only by refining themselves but by merging the key strengths offered by both the professional development and school improvement models.

·       This year’s research revealed that there are multiple fruitful roads to follow to continue to examine the processes that contribute to successful programs and the outcomes being sought.



Attachments:

PEEC X-Prog 02-03 Report we.pdf
583k
Attachments are downloaded and saved on your computer. Some files will open automatically, but you may have to open them separately, outside of your browser. PDF files are opened using Adobe Acrobat Reader, available at Adobe's web site.
Last Updated: Friday, Sep 12, 2003


PEEC Home Page | Research | Tools | Resources | Reports | Search