We work with the Science Museum of Minnesota’s Department of Evaluation and Research and the University of Minnesota’s Center for Applied Research and Educational Improvement (CAREI) to constantly understand and improve the impact of our professional development.
Professional development at Science House is not about a collection of activities to take into the classroom or program. It’s about changing mindframes, taking collective responsibility when learners don’t do well, changing our own practice, and creating inclusive learning environments for each and every learner.
Similarly, leadership professional development at Science House is not a train-the-trainer model; it is an adaptive approach. The distributed leadership processes that we promote and support are non-linear, creative, iterative, and context-specific. Rather than program fidelity, we are concerned with outcome fidelity.
The outcomes we are interested in are that participants adapt, refine, and implement research-based ideas and strategies with their colleagues and learners to increase inclusion, deepen learning, and create positive institutional change.
Our assessments and findings:
Case Studies: Assessing Impact on Organizations through Adaptive Implementation
We are currently working with CAREI to conduct case studies on districts and consortia involved in PAGE. While this work is ongoing, here are ways that groups we work with have creatively implemented what they’ve learned through Science House. They have…
- scrutinized unexamined beliefs that operate in schools/districts and serve to advantage some groups of students while disadvantaging others,
- discontinued ability tracking in science, math, and language arts sequences,
- analyzed status issues within classrooms, buildings and district staff structures, and worked to equalize status and increase participation,
- recognized the role of culture and society in STEM,
- modified STEM curricula, encouraging learners to investigate the social implications of STEM knowledge,
- modified language arts curricula, focusing on racial equity and social justice
- explicitly broadened conceptions of STEM and who does it,
- flexibly employed a variety of discussion structures and facilitation strategies that actively engage all present in sense- and decision-making,
- attended to both intellectual and affective needs in group settings,
- focused on changing cultural paradigms and relationships rather than technical elements and structures, and
- developed and implemented equity-oriented strategic plans.
We have strong quantitative data that indicates that our professional development leads to statistically significant increases in understanding, preparedness, and application of ideas around identity, distributed leadership, and access and equity in STEM education.
Science House and the Science Museum of Minnesota’s Department of Evaluation and Learning have closely collaborated over the past three years to develop and validate the PAGE Knowledge, Preparedness, and Application (KPA) Scale Survey to assess changes in teacher/leader knowledge, preparedness, and application of concepts related to equitable STEM education as articulated in the Science House Framework for Access and Equity in STEM Education. KPA analysis for the most recent PAGE Regional cohort shows that all factors had considerable growth. Over 95% of the items (60 out of 63 items) had statistically significant changes between pre and post scores.
We have a large qualitative data set that consists of daily open-ended written reflections from each person. At the end of each day, they write about their confusions, comprehensions, and connections. At the end of an institute, they each write a reflection addressing their learning from the week and the implications for their work moving forward. Here are a few representative statements:
“I have a better understanding of where the barriers are in STEM education. I now appreciate the method used that allowed us to self-reflect on our paradigm and come to a better place in how it needs to shift. You have provided us with some important tools that will allow us to shift paradigms in ourselves, our own departments and institutions in a kind and respectful way.” —PAGE National Informal STEM Center Participant
“This experience has transformed the way I look at my own institution and has given me many tools to begin changing many of the inequalities I have observed during my work. I will also bring this work to the institutions that I sit on boards of to help start a process of reflection on equity/access issues.” —PAGE National Informal STEM Center Participant
“My place as an agent of change is more important than I realized. I can help others to see inequities in how we engage learners. Looking at who enters the conversation about STEM practices and content; who dictates the norms of what is appropriate; who gives reinforcement of existing paradigms; who is left out as a result.” —PAGE National Informal STEM Center Participant
“I never viewed myself as a leader. I’ve always felt marginalized and dismissed, especially by the (bigger, louder) voices in the room. This week I feel myself emerging as a leader, albeit a quiet leader, and I’m looking forward to taking this confidence back and working for the students who have felt the ways that I have in the past.” – PAGE Regional District Participant
“For some reason, today has me doing more reflection about how I need to change, in a way that has me feeling uncomfortable. I will need some time to reflect on the things that I think I need to do differently—that is the cause for the discomfort—but, I know this is an important part of learning and growth so I am grateful.” – PAGE Regional District Participant
“What stories can I hear from the world around me? The wind rustles the leaves outside, and that tells so many different stories. When I dive, the water roars and whispers in my ears—I don’t know the language of the ocean, but I know I can explain it through natural means. So if I need to listen to expand my story of science, I must also listen to understand the story of leadership. I don’t understand my desire to listen yet. But I will, and it will make me a better scientist and leader.” – PAGE Regional District Participant
“I think this week has added the dimension of relationships to any STEM conversations I have or work I plan. Working with several districts, all in different places with their perception of STEM, will be a challenge. Many narrowly view it as curriculum, so it will be challenging to think about how to help broaden their thinking without blowing their mind.” – PAGE Regional District Participant
“This week has been validating, inspiring, frustrating, tiring and the most important training I have experienced. My undergrad is in anthropology, and I felt connected to that side of me all this week. The work you do is important. The work you do will be challenged. The work you do is not yet comprehendible to everyone, but your work is right for our future.” – PAGE Regional District Participant
Mixed Method: External evaluation findings for PAGE and Nexus district leaders
External evaluation of our programs has been conducted by Vivian P. Johnson, Hamline University; Bhaskar Upadhyay and Timothy Sheldon, both of the University of Minnesota; and Amy Grack Nelson of the Science Museum of Minnesota. Analysis has been conducted using end-of-day written reflections, pre/post KPA scale survey, pre/post participant analysis of classroom video, site-based activity reports, and interviews.
Overall we have found that (1) leaders experienced paradigm shifts regarding intersectionality and student identity formation, access and equity in STEM education, and professional culture and leadership, (2) they applied this understanding in their formal and informal work as STEM leaders in professional development and/or organizational settings, (3) they used the PAGE PD modules at triple the projected rate as they conducted formal and informal professional development with colleagues, (4) many leaders moved into increased leadership positions, and (5) they placed a high value on being part of a community of learners.
With support from the National Science Foundation, Science House is currently collaborating with Tripod Education Partners to use the 7C survey for professional growth, developed by Harvard Professor Ronald Ferguson and colleagues, to study the impact of our work on classroom practice and teacher effectiveness.
Science House uses and recommends this measure for a number of reasons. First, the survey has been extensively tested to establish reliability and validity in K-12 classrooms, and second, it has been shown to be a predictor of student achievement gains (see below). Third, the survey’s constructs are well aligned with the Science House Framework for Access and Equity in STEM Education, which guides our work. Finally, the survey captures the viewpoints of the people most directly affected by a teacher’s potential changes in instruction and support—the students.
The Measures of Effective Teaching (MET) project, which investigated ways to reliably measure teacher effectiveness, showed that students of teachers who score in the 75th percentile of the 7Cs survey achieve higher-than-expected student achievement gains. For example, the predicted difference for months of learning on state math tests is a positive difference of 4.7 months for teachers who score in the 75th percentile as compared to teachers who score 25th percentile on the 7Cs. According to the Tripod Education Partners, “Findings from the MET Project suggest that the 7Cs are significantly more reliable than more commonly used measures of teaching effectiveness including value-added estimates and observational ratings” (p. 43).
Specifically, the 7Cs student survey is a measure of students’ perceptions of teacher practices, classroom learning conditions, and student engagement. The survey consists of questions that ask students to rate their teacher on seven dimensions of effectiveness:
- Care: The emotional warmth of the student-teacher relationship
- Control: The extent to which the teacher cultivates an on-task class
- Clarify: Teacher behaviors that promote student understanding
- Challenge: The effort and rigor of a teacher’s classroom
- Captivate: The extent to which teachers spark and maintain student interest in learning
- Confer: The extent to which teachers elicit and value student voice
- Consolidate: The extent to which teachers help students organize course material