Similar English Learner Students, Different Results: Why Do Some Schools Do Better? (2007)

Williams, T., Hakuta, K., Haertel, E., et al. (2007). Similar English Learner Students, Different Results: Why Do Some Schools Do Better? A follow-up analysis, based on a large-scale survey of California elementary schools serving low-income and EL students. Mountain View, CA: EdSource.

This new extended analysis was based upon extensive survey data from 4,700 K-5 classroom teachers (80% or more at each school) and all principals in 237 California elementary schools from 137 different school districts across the state [in 2005]. These schools were initially randomly selected from 550 schools in California’s 25-35% School Characteristics Index band. All schools from this band have high levels of student poverty and low parent education levels; for this analysis we further narrowed our original sample to eliminate any school that didn’t have enough English Learner students to have an EL Academic Performance Index score.

The sample included low, middle, and high performing schools, which enabled an examination of the school practices that differentiate low from high performing schools.

School practice domains examined:

1. Prioritizing Student Achievement (Using Measurable and Monitored Objectives)
2. Implementing a Coherent, Standards-based Curriculum and Instructional Program
3. Using Assessment Data to Improve Student Achievement and Instruction
4. Ensuring Availability of Instructional Resources
5. Enforcing High Expectations for Student Behavior
6. Encouraging Teacher Collaboration and Providing Professional Development
7. Involving and Supporting Parents

Four broad effective schools practices were found in their analysis to have the most significant positive correlation with higher EL–API scores for elementary schools with high proportions of low income and Spanish speaking EL students:

1. Using Assessment Data to Improve Student Achievement and Instruction.
2. Ensuring Availability of Instructional Resources.
3. Implementing a Coherent, Standards-based Curriculum and Instructional Program.
4. Prioritizing Student Achievement (Using Measurable and Monitored Objectives).

Joe's Questions:

1. Why do some schools and districts use these practices more than others?
2. Do these practices have a causal relationship to higher EL-API scores? If so, what are the mechanism? Why do they work?
3. What conditions and contexts support or inhibit the use of these practices?
4. Are there any systematic reasons or conditions that explain the patterns of use of these practices? 

Comments:
+ the sample of schools they chose was fairly narrow - they only look at schools in the 25th to 35th percentile band of California’s School Characteristics Index (SCI) which basically means that most of the children in these school are from low-income households. The results of study might be significantly different with schools that serve  ELs with higher-SES or even lower-SES so there is some doubt and concerns about how generalizable the results are of the study to other schools who serve ELs.

+ There is this interesting comment in the technical appendicies: "The outcomes [the 2005 EL Base API] expressed as percentages underwent two transformations in order to avoid the violations of linear regression assumptions that otherwise ensue when using a proportion as the dependent variable." The conclusions of the researchers are based on looking at which schools practices have the most significant positive correlations with higher EL–API scores for elementary schools. There isn't enough information in the technical appendicies to determine if the two transformations they performed were appropriate to determine the correlations.

EdSource Web Page for Similar English Learner Students, Different Results

Different Tests, Different Answers - Papay (2011) - VAM

Papay, J. P. (2011). Different Tests, Different Answers. American Educational Research Journal, 48(1):163-193.

Conclusions and Implications
Much more variation in teacher value-added estimates arises from the choice of outcome than the model specification.

Instead, Papay's results suggest that test timing and inconsistency, such as measurement error, play a much greater role. In particular, the finding that the timing of the test alone may produce substantial variation in teacher productivity estimates across outcome measures raises important questions for teacher accountability policies.

The analyses presented in this research suggest that the correlations between teacher value-added estimates derived from three separate reading tests—the state test, SRI, and SAT—range from 0.15 to 0.58 across a wide range of model specifications.

Although these correlations are moderately high, these assessments produce substantially different answers about individual teacher performance and do not rank individual teachers consistently. Even using the same test but varying the timing of the baseline and outcome measure introduces a great deal of instability to teacher rankings.

Therefore, if a school district were to reward teachers for their performance, it would identify a quite different set of teachers as the best performers depending simply on the specific reading assessment used.

Papay's results suggest that test timing also contributes substantially to differences in teacher effectiveness estimates across outcome measures. This is an important finding that merits further study.

If policymakers intend to continue using value-added measures to make high-stakes decisions about teacher performance, more attention should be paid to the tests themselves. Currently, all value-added estimates of teacher effectiveness use tests designed to measure student, not teacher, performance. The ideal properties of tests designed to identify a district’s best teachers may well differ from those designed to assess student proficiency.

Furthermore, the timing of tests must be considered more carefully. For example, the practice of giving high-stakes tests in early spring may not matter much for inferences about student performance in the district—having an assessment of student skills in February may be just as useful as one in May. However, decisions about timing have substantial implications for teacher value-added estimation.

Given the amount of inaccuracy in any single assessment of teacher performance—whether based on test scores or observations—combining multiple sources of information could provide schools and teachers with a better sense of their performance on a wider range of domains.

While multiple measures may provide a more robust assessment of teacher performance and may mitigate the effects of measurement error from using any single test, policymakers and district officials must take care in deciding how to combine measures. Douglas (2007) found that using multiple assessments increases evaluation reliability when the measures are highly related, but this result is not consistent with less correlated measures.

Importantly, additional research is needed into the different implications of high- and low-stakes tests for estimating teacher effects. Teachers who appear to perform well using a high-stakes examination but not well with a low-stakes test may be effectively teaching state standards or may be engaged in inappropriate coaching.

Notes:
All value-added models rely on the assumption that teacher effectiveness can be estimated reliably and validly through student achievement tests.

In practice, the reliability of student achievement growth is lower than that of the individual tests themselves. (jc: e.g., 5th grade CST test is different from the 4th grade one, so measuring the gain score can be tricky)

Additional variation in teacher estimates arises from the nature of testing. Students take tests on different days and at different times of the year. Because students, particularly those in urban schools, have relatively high absenteeism and mobility, the students present to take each test may vary substantially. Thus, teacher value-added estimates may vary across outcomes in part because different samples of students take each test.

As seen in Table 5 (on p. 180), approximately half of the teachers who would earn a $7,000 bonus using the state test would lose money if the district used the SRI instead.

The average teacher in the district would see his or her pay changed by $2,178 simply by switching outcome measures. Interestingly, the instability in teacher estimates across outcome measures is much greater for teachers in the middle two quartiles. (p181)

Papay found that differences in test content and scaling do not appear to explain the variation in teacher effects across outcomes in this district. The different samples of students who take each of the tests contribute somewhat, but they do not account for most of the differences. Test timing appears to play a greater role in producing these differences. Nonetheless, it does not explain all of the variation, suggesting that measurement error also contributes to the instability in teacher rankings. (p183)

Papay made comparisons that suggest that summer learning loss (or gain) may produce important differences in teacher effects. Here, the fall-to-fall estimates attribute one summer’s learning loss to the teacher, while the spring-to-spring estimates attribute a different summer’s loss. Thus, the fact that the fall-to-fall and spring-to-spring estimates produce substantially different answers likely reflects, in part, the inclusion of a different summer in each estimate. (p187)

Integrating inquiry science and language development for English language learners - Stoddart (2002)

Stoddart, T., Pinal, A., Latzke, M., and Canaday, D. (2002). Integrating inquiry science and language development for English language learners. Journal of Research in Science Teaching, 39(8):664-687.

Context:
The context for this study is Language Acquisition through Science Education in Rural Schools (LASERS), a National Science Foundation–funded Local Systemic Change project in central California that prepares experienced teachers to provide inquiry science instruction to Latino students learning English as a second language. The science–language integration rubric was developed to provide a conceptual framework for teacher staff development activities and to gauge changes in teachers’ beliefs and practice.

Research questions:
(a) How do teachers conceive of science language integration? and (b) What are the cognitive demands that underlie the development of teacher expertise in domain integration?

Methods: 
Interviews were conducted with 24 first- through sixth-grade teachers (21 female, 3 male) who participated in the LASERS summer school academy in 1998. The majority of the 24 teachers (19 of 24) had more than 3 years of teaching experience. The sample includes teachers with differing levels of participation in the LASERS project and a range of teaching experience. Therefore, they represent a range of perspectives on language-science integration.
The literature on curriculum domain integration, the development of expertise in teaching, and cognitive complexity are used as a framework for a rubric that describes science– language integration as a continuum from isolated domain-specific instruction to fully-integrated synergistic instruction with the emphasis on commonalties in structure and process across domains.

The view of integration presented in this article is based on Huntley’s definition of synergistic integration. Effective language instruction enhances the learning of science concepts, and effective science inquiry instruction enhances language development and promotes the development of higher-order thinking skills. This approach aligns with work on the integration of reading and writing with science instruction

In viewing the teaching of science and language as a synergistic process, we support the view of bilingual educators such as Cummins (1994) and Met (1994), who argue that the teaching of English and subject matter content should be so integrated that "all content teachers are also teachers of language" (Cummins, 1994, p. 42) and "view every content lesson as a language lesson" (Met, 1994, p. 161). There is currently little information available, however, on successful approaches to preparing teachers to teach inquiry science to second language learners (Lee & Fradd, 1998).

This evolution of teacher understanding could be characterized as a shift from "knowing that" to "knowing how" (Dreyfus & Dreyfus, 1986; Kuhn, 1970; Polanyi, 1958). "Knowing that" understanding is characterized by a rule-governed, theoretical orientation, whereas "knowing how" is the flexible application of principles in practice.

Dreyfus framework: novice, advanced beginner, competent, proficient, expert

Conclusions
The traditional approach to educating English language learners, which separates the teaching of language from the teaching of science content, presents an unnecessary obstacle to the academic progress of language minority students.

The findings of this report suggest the need to rethink staff development activities and science teacher education. The artificial and rigid distinctions between the role of science teacher and language teacher must be broken down.

The critical point is that language processes can be used to promote understanding of content across all subject matter domains, and that language use should be contextualized in authentic and concrete activity. In states such as California, where language minority students represent a significant percentage of the school-age population, methods of English language development should be integrated into all elementary and secondary subject matter methods classes and staff development programs. Integrated instruction will assist language minority students in mastering the English language and simultaneously improve their achievement in academic subjects.

See:
Huntley, M.A. (1998). Design and implementation of a framework for defining integrated mathematics and science education. School Science and Mathematics, 98, 320–327.

Conceptual change pedagogy - Stofflett and Stoddart (1994)

Stofflett, R. T. and Stoddart, T. (1994). The ability to understand and use conceptual change pedagogy as a function of prior content learning experience. Journal of Research in Science Teaching, 31(1):31-51.

The research questions were
1 . Were there any differences in the degree of science content understandings held by teacher candidates in the traditional and conceptual change groups following science content instruction?
2. What effects did the treatments have on the teacher candidates’ understandings of science pedagogy?
3. What effects did the treatments have on the teacher candidates’ ability to use conceptual change pedagogy in their instructional practice?

Participants:
27 college seniors enrolled in a university elementary teacher education program located in the western United States. The subjects were randomly placed by the university certification admissions committee into two sections of a 10-week science methods course. There were 17 conceptual change (14F, 3M) and 10 (8F, 2M) traditional subjects, All subjects had previously completed the two science content courses required for their certificate. The most frequently taken courses were "Common Medicines ," "Trees and Shrubs ," and "Energy Resources."

The research presented in this article is based on three hypotheses.

• First, the pedagogy through which teachers learn science content is a primary determinant of how they understand and teach science.
• Second, in order for most elementary teacher candidates to develop conceptual understanding of science content they need to reconstruct their subject-matter knowledge through a process of conceptual change.
• Third, the experience of learning science content themselves through the conceptual change process will facilitate the understanding and application of conceptual change pedagogy in science instruction.

Posner et al. framework
The conceptual change instruction was developed around the four conditions necessary for accommodation of a scientific conception as described by Posner et al. (1982). These conditions are intelligibility (ability to understand the concept), plausibility (believability and consistency of the concept), dissatisfaction with existing conceptions, and fruitfulness of the concept for use in external contexts.

Five step strategy

• The first step involved the diagnosis of misconceptions.
• The second step involved exploring the phenomena in question using guided discovery methods.
• The third step consisted of a discussion of the results of the experiments.
• The fourth step was used to facilitate the development of dissatisfaction with the preexisting conceptions.
• When students were able to distinguish between scientifically accepted ideas and naive theories, the instruction moved to the final step, where teacher candidates were given the opportunity to develop fruitfulness by applying the new concepts to real-world examples. The instructor asked the students to provide examples of the phenomena occurring in their own lives and to explain the concept in context. For each concept explored, the five-step model was used.

Findings

• The teacher candidates in the conceptual change group had significantly higher gains in their content understandings than did those in the traditional group.
• This finding supports previous research (e.g., Champagne et al., 1985; McClosky, 1983) that traditional science instruction, even that with a heavy emphasis on laboratory work, does not significantly improve students’ conceptual understandings. Challenging students’ countertheories must be an integral part of science instruction, if students’ conceptual knowledge is to improve.
• This finding was also consistent with the idea that teachers teach as they were taught (Stoddart & Stofflett, 1992).
• The subjects in the conceptual change group were better able to translate their cognition of conceptual change pedagogy into practice than were the traditional subjects.

Recommendations based on study outcomes

• The findings of this study indicate that this round of reform will fail, as previous rounds have, unless the recommendations on teaching and learning are applied to teachers as well as students.
• The data presented in this study indicate that many teacher candidates hold similar misconceptions and learn in the same way as students do: To improve their understanding they also need to be taught conceptually.
• The pedagogy used in college science content courses will need to incorporate the new views of teaching and learning if teacher knowledge is to be improved.
• Showing teachers how to use innovative curriculum and instructional materials and modeling innovative practice will not be sufficient to bring about changes in their science teaching.
• Teachers must experience the innovative pedagogy first as learners before they can develop intelligibility of the methods being taught.

Conclusions

• The findings of this study draw attention to a fundamental dilemma in science education reform: the expectation that teachers can learn to be constructivist teachers when they have not been constructivist learners.
• Educational reformers, however, typically expect teachers to change their pedagogical conceptions by being shown and told about innovative practice (Shulman, 1986; Stoddart, 1993)
• the authors propose that the use of conceptual change pedagogy in teacher preparation is the one that is most likely to bring about change in teacher candidates’ conceptions of teaching and learning precisely because it involves the challenging of preconceptions and reconstruction of knowledge structures.
• These findings are a powerful example of constructivist theory in practice: They demonstrate the importance of the personal construction of knowledge over teaching as showing and telling.