Papay, J. P. (2011). Different Tests, Different Answers. American Educational Research Journal, 48(1):163-193.
Conclusions and Implications
Much more
variation in teacher value-added estimates arises from the choice of outcome
than the model specification.
Instead, Papay's results suggest that test timing and inconsistency,
such as measurement error, play a much greater role. In particular,
the finding that the timing of the test alone may produce substantial variation
in teacher productivity estimates across outcome measures raises important
questions for teacher accountability policies.
The analyses presented in this research suggest that the correlations between
teacher value-added estimates derived from three separate reading
tests—the state test, SRI, and SAT—range from 0.15 to 0.58 across a wide
range of model specifications.
Although these correlations are moderately
high, these assessments produce substantially different answers about individual
teacher performance and do not rank individual teachers consistently.
Even using the same test but varying the timing of the baseline and outcome
measure introduces a great deal of instability to teacher rankings.
Therefore,
if a school district were to reward teachers for their performance, it would
identify a quite different set of teachers as the best performers depending
simply on the specific reading assessment used.
Papay's results suggest that test timing also contributes substantially
to differences in teacher effectiveness estimates across outcome measures.
This is an important finding that merits further study.
If policymakers intend to continue using value-added measures to make
high-stakes decisions about teacher performance, more attention should be
paid to the tests themselves. Currently, all value-added estimates of teacher
effectiveness use tests designed to measure student, not teacher, performance.
The ideal properties of tests designed to identify a district’s best teachers may
well differ from those designed to assess student proficiency.
Furthermore, the
timing of tests must be considered more carefully. For example, the practice of
giving high-stakes tests in early spring may not matter much for inferences
about student performance in the district—having an assessment of student
skills in February may be just as useful as one in May. However, decisions
about timing have substantial implications for teacher value-added estimation.
Given the amount of inaccuracy in any single assessment of teacher
performance—whether based on test scores or observations—combining
multiple sources of information could provide schools and teachers with
a better sense of their performance on a wider range of domains.
While multiple measures may provide a more robust assessment of
teacher performance and may mitigate the effects of measurement error
from using any single test, policymakers and district officials must take
care in deciding how to combine measures. Douglas (2007) found that using
multiple assessments increases evaluation reliability when the measures are
highly related, but this result is not consistent with less correlated measures.
Importantly, additional research is needed into the different implications
of high- and low-stakes tests for estimating teacher effects. Teachers who
appear to perform well using a high-stakes examination but not well with
a low-stakes test may be effectively teaching state standards or may be
engaged in inappropriate coaching.
Notes:
All value-added models rely on the assumption that teacher effectiveness
can be estimated reliably and validly through student achievement tests.
In practice, the reliability of
student achievement growth is lower than that of the individual tests
themselves. (jc: e.g., 5th grade CST test is different from the 4th grade one, so measuring the gain score can be tricky)
Additional variation in teacher estimates arises from the nature of testing.
Students take tests on different days and at different times of the year.
Because students, particularly those in urban schools, have relatively high
absenteeism and mobility, the students present to take each test may vary
substantially. Thus, teacher value-added estimates may vary across outcomes
in part because different samples of students take each test.
As seen in Table 5 (on p. 180), approximately half of the teachers who would earn
a $7,000 bonus using the state test would lose money if the district used the
SRI instead.
The average teacher in the district would see his or her pay changed by
$2,178 simply by switching outcome measures. Interestingly, the instability
in teacher estimates across outcome measures is much greater for teachers
in the middle two quartiles. (p181)
Papay found that differences in test content and scaling do not
appear to explain the variation in teacher effects across outcomes in this district.
The different samples of students who take each of the tests contribute
somewhat, but they do not account for most of the differences. Test timing
appears to play a greater role in producing these differences. Nonetheless, it
does not explain all of the variation, suggesting that measurement error also
contributes to the instability in teacher rankings. (p183)
Papay made comparisons that suggest
that summer learning loss (or gain) may produce important differences in
teacher effects. Here, the fall-to-fall estimates attribute one summer’s learning
loss to the teacher, while the spring-to-spring estimates attribute a different
summer’s loss. Thus, the fact that the fall-to-fall and spring-to-spring
estimates produce substantially different answers likely reflects, in part,
the inclusion of a different summer in each estimate. (p187)
Tuesday, January 25, 2011
Tuesday, January 18, 2011
Integrating inquiry science and language development for English language learners - Stoddart (2002)
Stoddart, T., Pinal, A., Latzke, M., and Canaday, D. (2002). Integrating inquiry science and language development for English language learners. Journal of Research in Science Teaching, 39(8):664-687.
Context:
The context for this study is Language Acquisition through Science Education in Rural Schools (LASERS), a National Science Foundation–funded Local Systemic Change project in central California that prepares experienced teachers to provide inquiry science instruction to Latino students learning English as a second language. The science–language integration rubric was developed to provide a conceptual framework for teacher staff development activities and to gauge changes in teachers’ beliefs and practice.
Research questions:
(a) How do teachers conceive of science language integration? and (b) What are the cognitive demands that underlie the development of teacher expertise in domain integration?
Methods:
Interviews were conducted with 24 first- through sixth-grade teachers (21 female, 3 male) who participated in the LASERS summer school academy in 1998. The majority of the 24 teachers (19 of 24) had more than 3 years of teaching experience. The sample includes teachers with differing levels of participation in the LASERS project and a range of teaching experience. Therefore, they represent a range of perspectives on language-science integration.
The literature on curriculum domain integration, the development of expertise in teaching, and cognitive complexity are used as a framework for a rubric that describes science– language integration as a continuum from isolated domain-specific instruction to fully-integrated synergistic instruction with the emphasis on commonalties in structure and process across domains.
The view of integration presented in this article is based on Huntley’s definition of synergistic integration. Effective language instruction enhances the learning of science concepts, and effective science inquiry instruction enhances language development and promotes the development of higher-order thinking skills. This approach aligns with work on the integration of reading and writing with science instruction
In viewing the teaching of science and language as a synergistic process, we support the view of bilingual educators such as Cummins (1994) and Met (1994), who argue that the teaching of English and subject matter content should be so integrated that "all content teachers are also teachers of language" (Cummins, 1994, p. 42) and "view every content lesson as a language lesson" (Met, 1994, p. 161). There is currently little information available, however, on successful approaches to preparing teachers to teach inquiry science to second language learners (Lee & Fradd, 1998).
This evolution of teacher understanding could be characterized as a shift from "knowing that" to "knowing how" (Dreyfus & Dreyfus, 1986; Kuhn, 1970; Polanyi, 1958). "Knowing that" understanding is characterized by a rule-governed, theoretical orientation, whereas "knowing how" is the flexible application of principles in practice.
Dreyfus framework: novice, advanced beginner, competent, proficient, expert
Conclusions
The traditional approach to educating English language learners, which separates the teaching of language from the teaching of science content, presents an unnecessary obstacle to the academic progress of language minority students.
The findings of this report suggest the need to rethink staff development activities and science teacher education. The artificial and rigid distinctions between the role of science teacher and language teacher must be broken down.
The critical point is that language processes can be used to promote understanding of content across all subject matter domains, and that language use should be contextualized in authentic and concrete activity. In states such as California, where language minority students represent a significant percentage of the school-age population, methods of English language development should be integrated into all elementary and secondary subject matter methods classes and staff development programs. Integrated instruction will assist language minority students in mastering the English language and simultaneously improve their achievement in academic subjects.
See:
Huntley, M.A. (1998). Design and implementation of a framework for defining integrated mathematics and science education. School Science and Mathematics, 98, 320–327.
Context:
The context for this study is Language Acquisition through Science Education in Rural Schools (LASERS), a National Science Foundation–funded Local Systemic Change project in central California that prepares experienced teachers to provide inquiry science instruction to Latino students learning English as a second language. The science–language integration rubric was developed to provide a conceptual framework for teacher staff development activities and to gauge changes in teachers’ beliefs and practice.
Research questions:
(a) How do teachers conceive of science language integration? and (b) What are the cognitive demands that underlie the development of teacher expertise in domain integration?
Methods:
Interviews were conducted with 24 first- through sixth-grade teachers (21 female, 3 male) who participated in the LASERS summer school academy in 1998. The majority of the 24 teachers (19 of 24) had more than 3 years of teaching experience. The sample includes teachers with differing levels of participation in the LASERS project and a range of teaching experience. Therefore, they represent a range of perspectives on language-science integration.
The literature on curriculum domain integration, the development of expertise in teaching, and cognitive complexity are used as a framework for a rubric that describes science– language integration as a continuum from isolated domain-specific instruction to fully-integrated synergistic instruction with the emphasis on commonalties in structure and process across domains.
The view of integration presented in this article is based on Huntley’s definition of synergistic integration. Effective language instruction enhances the learning of science concepts, and effective science inquiry instruction enhances language development and promotes the development of higher-order thinking skills. This approach aligns with work on the integration of reading and writing with science instruction
In viewing the teaching of science and language as a synergistic process, we support the view of bilingual educators such as Cummins (1994) and Met (1994), who argue that the teaching of English and subject matter content should be so integrated that "all content teachers are also teachers of language" (Cummins, 1994, p. 42) and "view every content lesson as a language lesson" (Met, 1994, p. 161). There is currently little information available, however, on successful approaches to preparing teachers to teach inquiry science to second language learners (Lee & Fradd, 1998).
This evolution of teacher understanding could be characterized as a shift from "knowing that" to "knowing how" (Dreyfus & Dreyfus, 1986; Kuhn, 1970; Polanyi, 1958). "Knowing that" understanding is characterized by a rule-governed, theoretical orientation, whereas "knowing how" is the flexible application of principles in practice.
Dreyfus framework: novice, advanced beginner, competent, proficient, expert
Conclusions
The traditional approach to educating English language learners, which separates the teaching of language from the teaching of science content, presents an unnecessary obstacle to the academic progress of language minority students.
The findings of this report suggest the need to rethink staff development activities and science teacher education. The artificial and rigid distinctions between the role of science teacher and language teacher must be broken down.
The critical point is that language processes can be used to promote understanding of content across all subject matter domains, and that language use should be contextualized in authentic and concrete activity. In states such as California, where language minority students represent a significant percentage of the school-age population, methods of English language development should be integrated into all elementary and secondary subject matter methods classes and staff development programs. Integrated instruction will assist language minority students in mastering the English language and simultaneously improve their achievement in academic subjects.
See:
Huntley, M.A. (1998). Design and implementation of a framework for defining integrated mathematics and science education. School Science and Mathematics, 98, 320–327.
Conceptual change pedagogy - Stofflett and Stoddart (1994)
Stofflett, R. T. and Stoddart, T. (1994). The ability to understand and use conceptual change pedagogy as a function of prior content learning experience. Journal of Research in Science Teaching, 31(1):31-51.
The research questions were
1 . Were there any differences in the degree of science content understandings held by teacher candidates in the traditional and conceptual change groups following science content instruction?
2. What effects did the treatments have on the teacher candidates’ understandings of science pedagogy?
3. What effects did the treatments have on the teacher candidates’ ability to use conceptual change pedagogy in their instructional practice?
Participants:
27 college seniors enrolled in a university elementary teacher education program located in the western United States. The subjects were randomly placed by the university certification admissions committee into two sections of a 10-week science methods course. There were 17 conceptual change (14F, 3M) and 10 (8F, 2M) traditional subjects, All subjects had previously completed the two science content courses required for their certificate. The most frequently taken courses were "Common Medicines ," "Trees and Shrubs ," and "Energy Resources."
The research presented in this article is based on three hypotheses.
The conceptual change instruction was developed around the four conditions necessary for accommodation of a scientific conception as described by Posner et al. (1982). These conditions are intelligibility (ability to understand the concept), plausibility (believability and consistency of the concept), dissatisfaction with existing conceptions, and fruitfulness of the concept for use in external contexts.
Five step strategy
The research questions were
1 . Were there any differences in the degree of science content understandings held by teacher candidates in the traditional and conceptual change groups following science content instruction?
2. What effects did the treatments have on the teacher candidates’ understandings of science pedagogy?
3. What effects did the treatments have on the teacher candidates’ ability to use conceptual change pedagogy in their instructional practice?
Participants:
27 college seniors enrolled in a university elementary teacher education program located in the western United States. The subjects were randomly placed by the university certification admissions committee into two sections of a 10-week science methods course. There were 17 conceptual change (14F, 3M) and 10 (8F, 2M) traditional subjects, All subjects had previously completed the two science content courses required for their certificate. The most frequently taken courses were "Common Medicines ," "Trees and Shrubs ," and "Energy Resources."
The research presented in this article is based on three hypotheses.
- First, the pedagogy through which teachers learn science content is a primary determinant of how they understand and teach science.
- Second, in order for most elementary teacher candidates to develop conceptual understanding of science content they need to reconstruct their subject-matter knowledge through a process of conceptual change.
- Third, the experience of learning science content themselves through the conceptual change process will facilitate the understanding and application of conceptual change pedagogy in science instruction.
The conceptual change instruction was developed around the four conditions necessary for accommodation of a scientific conception as described by Posner et al. (1982). These conditions are intelligibility (ability to understand the concept), plausibility (believability and consistency of the concept), dissatisfaction with existing conceptions, and fruitfulness of the concept for use in external contexts.
Five step strategy
- The first step involved the diagnosis of misconceptions.
- The second step involved exploring the phenomena in question using guided discovery methods.
- The third step consisted of a discussion of the results of the experiments.
- The fourth step was used to facilitate the development of dissatisfaction with the preexisting conceptions.
- When students were able to distinguish between scientifically accepted ideas and naive theories, the instruction moved to the final step, where teacher candidates were given the opportunity to develop fruitfulness by applying the new concepts to real-world examples. The instructor asked the students to provide examples of the phenomena occurring in their own lives and to explain the concept in context. For each concept explored, the five-step model was used.
- The teacher candidates in the conceptual change group had significantly higher gains in their content understandings than did those in the traditional group.
- This finding supports previous research (e.g., Champagne et al., 1985; McClosky, 1983) that traditional science instruction, even that with a heavy emphasis on laboratory work, does not significantly improve students’ conceptual understandings. Challenging students’ countertheories must be an integral part of science instruction, if students’ conceptual knowledge is to improve.
- This finding was also consistent with the idea that teachers teach as they were taught (Stoddart & Stofflett, 1992).
- The subjects in the conceptual change group were better able to translate their cognition of conceptual change pedagogy into practice than were the traditional subjects.
- The findings of this study indicate that this round of reform will fail, as previous rounds have, unless the recommendations on teaching and learning are applied to teachers as well as students.
- The data presented in this study indicate that many teacher candidates hold similar misconceptions and learn in the same way as students do: To improve their understanding they also need to be taught conceptually.
- The pedagogy used in college science content courses will need to incorporate the new views of teaching and learning if teacher knowledge is to be improved.
- Showing teachers how to use innovative curriculum and instructional materials and modeling innovative practice will not be sufficient to bring about changes in their science teaching.
- Teachers must experience the innovative pedagogy first as learners before they can develop intelligibility of the methods being taught.
- The findings of this study draw attention to a fundamental dilemma in science education reform: the expectation that teachers can learn to be constructivist teachers when they have not been constructivist learners.
- Educational reformers, however, typically expect teachers to change their pedagogical conceptions by being shown and told about innovative practice (Shulman, 1986; Stoddart, 1993)
- the authors propose that the use of conceptual change pedagogy in teacher preparation is the one that is most likely to bring about change in teacher candidates’ conceptions of teaching and learning precisely because it involves the challenging of preconceptions and reconstruction of knowledge structures.
- These findings are a powerful example of constructivist theory in practice: They demonstrate the importance of the personal construction of knowledge over teaching as showing and telling.
Monday, December 20, 2010
Teacher Science Knowledge and Student Science Achievement - meta-analysis
Becker, B.J. & Aloe, A.M. (2008). Teacher Science Knowledge and Student Science Achievement. Paper presented at the annual meeting of the American Educational Research Association, New York, March 2008
In this research the authors ask whether teachers’ knowledge of science is an important predictor of student science achievement. the authors use meta-analysis to examine a set of studies done in the United States since 1960, and find that increased content knowledge has a positive and significant, but small, bivariate relationship with student achievement. However, effects from more complex analyses are essentially nil.
The authors address these questions:
1) Are teachers’ levels of content knowledge in science related to their students’ achievement?
2) Do differences in school level, area of science achievement, and how teacher knowledge is measured relate to the strength of relationship found?
3) For more complex studies, does whether prior student achievement is controlled affect the strength of relationship between teachers’ science content knowledge and student achievement in science?
The authors analyses have revealed, first, that the amount of evidence on the relationship of science teacher knowledge to student outcomes in science is not extensive, and is not of very high quality. Fewer than 30 studies report results pertinent to this topic. The bulk of those studies have measured teacher knowledge using variables that appear to be, at best, proxy variables for actual levels of knowledge. If these measures indeed have low validity as indices of content knowledge, the correlations that the authors have observed will be lower than the true correlation values (see, e.g., Hunter & Schmidt, 2004, for a discussion of invalidity as an artifact in meta-analysis).
The correlational studies reveal that, on average, teacher knowledge in science has a slight positive correlation with student science achievement. However, there is variation that is not accounted for by any of the explanatory variables examined in the authors analyses. More notably, when other variables are held constant (in the five studies reporting regressions), the relationship disappears.
These results are a bit oversimplified, however. The authors analyses revealed that a variety of factors relate to the size of the bivariate correlations between measures of teacher content knowledge and student science achievement. Of most interest is that when the authors examine studies where the measures of teacher knowledge and student achievement focus on the same content, two content areas show larger correlations: .32 on average for biology and .18 for physical sciences. Though neither of these values is large, they are larger than the means overall and for any other subsets of effects.
Also worth noting are the conflicting results for numbers of credits in science and counts of courses taken. While the mean r for credit hours in science was a significant .15, the mean for course counts was significantly negative, if trivial, at -.03. These two proxy-like predictors were studied in different kinds of samples, which may have affected the values as well.
Finally there is the issue of the national samples. In the authors analyses of correlations, the results of the national studies, all of which were based on large multi-stage probability samples, showed essentially no relationship of teacher knowledge to student achievement. The same was true for the regression studies, all of which similarly drew on national probability samples. These studies might be considered to provide the authors “best evidence” on the issue at hand, since they allow for inferences to be made to a well defined population. However, some caution is in order due to the nature of the measures used in these studies. All of these studies used broad measures of student science achievement, not measures of specific science content. Similarly all used coarse self-report proxies to represent teachers’ science knowledge, many of which were course counts and indicators of whether a teacher had a major or an advanced degree in science. These may have a less direct relation to the construct of interest than more targeted measure of teacher knowledge, thus attenuating the observed relationship. Last, two of the national surveys (LSAY and NELS) appear both data sets, and appear twice in the set of regression results, because several authors have analyzed these important data sources. Thus their findings, which appear to be weaker than those of local samples, play a large role in the authors conclusions.
Clearly, a multitude of factors aside from teacher subject-matter knowledge have been documented to impact student science achievement. Such things as students’ verbal, spatial and reasoning skills (Piburn, 1993), a diversity of teaching strategies (Bowen, 2000; Johnson, Kahle, & Fargo, 2007; Schroeder et al., 2007) and curricular interventions (Shymansky, Kyle, & Alport, 1983) have been found to impact achievement. Piburn (1993) reported correlations ranging from about .20 to .45 for ability predictors with school science outcomes, larger than most of the values the authors report. Bowen (2000) examined a set of studies of cooperative learning activities in high-school and college chemistry courses, and found effects that would average about .18 on the correlation scale. Other influences have been found to have even more sizeable impacts on achievement. For instance, Schroeder and colleagues (2007) examined 61 experiments or quasi-experiments on a wide range of science teaching strategies. They found effects ranging from .14 to roughly .60 on the correlation scale.
We can also compare the authors results to those found in a recent meta-analysis of the importance of subject-matter knowledge in mathematics. Choi, Ahn and Kennedy (2007) examined results from 16 studies of student math achievement. They found that teachers’ arithmetic knowledge correlated on average only .07 with student arithmetic performance, while results were mixed for algebra achievement. Performance on algebra concept tests showed a correlation of .12 with teacher knowledge, whereas computation in algebra was unrelated to teacher knowledge. As was true for the authors' analyses the correlation of teacher knowledge to student math outcomes varied according to a variety of features of the studies and measures used. However, none of the mean correlations reported by Choi and colleagues averaged above .2, and as was true for the authors results, some mean correlations were significantly negative.
Re: claims that have been made about the importance of subject matter knowledge - In 2002 the report of the Secretary of the U.S. of Education asserted “Rigorous research indicates that verbal ability and content knowledge are the most important attributes of highly qualified teachers” (2002, p. 19, emphasis added). the authors would argue that teacher knowledge is only one factor among many other more important ones leading to increased science achievement for students. It may be that with more targeted, higher-quality measures of teacher content knowledge, a stronger relationship would be found. This leads to one clear suggestion for future work: Use better, more specific measures of teachers’ science knowledge. However, the existing literature suggests that the relation of teachers’ content knowledge to science achievement is weak, and provides a very poor basis for claiming that science subject-matter knowledge is among the “most important” attributes of highly qualified teachers.
In this research the authors ask whether teachers’ knowledge of science is an important predictor of student science achievement. the authors use meta-analysis to examine a set of studies done in the United States since 1960, and find that increased content knowledge has a positive and significant, but small, bivariate relationship with student achievement. However, effects from more complex analyses are essentially nil.
The authors address these questions:
1) Are teachers’ levels of content knowledge in science related to their students’ achievement?
2) Do differences in school level, area of science achievement, and how teacher knowledge is measured relate to the strength of relationship found?
3) For more complex studies, does whether prior student achievement is controlled affect the strength of relationship between teachers’ science content knowledge and student achievement in science?
The authors analyses have revealed, first, that the amount of evidence on the relationship of science teacher knowledge to student outcomes in science is not extensive, and is not of very high quality. Fewer than 30 studies report results pertinent to this topic. The bulk of those studies have measured teacher knowledge using variables that appear to be, at best, proxy variables for actual levels of knowledge. If these measures indeed have low validity as indices of content knowledge, the correlations that the authors have observed will be lower than the true correlation values (see, e.g., Hunter & Schmidt, 2004, for a discussion of invalidity as an artifact in meta-analysis).
The correlational studies reveal that, on average, teacher knowledge in science has a slight positive correlation with student science achievement. However, there is variation that is not accounted for by any of the explanatory variables examined in the authors analyses. More notably, when other variables are held constant (in the five studies reporting regressions), the relationship disappears.
These results are a bit oversimplified, however. The authors analyses revealed that a variety of factors relate to the size of the bivariate correlations between measures of teacher content knowledge and student science achievement. Of most interest is that when the authors examine studies where the measures of teacher knowledge and student achievement focus on the same content, two content areas show larger correlations: .32 on average for biology and .18 for physical sciences. Though neither of these values is large, they are larger than the means overall and for any other subsets of effects.
Also worth noting are the conflicting results for numbers of credits in science and counts of courses taken. While the mean r for credit hours in science was a significant .15, the mean for course counts was significantly negative, if trivial, at -.03. These two proxy-like predictors were studied in different kinds of samples, which may have affected the values as well.
Finally there is the issue of the national samples. In the authors analyses of correlations, the results of the national studies, all of which were based on large multi-stage probability samples, showed essentially no relationship of teacher knowledge to student achievement. The same was true for the regression studies, all of which similarly drew on national probability samples. These studies might be considered to provide the authors “best evidence” on the issue at hand, since they allow for inferences to be made to a well defined population. However, some caution is in order due to the nature of the measures used in these studies. All of these studies used broad measures of student science achievement, not measures of specific science content. Similarly all used coarse self-report proxies to represent teachers’ science knowledge, many of which were course counts and indicators of whether a teacher had a major or an advanced degree in science. These may have a less direct relation to the construct of interest than more targeted measure of teacher knowledge, thus attenuating the observed relationship. Last, two of the national surveys (LSAY and NELS) appear both data sets, and appear twice in the set of regression results, because several authors have analyzed these important data sources. Thus their findings, which appear to be weaker than those of local samples, play a large role in the authors conclusions.
Clearly, a multitude of factors aside from teacher subject-matter knowledge have been documented to impact student science achievement. Such things as students’ verbal, spatial and reasoning skills (Piburn, 1993), a diversity of teaching strategies (Bowen, 2000; Johnson, Kahle, & Fargo, 2007; Schroeder et al., 2007) and curricular interventions (Shymansky, Kyle, & Alport, 1983) have been found to impact achievement. Piburn (1993) reported correlations ranging from about .20 to .45 for ability predictors with school science outcomes, larger than most of the values the authors report. Bowen (2000) examined a set of studies of cooperative learning activities in high-school and college chemistry courses, and found effects that would average about .18 on the correlation scale. Other influences have been found to have even more sizeable impacts on achievement. For instance, Schroeder and colleagues (2007) examined 61 experiments or quasi-experiments on a wide range of science teaching strategies. They found effects ranging from .14 to roughly .60 on the correlation scale.
We can also compare the authors results to those found in a recent meta-analysis of the importance of subject-matter knowledge in mathematics. Choi, Ahn and Kennedy (2007) examined results from 16 studies of student math achievement. They found that teachers’ arithmetic knowledge correlated on average only .07 with student arithmetic performance, while results were mixed for algebra achievement. Performance on algebra concept tests showed a correlation of .12 with teacher knowledge, whereas computation in algebra was unrelated to teacher knowledge. As was true for the authors' analyses the correlation of teacher knowledge to student math outcomes varied according to a variety of features of the studies and measures used. However, none of the mean correlations reported by Choi and colleagues averaged above .2, and as was true for the authors results, some mean correlations were significantly negative.
Re: claims that have been made about the importance of subject matter knowledge - In 2002 the report of the Secretary of the U.S. of Education asserted “Rigorous research indicates that verbal ability and content knowledge are the most important attributes of highly qualified teachers” (2002, p. 19, emphasis added). the authors would argue that teacher knowledge is only one factor among many other more important ones leading to increased science achievement for students. It may be that with more targeted, higher-quality measures of teacher content knowledge, a stronger relationship would be found. This leads to one clear suggestion for future work: Use better, more specific measures of teachers’ science knowledge. However, the existing literature suggests that the relation of teachers’ content knowledge to science achievement is weak, and provides a very poor basis for claiming that science subject-matter knowledge is among the “most important” attributes of highly qualified teachers.
Friday, December 17, 2010
Effective Programs in Middle and High School Mathematics: A Best-Evidence Synthesis
Slavin, R. E., Lake, C., and Groff, C. (2009).
Effective Programs in Middle and High School Mathematics: A
Best-Evidence Synthesis.
Review of Educational Research, 79(2):839-911. (abstract)
This review examined 100 studies of three types of programs designed to improve achievement in mathematics (Slavin, Lake, & Groff, 2009). In this review, 40 studies of mathematics curricula found very small effects (ES = +0.03); 38 studies of computer-assisted instruction found small effects (ES = +0.10); and 22 studies of instructional process programs found small effects (ES = +0.18); although the effects of specific programs varied widely, with studies of two forms of cooperative learning having medium effects (ES = +0.48).
An earlier review examined 33 studies of four types of programs designed to improve achievement in reading (Slavin, Cheung, Groff, & Lake, 2008); Regarding these programs, no studies of secondary reading curricula met the criteria to be included in the review; eight studies of computer-assisted instruction found small effects (ES = +0.10); 16 studies of instructional-process programs had small effects (ES = +0.21); and nine studies of two mixed-method models that combined large-group, small-group, and computer-assisted, individualized instruction had small effects (ES = +0.23). The third review was conducted by the What Works Clearinghouse based on three studies of a computer-based adolescent literacy program that supplements regular classroom reading instruction in grades K-8. The review found that the program had small effects on reading comprehension (ES = .27) and literacy achievement (ES = .28).
Abstract
This article reviews research on the achievement outcomes of mathematics programs for middle and high schools. Study inclusion requirements include use of a randomized or matched control group, a study duration of at least 12 weeks, and equality at pretest. There were 100 qualifying studies, 26 of which used random assignment to treatments. Effect sizes were very small for mathematics curricula and for computer-assisted instruction. Positive effects were found for two cooperative learning programs. Outcomes were similar for disadvantaged and nondisadvantaged students and for students of different ethnicities. Consistent with an earlier review of elementary programs, this article concludes that programs that affect daily teaching practices and student interactions have more promise than those emphasizing textbooks or technology alone.
This review examined 100 studies of three types of programs designed to improve achievement in mathematics (Slavin, Lake, & Groff, 2009). In this review, 40 studies of mathematics curricula found very small effects (ES = +0.03); 38 studies of computer-assisted instruction found small effects (ES = +0.10); and 22 studies of instructional process programs found small effects (ES = +0.18); although the effects of specific programs varied widely, with studies of two forms of cooperative learning having medium effects (ES = +0.48).
An earlier review examined 33 studies of four types of programs designed to improve achievement in reading (Slavin, Cheung, Groff, & Lake, 2008); Regarding these programs, no studies of secondary reading curricula met the criteria to be included in the review; eight studies of computer-assisted instruction found small effects (ES = +0.10); 16 studies of instructional-process programs had small effects (ES = +0.21); and nine studies of two mixed-method models that combined large-group, small-group, and computer-assisted, individualized instruction had small effects (ES = +0.23). The third review was conducted by the What Works Clearinghouse based on three studies of a computer-based adolescent literacy program that supplements regular classroom reading instruction in grades K-8. The review found that the program had small effects on reading comprehension (ES = .27) and literacy achievement (ES = .28).
Abstract
This article reviews research on the achievement outcomes of mathematics programs for middle and high schools. Study inclusion requirements include use of a randomized or matched control group, a study duration of at least 12 weeks, and equality at pretest. There were 100 qualifying studies, 26 of which used random assignment to treatments. Effect sizes were very small for mathematics curricula and for computer-assisted instruction. Positive effects were found for two cooperative learning programs. Outcomes were similar for disadvantaged and nondisadvantaged students and for students of different ethnicities. Consistent with an earlier review of elementary programs, this article concludes that programs that affect daily teaching practices and student interactions have more promise than those emphasizing textbooks or technology alone.
Improving High School Performance
Rumberger, R. What the Federal Government Can Do to Improve High School
Performance. Washington, D.C.: Center on Education Policy. 2009.
[Technical Report]
Improvement strategies, especially more comprehensive ones, will not be successful until critical aspects of capacity and context are improved (p83)
The research literature has identified two broad factors that affect implementation: will and capacity (McLaughlin, 1987; McLaughlin, 1990). Will and capacity refer to traits of both individuals and institutions. At the individual level, will refers to the motivation and commitment of educators—teachers and administrators—to implement reform strategies. (p75)
The individual capacity of teachers and administrators to carry out reforms is clearly important. The capacity of teachers to implement reforms, which, again, usually means changing their instructional practices, is a time-consuming, multi-stage process that includes persuasion over the need for reform. (p75)
The capacity of individuals—teachers and administrators—as well as the institutional capacity of the school itself are key factors to successful implementation. School capacity depends on having sufficient and correct alignment of resources (including sufficient time); it also depends on coherence in its efforts across all the demands placed on schools and their staffs by districts, as well as state and federal policy requirements. Building capacity also depends on having sufficient will or readiness, especially among school and district leadership, to build capacity and initiate reform. (p83)
Summary:
High schools play a crucial role in preparing students for college, work, and citizenship. Yet, by many accounts, U.S. high schools are not performing any of these tasks well. This situation has prompted calls for improving high school performance. This report reviews past efforts to reform high schools, examines why those efforts have largely been unsuccessful, and suggests what the federal government can do to improve high school performance.
In order to improve the performance of U.S. high schools, it is first necessary to identify the purposes and goals of high schools and then develop suitable measures of school performance to determine the extent to which those goals are met. Only then can any serious effort be made to improve high school performance. In the current era of standards-based accountability, reform efforts have focused on raising student academic performance as measured by course credits, test scores, and educational credentials. Yet research studies and surveys of employers suggest students need a wide variety of non-academic as well as academic skills to be successful in college, the workplace, and in their adult lives.
A number of approaches have been developed for improving high schools, including targeted approaches that focus on specific facets of the school (instruction, student support, school restructuring); comprehensive strategies that redesign all aspects of the school or create new schools; collaborative approaches that create partnerships between schools and outside agencies; and systemic approaches that alter requirements for all schools in the system. Although the research evidence on the effectiveness of specific approaches is limited, it does suggest that no one strategy is inherently more effective than the others.
Numerous large-scale initiatives to improve the performance of high schools in the U.S. have been undertaken in the past 20 years by government agencies, foundations, non-profit organizations, and independent developers. For the most part these efforts have been unsuccessful, although there was widespread variability in both the implementation and impact of the initiatives across schools, districts, and states. Evaluations of these efforts have identified a number of factors that limited their implementation and impact, with the most important being the lack of will and capacity of both individual educators and institutions to engage in sustained improvement efforts. One implication is that strategies for improving high schools will not be successful until critical aspects of capacity and context are improved.
The federal government can play an important role in improving U.S. high schools by shifting its focus from short-term accountability to long-term capacity building. Specifically, the federal government should:
1. Support the development of broader indicators of student progress and outcomes, and include these indicators in the National Assessment of Educational Progress.
2. Help build the capacity of state governments and technical-assistance providers to support improvement efforts and capacity building in districts and schools.
3. Develop guidelines to insure that states do a better job of matching reform strategies to the capacity of schools and districts in need of improvement.
4. Improve coherence among federal policy initiatives, between federal and state initiatives, and between government and foundation initiatives.
5. Support the development of more comprehensive state and local data systems that not only measure educational inputs and outputs, but also district and school readiness and capacity to initiate reform as well as progress toward improving student outcomes.
Improvement strategies, especially more comprehensive ones, will not be successful until critical aspects of capacity and context are improved (p83)
The research literature has identified two broad factors that affect implementation: will and capacity (McLaughlin, 1987; McLaughlin, 1990). Will and capacity refer to traits of both individuals and institutions. At the individual level, will refers to the motivation and commitment of educators—teachers and administrators—to implement reform strategies. (p75)
The individual capacity of teachers and administrators to carry out reforms is clearly important. The capacity of teachers to implement reforms, which, again, usually means changing their instructional practices, is a time-consuming, multi-stage process that includes persuasion over the need for reform. (p75)
The capacity of individuals—teachers and administrators—as well as the institutional capacity of the school itself are key factors to successful implementation. School capacity depends on having sufficient and correct alignment of resources (including sufficient time); it also depends on coherence in its efforts across all the demands placed on schools and their staffs by districts, as well as state and federal policy requirements. Building capacity also depends on having sufficient will or readiness, especially among school and district leadership, to build capacity and initiate reform. (p83)
Summary:
High schools play a crucial role in preparing students for college, work, and citizenship. Yet, by many accounts, U.S. high schools are not performing any of these tasks well. This situation has prompted calls for improving high school performance. This report reviews past efforts to reform high schools, examines why those efforts have largely been unsuccessful, and suggests what the federal government can do to improve high school performance.
In order to improve the performance of U.S. high schools, it is first necessary to identify the purposes and goals of high schools and then develop suitable measures of school performance to determine the extent to which those goals are met. Only then can any serious effort be made to improve high school performance. In the current era of standards-based accountability, reform efforts have focused on raising student academic performance as measured by course credits, test scores, and educational credentials. Yet research studies and surveys of employers suggest students need a wide variety of non-academic as well as academic skills to be successful in college, the workplace, and in their adult lives.
A number of approaches have been developed for improving high schools, including targeted approaches that focus on specific facets of the school (instruction, student support, school restructuring); comprehensive strategies that redesign all aspects of the school or create new schools; collaborative approaches that create partnerships between schools and outside agencies; and systemic approaches that alter requirements for all schools in the system. Although the research evidence on the effectiveness of specific approaches is limited, it does suggest that no one strategy is inherently more effective than the others.
Numerous large-scale initiatives to improve the performance of high schools in the U.S. have been undertaken in the past 20 years by government agencies, foundations, non-profit organizations, and independent developers. For the most part these efforts have been unsuccessful, although there was widespread variability in both the implementation and impact of the initiatives across schools, districts, and states. Evaluations of these efforts have identified a number of factors that limited their implementation and impact, with the most important being the lack of will and capacity of both individual educators and institutions to engage in sustained improvement efforts. One implication is that strategies for improving high schools will not be successful until critical aspects of capacity and context are improved.
The federal government can play an important role in improving U.S. high schools by shifting its focus from short-term accountability to long-term capacity building. Specifically, the federal government should:
1. Support the development of broader indicators of student progress and outcomes, and include these indicators in the National Assessment of Educational Progress.
2. Help build the capacity of state governments and technical-assistance providers to support improvement efforts and capacity building in districts and schools.
3. Develop guidelines to insure that states do a better job of matching reform strategies to the capacity of schools and districts in need of improvement.
4. Improve coherence among federal policy initiatives, between federal and state initiatives, and between government and foundation initiatives.
5. Support the development of more comprehensive state and local data systems that not only measure educational inputs and outputs, but also district and school readiness and capacity to initiate reform as well as progress toward improving student outcomes.
Friday, December 3, 2010
Challenges New Science Teachers Face - Lit Review (2006)
Davis E.A., Petish, D. & Smithey, J. (2006). Challenges new science teachers face.
Review of Educational Research, 76, 607–651
What are the challenges that new science teachers face in trying to meet the increasingly high expectations laid out for them in current reform documents? What do we expect new science teachers to know and be able to do? [p609] Science teachers are expected to understand: (1) the content and disciplines of science, (2) learners, (3) instruction, (4) learning environments, and (5) professionalism. [p607] New elementary teachers may face even greater challenges in teaching science than do their secondary counterparts, since they typically teach multiple subjects, including all areas of science.[p608]
The authors see standards [INTASC, 2002 & NSES] for teaching as appropriate to use as a frame for their work: They concisely represent the kinds of things that teachers should probably be able to do—and indeed, that teacher educators should help them achieve—and are the result of some form of consensus-building at a higher level than a single scholar’s viewpoint.
1. The first theme, understanding the content and disciplines of science, focuses on the teacher’s understanding of “the major concepts, assumptions, debates, processes of inquiry, and ways of knowing that are central” to the science discipline(s) she teaches (INTASC, 1992, p. 14). (p613)
3. The third theme, understanding instruction, means that the teacher “understands principles and techniques, along with advantages and limitations, associated with various instructional strategies” (INTASC, 1992, p. 20) and “uses a variety of instructional strategies” (INTASC, 2002, p. 4). [p621]
Supportive Science Coursework
p624 "In sum, new teachers face many challenges with regard to using effective instructional approaches, including lacking relevant subject matter knowledge, not knowing how to enact their instructional ideas, and being resistant to certain innovative practices. With support, though, teachers can begin to move along a positive trajectory."
p626 "One study provides a counterpoint to this general trend, though it focuses on only a single teacher (Abell & Roth, 1994). This preservice elementary teacher developed effective coping strategies while taking her science methods course. She infused additional science into an otherwise limited science curriculum and inspired the other teachers in her school to use cooperative learning experiences. Her personal attributes and the features of her student teaching context helped her to take risks in her environment."
Abell, S. K., & Roth, M. (1994). Constructing science teaching in the elementary school: The socialization of a science enthusiast student teacher. Journal of Research in Science Teaching, 31(1), 77–90.
What are the challenges that new science teachers face in trying to meet the increasingly high expectations laid out for them in current reform documents? What do we expect new science teachers to know and be able to do? [p609] Science teachers are expected to understand: (1) the content and disciplines of science, (2) learners, (3) instruction, (4) learning environments, and (5) professionalism. [p607] New elementary teachers may face even greater challenges in teaching science than do their secondary counterparts, since they typically teach multiple subjects, including all areas of science.[p608]
The authors see standards [INTASC, 2002 & NSES] for teaching as appropriate to use as a frame for their work: They concisely represent the kinds of things that teachers should probably be able to do—and indeed, that teacher educators should help them achieve—and are the result of some form of consensus-building at a higher level than a single scholar’s viewpoint.
1. The first theme, understanding the content and disciplines of science, focuses on the teacher’s understanding of “the major concepts, assumptions, debates, processes of inquiry, and ways of knowing that are central” to the science discipline(s) she teaches (INTASC, 1992, p. 14). (p613)
- new teachers have relatively weak understandings of science overall (p613)
- In general, the preservice teachers held alternative ideas that were similar to those that have been identified in students (Bendall et al., 1993; Ginns & Watters, 1995; Schoon & Boone, 1998; Trumper, 2003). Even secondary preservice teachers showed poor understandings of topics (Haidar, 1997). (p615)
- preservice secondary science teachers in their studies initially lacked understanding of the connections between concepts in the disciplines they were to teach; but these understandings improved over time and with experience.(p615)
- In sum, preservice teachers seem, for the most part, to lack adequate understandings of science content. This trend is especially pronounced at the elementary level; results are more mixed at the secondary level. Though most studies do not characterize change over time, those that do indicate that the preservice teachers’ knowledge may improve over time. (p615)
- Overall, these papers illustrate that many preservice teachers have unsophisticated understandings of inquiry and related skills, though of course individuals vary. p616
- The studies in this area consistently find that most (though not all) new teachers have naive beliefs about the nature of science (see Lederman, 1992, for a review). p616
- new science teachers’ ideas about learners can become more sophisticated with time and support p618
- in general these teachers struggle with understanding their learners; p618
- their practices with regard to their learners are often naive. p618
- preservice teachers tend not to consider students or student learning very extensively, very carefully, or in very sophisticated ways p618
- The preservice elementary and secondary teachers tended to have very limited ideas about what to do, instructionally, with students’ ideas p619
- Rodriguez, for example, studied 18 preservice secondary teachers, including 4 focus teachers, and found that the preservice teachers tended to feel hopeless and overwhelmed about working with diverse students. p620
- The studies reported within this theme show that, in general, new teachers do not have very clear ideas about what to do with regard to students’ ideas or backgrounds; at least at the elementary level, preservice teachers seem initially to want mainly to engage, interest, motivate, or manage their students. p620
3. The third theme, understanding instruction, means that the teacher “understands principles and techniques, along with advantages and limitations, associated with various instructional strategies” (INTASC, 1992, p. 20) and “uses a variety of instructional strategies” (INTASC, 2002, p. 4). [p621]
- Overall, these studies illustrate a mismatch between teachers’ ideas and practices—their ideas about instruction seem generally to be more sophisticated and innovative than are their actual practices. [p621]
- One basic challenge that new teachers face is developing sophisticated ideas about science instruction; these papers indicate that though improvement can occur, it is neither guaranteed nor necessarily long-lasting.
- In general, these studies — mostly conducted with secondary teachers — indicate that when new teachers have stronger subject matter knowledge, they are more likely to engage in more sophisticated teaching practices. p622
- Overall, then, teachers’ subject matter knowledge seems related to their instructional ideas and practice; stronger science knowledge typically co-occurs with more sophisticated ideas or practices with regard to instruction (though most of the studies related to this point were conducted with secondary teachers). p623
- In sum, new teachers face many challenges with regard to using effective instructional approaches, including lacking relevant subject matter knowledge, not knowing how to enact their instructional ideas, and being resistant to certain innovative practices. With support, though, teachers can begin to move along a positive trajectory.p624
- In sum, based on the few studies we identified in this area, we see that new teachers tend to have concerns about and struggles with management, sometimes leading them to engage in less reform-oriented teaching practices. p628
- Appleton and Kindt (2002) identify three important aspects of knowledge of schools: understanding the priority of science in a school culture, understanding the degree of personal choice one has about the curriculum, and identifying and obtaining resources for science teaching. p629
Supportive Science Coursework
- simply requiring more science content courses is not enough to enable teachers to develop improved understanding of science p633
- Many science teacher educators assume that teachers should engage in reform-oriented practices as learners if they are to learn more inquiry-oriented teaching practices and become more knowledgeable about the science content, scientific inquiry, and the nature of science;
- Science methods courses and teacher education programs can, of course, help to promote improved understanding of instruction p634
- Zembal-Saul and her colleagues (2000) describe the importance of engaging preservice elementary teachers in multiple cycles of planning, teaching, and reflection, over the course of a year. The preservice teachers who participated in the program emphasizing elementary science teaching improved in how they organized instruction around important scientific ideas (a challenge we identified for elementary teachers, who tended instead to focus on activities) and came to recognize the importance of accounting for their learners as they planned instruction p634
- Field experiences also help preservice teachers to overcome certain challenges [p635]
- action research
- collegial relationships
- educative curriculum materials
p624 "In sum, new teachers face many challenges with regard to using effective instructional approaches, including lacking relevant subject matter knowledge, not knowing how to enact their instructional ideas, and being resistant to certain innovative practices. With support, though, teachers can begin to move along a positive trajectory."
p626 "One study provides a counterpoint to this general trend, though it focuses on only a single teacher (Abell & Roth, 1994). This preservice elementary teacher developed effective coping strategies while taking her science methods course. She infused additional science into an otherwise limited science curriculum and inspired the other teachers in her school to use cooperative learning experiences. Her personal attributes and the features of her student teaching context helped her to take risks in her environment."
Abell, S. K., & Roth, M. (1994). Constructing science teaching in the elementary school: The socialization of a science enthusiast student teacher. Journal of Research in Science Teaching, 31(1), 77–90.
Subscribe to:
Posts (Atom)