Teacher Science Knowledge and Student Science Achievement - meta-analysis

Becker, B.J. & Aloe, A.M. (2008). Teacher Science Knowledge and Student Science Achievement. Paper presented at the annual meeting of the American Educational Research Association, New York, March 2008

In this research the authors ask whether teachers’ knowledge of science is an important predictor of student science achievement. the authors use meta-analysis to examine a set of studies done in the United States since 1960, and find that increased content knowledge has a positive and significant, but small, bivariate relationship with student achievement. However, effects from more complex analyses are essentially nil.

The authors address these questions:
1) Are teachers’ levels of content knowledge in science related to their students’ achievement?
2) Do differences in school level, area of science achievement, and how teacher knowledge is measured relate to the strength of relationship found?
3) For more complex studies, does whether prior student achievement is controlled affect the strength of relationship between teachers’ science content knowledge and student achievement in science?

The authors analyses have revealed, first, that the amount of evidence on the relationship of science teacher knowledge to student outcomes in science is not extensive, and is not of very high quality.  Fewer than 30 studies report results pertinent to this topic. The bulk of those studies have measured teacher knowledge using variables that appear to be, at best, proxy variables for actual levels of knowledge. If these measures indeed have low validity as indices of content knowledge, the correlations that the authors have observed will be lower than the true correlation values (see, e.g., Hunter & Schmidt, 2004, for a discussion of invalidity as an artifact in meta-analysis).

The correlational studies reveal that, on average, teacher knowledge in science has a slight positive correlation with student science achievement. However, there is variation that is not accounted for by any of the explanatory variables examined in the authors analyses. More notably, when other variables are held constant (in the five studies reporting regressions), the relationship disappears.

These results are a bit oversimplified, however.  The authors analyses revealed that a variety of factors relate to the size of the bivariate correlations between measures of teacher content knowledge and student science achievement. Of most interest is that when the authors examine studies where the measures of teacher knowledge and student achievement focus on the same content, two content areas show larger correlations: .32 on average for biology and .18 for physical sciences. Though neither of these values is large, they are larger than the means overall and for any other subsets of effects.

Also worth noting are the conflicting results for numbers of credits in science and counts of courses taken. While the mean r for credit hours in science was a significant .15, the mean for course counts was significantly negative, if trivial, at -.03. These two proxy-like predictors were studied in different kinds of samples, which may have affected the values as well.

Finally there is the issue of the national samples. In the authors analyses of correlations, the results of the national studies, all of which were based on large multi-stage probability samples, showed essentially no relationship of teacher knowledge to student achievement. The same was true for the regression studies, all of which similarly drew on national probability samples. These studies might be considered to provide the authors “best evidence” on the issue at hand, since they allow for inferences to be made to a well defined population. However, some caution is in order due to the nature of the measures used in these studies. All of these studies used broad measures of student science achievement, not measures of specific science content. Similarly all used coarse self-report proxies to represent teachers’ science knowledge, many of which were course counts and indicators of whether a teacher had a major or an advanced degree in science. These may have a less direct relation to the construct of interest than more targeted measure of teacher knowledge, thus attenuating the observed relationship. Last, two of the national surveys (LSAY and NELS) appear both data sets, and appear twice in the set of regression results, because several authors have analyzed these important data sources. Thus their findings, which appear to be weaker than those of local samples, play a large role in the authors conclusions.

Clearly, a multitude of factors aside from teacher subject-matter knowledge have been documented to impact student science achievement. Such things as students’ verbal, spatial and reasoning skills (Piburn, 1993), a diversity of teaching strategies (Bowen, 2000; Johnson, Kahle, & Fargo, 2007; Schroeder et al., 2007) and curricular interventions (Shymansky, Kyle, & Alport, 1983) have been found to impact achievement. Piburn  (1993) reported correlations ranging from about .20 to .45 for ability predictors with school science outcomes, larger than most of the values the authors report.  Bowen (2000) examined a set of studies of cooperative learning activities in high-school and college chemistry courses, and found effects that would average about .18 on the correlation scale. Other influences have been found to have even more sizeable impacts on achievement. For instance, Schroeder and colleagues (2007) examined 61 experiments or quasi-experiments on a wide range of science teaching strategies. They found effects ranging from .14 to roughly .60 on the correlation scale.

We can also compare the authors results to those found in a recent meta-analysis of the importance of subject-matter knowledge in mathematics. Choi, Ahn and Kennedy (2007) examined  results from 16 studies of student math achievement. They found that teachers’ arithmetic knowledge correlated on average only .07 with student arithmetic performance, while results were mixed for algebra achievement. Performance on algebra concept tests showed a correlation of .12 with teacher knowledge, whereas computation in algebra was unrelated to teacher knowledge. As was true for the authors' analyses the correlation of teacher knowledge to student math outcomes varied according to a variety of features of the studies and measures used. However, none of the mean correlations reported by Choi and colleagues averaged above .2, and as was true for the authors results, some mean correlations were significantly negative.

Re: claims that have been made about the importance of subject matter knowledge - In 2002 the report of the Secretary of the U.S. of Education asserted “Rigorous research indicates that verbal ability and content knowledge are the most important attributes of highly qualified teachers” (2002, p. 19, emphasis added).   the authors would argue that teacher knowledge is only one factor among many other more important ones leading to increased science achievement for students. It may be that with more targeted, higher-quality measures of teacher content knowledge, a stronger relationship would be found. This leads to one clear suggestion for future work: Use better, more specific measures of teachers’ science knowledge. However, the existing literature suggests that the relation of teachers’ content knowledge to science achievement is weak, and provides a very poor basis for claiming that science subject-matter knowledge is among the “most important” attributes of highly qualified teachers.

Effective Programs in Middle and High School Mathematics: A Best-Evidence Synthesis

Slavin, R. E., Lake, C., and Groff, C. (2009). Effective Programs in Middle and High School Mathematics: A Best-Evidence Synthesis. Review of Educational Research, 79(2):839-911. (abstract)

This review examined 100 studies of three types of programs designed to improve achievement in mathematics (Slavin, Lake, & Groff, 2009). In this review, 40 studies of mathematics curricula found very small effects (ES = +0.03); 38 studies of computer-assisted instruction found small effects (ES = +0.10); and 22 studies of instructional process programs found small effects (ES = +0.18); although the effects of specific programs varied widely, with studies of two forms of cooperative learning having medium effects (ES = +0.48).

An earlier review examined 33 studies of four types of programs designed to improve achievement in reading (Slavin, Cheung, Groff, & Lake, 2008); Regarding these programs, no studies of secondary reading curricula met the criteria to be included in the review; eight studies of computer-assisted instruction found small effects (ES = +0.10); 16 studies of instructional-process programs had small effects (ES = +0.21); and nine studies of two mixed-method models that combined large-group, small-group, and computer-assisted, individualized instruction had small effects (ES = +0.23). The third review was conducted by the What Works Clearinghouse based on three studies of a computer-based adolescent literacy program that supplements regular classroom reading instruction in grades K-8. The review found that the program had small effects on reading comprehension (ES = .27) and literacy achievement (ES = .28).


Abstract
This article reviews research on the achievement outcomes of mathematics programs for middle and high schools. Study inclusion requirements include use of a randomized or matched control group, a study duration of at least 12 weeks, and equality at pretest. There were 100 qualifying studies, 26 of which used random assignment to treatments. Effect sizes were very small for mathematics curricula and for computer-assisted instruction. Positive effects were found for two cooperative learning programs. Outcomes were similar for disadvantaged and nondisadvantaged students and for students of different ethnicities. Consistent with an earlier review of elementary programs, this article concludes that programs that affect daily teaching practices and student interactions have more promise than those emphasizing textbooks or technology alone.

Improving High School Performance

Rumberger, R. What the Federal Government Can Do to Improve High School Performance. Washington, D.C.: Center on Education Policy. 2009. [Technical Report]

Improvement strategies, especially more comprehensive ones, will not be successful until critical aspects of capacity and context are improved (p83)

The research literature has identified two broad factors that affect implementation: will and capacity (McLaughlin, 1987; McLaughlin, 1990). Will and capacity refer to traits of both individuals and institutions. At the individual level, will refers to the motivation and commitment of educators—teachers and administrators—to implement reform strategies. (p75)

The individual capacity of teachers and administrators to carry out reforms is clearly important. The capacity of teachers to implement reforms, which, again, usually means changing their instructional practices, is a time-consuming, multi-stage process that includes persuasion over the need for reform. (p75)

The capacity of individuals—teachers and administrators—as well as the institutional capacity of the school itself are key factors to successful implementation. School capacity depends on having sufficient and correct alignment of resources (including sufficient time); it also depends on coherence in its efforts across all the demands placed on schools and their staffs by districts, as well as state and federal policy requirements. Building capacity also depends on having sufficient will or readiness, especially among school and district leadership, to build capacity and initiate reform. (p83)

Summary:

High schools play a crucial role in preparing students for college, work, and citizenship. Yet, by many accounts, U.S. high schools are not performing any of these tasks well. This situation has prompted calls for improving high school performance. This report reviews past efforts to reform high schools, examines why those efforts have largely been unsuccessful, and suggests what the federal government can do to improve high school performance.

In order to improve the performance of U.S. high schools, it is first necessary to identify the purposes and goals of high schools and then develop suitable measures of school performance to determine the extent to which those goals are met. Only then can any serious effort be made to improve high school performance. In the current era of standards-based accountability, reform efforts have focused on raising student academic performance as measured by course credits, test scores, and educational credentials. Yet research studies and surveys of employers suggest students need a wide variety of non-academic as well as academic skills to be successful in college, the workplace, and in their adult lives.

A number of approaches have been developed for improving high schools, including targeted approaches that focus on specific facets of the school (instruction, student support, school restructuring); comprehensive strategies that redesign all aspects of the school or create new schools; collaborative approaches that create partnerships between schools and outside agencies; and systemic approaches that alter requirements for all schools in the system. Although the research evidence on the effectiveness of specific approaches is limited, it does suggest that no one strategy is inherently more effective than the others.

Numerous large-scale initiatives to improve the performance of high schools in the U.S. have been undertaken in the past 20 years by government agencies, foundations, non-profit organizations, and independent developers. For the most part these efforts have been unsuccessful, although there was widespread variability in both the implementation and impact of the initiatives across schools, districts, and states. Evaluations of these efforts have identified a number of factors that limited their implementation and impact, with the most important being the lack of will and capacity of both individual educators and institutions to engage in sustained improvement efforts. One implication is that strategies for improving high schools will not be successful until critical aspects of capacity and context are improved.

The federal government can play an important role in improving U.S. high schools by shifting its focus from short-term accountability to long-term capacity building. Specifically, the federal government should:
1. Support the development of broader indicators of student progress and outcomes, and include these indicators in the National Assessment of Educational Progress.
2. Help build the capacity of state governments and technical-assistance providers to support improvement efforts and capacity building in districts and schools.
3. Develop guidelines to insure that states do a better job of matching reform strategies to the capacity of schools and districts in need of improvement.
4. Improve coherence among federal policy initiatives, between federal and state initiatives, and between government and foundation initiatives.
5. Support the development of more comprehensive state and local data systems that not only measure educational inputs and outputs, but also district and school readiness and capacity to initiate reform as well as progress toward improving student outcomes.

Challenges New Science Teachers Face - Lit Review (2006)

Davis E.A., Petish, D. & Smithey, J. (2006). Challenges new science teachers face. Review of Educational Research, 76, 607–651

What are the challenges that new science teachers face in trying to meet the increasingly high expectations laid out for them in current reform documents? What do we expect new science teachers to know and be able to do? [p609] Science teachers are expected to understand: (1) the content and disciplines of science, (2) learners, (3) instruction, (4) learning environments, and (5) professionalism. [p607] New elementary teachers may face even greater challenges in teaching science than do their secondary counterparts, since they typically teach multiple subjects, including all areas of science.[p608]

The authors see standards [INTASC, 2002 & NSES] for teaching as appropriate to use as a frame for their work: They concisely represent the kinds of things that teachers should probably be able to do—and indeed, that teacher educators should help them achieve—and are the result of some form of consensus-building at a higher level than a single scholar’s viewpoint.

1. The first theme, understanding the content and disciplines of science, focuses on the teacher’s understanding of “the major concepts, assumptions, debates, processes of inquiry, and ways of knowing that are central” to the science discipline(s) she teaches (INTASC, 1992, p. 14). (p613)

• new teachers have relatively weak understandings of science overall (p613)
• In general, the preservice teachers held alternative ideas that were similar to those that have been identified in students (Bendall et al., 1993; Ginns & Watters, 1995; Schoon & Boone, 1998; Trumper, 2003). Even secondary preservice teachers showed poor understandings of topics (Haidar, 1997). (p615)
• preservice secondary science teachers in their studies initially lacked understanding of the connections between concepts in the disciplines they were to teach; but these understandings improved over time and with experience.(p615)
• In sum, preservice teachers seem, for the most part, to lack adequate understandings of science content. This trend is especially pronounced at the elementary level; results are more mixed at the secondary level. Though most studies do not characterize change over time, those that do indicate that the preservice teachers’ knowledge may improve over time. (p615)
• Overall, these papers illustrate that many preservice teachers have unsophisticated understandings of inquiry and related skills, though of course individuals vary. p616
• The studies in this area consistently find that most (though not all) new teachers have naive beliefs about the nature of science (see Lederman, 1992, for a review). p616

2. Theme 2, understanding learners, focuses on teachers’ understanding of how students learn and develop, and includes an appreciation of the variation in learners and in how they approach learning (INTASC, 1992). Teachers should believe that all students can “learn at high levels” (INTASC, 1992, p. 18), regardless of their cultural and language backgrounds. Teachers should “recognize and respond to student diversity and encourage all students to participate fully in science learning” (NRC, 1996, p. 32), and they should “display and demand respect for the diverse ideas, skills, and experiences of all students” (NRC, 1996, p. 46). p618

• new science teachers’ ideas about learners can become more sophisticated with time and support p618
• in general these teachers struggle with understanding their learners; p618
• their practices with regard to their learners are often naive. p618
• preservice teachers tend not to consider students or student learning very extensively, very carefully, or in very sophisticated ways p618
• The preservice elementary and secondary teachers tended to have very limited ideas about what to do, instructionally, with students’ ideas p619
• Rodriguez, for example, studied 18 preservice secondary teachers, including 4 focus teachers, and found that the preservice teachers tended to feel hopeless and overwhelmed about working with diverse students. p620
• The studies reported within this theme show that, in general, new teachers do not have very clear ideas about what to do with regard to students’ ideas or backgrounds; at least at the elementary level, preservice teachers seem initially to want mainly to engage, interest, motivate, or manage their students. p620

3. The third theme, understanding instruction, means that the teacher “understands principles and techniques, along with advantages and limitations, associated with various instructional strategies” (INTASC, 1992, p. 20) and “uses a variety of instructional strategies” (INTASC, 2002, p. 4). [p621]

• Overall, these studies illustrate a mismatch between teachers’ ideas and practices—their ideas about instruction seem generally to be more sophisticated and innovative than are their actual practices. [p621]
• One basic challenge that new teachers face is developing sophisticated ideas about science instruction; these papers indicate that though improvement can occur, it is neither guaranteed nor necessarily long-lasting.
• In general, these studies — mostly conducted with secondary teachers — indicate that when new teachers have stronger subject matter knowledge, they are more likely to engage in more sophisticated teaching practices. p622
• Overall, then, teachers’ subject matter knowledge seems related to their instructional ideas and practice; stronger science knowledge typically co-occurs with more sophisticated ideas or practices with regard to instruction (though most of the studies related to this point were conducted with secondary teachers). p623
• In sum, new teachers face many challenges with regard to using effective instructional approaches, including lacking relevant subject matter knowledge, not knowing how to enact their instructional ideas, and being resistant to certain innovative practices. With support, though, teachers can begin to move along a positive trajectory.p624

4. The fourth theme, understanding learning environments, emphasizes teachers’ understandings of how to set up productive classroom environments for science learning. This involves creating “a learning environment that encourages positive social interaction, active engagement in learning, and self-motivation” (INTASC, 1992, p. 22) and understanding “the principles of effective classroom management and [using] a range of strategies to promote positive relationships, cooperation, and purposeful learning in the classroom” (INTASC, 1992, p. 22). [p627]

• In sum, based on the few studies we identified in this area, we see that new teachers tend to have concerns about and struggles with management, sometimes leading them to engage in less reform-oriented teaching practices. p628

5. The final theme, professionalism—or becoming a professional—emphasizes being “a reflective practitioner who continually evaluates the effects of his/her choices and actions on others (students, parents, and other professionals in the learning community) and who actively seeks out opportunities to grow professionally” (INTASC, 1992, p. 31). p629

• Appleton and Kindt (2002) identify three important aspects of knowledge of schools: understanding the priority of science in a school culture, understanding the degree of personal choice one has about the curriculum, and identifying and obtaining resources for science teaching. p629

Supports
Supportive Science Coursework

• simply requiring more science content courses is not enough to enable teachers to develop improved understanding of science p633
• Many science teacher educators assume that teachers should engage in reform-oriented practices as learners if they are to learn more inquiry-oriented teaching practices and become more knowledgeable about the science content, scientific inquiry, and the nature of science;

Supportive Preservice Teacher Education

• Science methods courses and teacher education programs can, of course, help to promote improved understanding of instruction p634
• Zembal-Saul and her colleagues (2000) describe the importance of engaging preservice elementary teachers in multiple cycles of planning, teaching, and reflection, over the course of a year. The preservice teachers who participated in the program emphasizing elementary science teaching improved in how they organized instruction around important scientific ideas (a challenge we identified for elementary teachers, who tended instead to focus on activities) and came to recognize the importance of accounting for their learners as they planned instruction p634
• Field experiences also help preservice teachers to overcome certain challenges [p635]

Supportive Induction and Professional Development Programs

• action research
• collegial relationships
• educative curriculum materials

---

p624 "In sum, new teachers face many challenges with regard to using effective instructional approaches, including lacking relevant subject matter knowledge, not knowing how to enact their instructional ideas, and being resistant to certain innovative practices. With support, though, teachers can begin to move along a positive trajectory."

p626 "One study provides a counterpoint to this general trend, though it focuses on only a single teacher (Abell & Roth, 1994). This preservice elementary teacher developed effective coping strategies while taking her science methods course. She infused additional science into an otherwise limited science curriculum and inspired the other teachers in her school to use cooperative learning experiences. Her personal attributes and the features of her student teaching context helped her to take risks in her environment."

Abell, S. K., & Roth, M. (1994). Constructing science teaching in the elementary school: The socialization of a science enthusiast student teacher. Journal of Research in Science Teaching, 31(1), 77–90.