Lesh & Clarke (2000) Formulating operational definitions of desired outcomes of instruction in mathematics and science education

Lesh, R.A. & Clarke, D. (2000). Formulating operational definitions of desired outcomes of instruction in mathematics and science education. In A.E. Kelly & R.A. Lesh (eds.) Handbook of Research Design in Mathematics and Science Education, pp 113-49. Mahwah, NJ: Lawrence Erlbaum Associates

p120: Later in this chapter, other explanations will be given about why teaching to tests, over a long period of time, tends to be a losing strategy. For now, a brief explanation is that: (a) students soon forget disorganized lists of facts and skills; (b) when facts and skills are "mastered" one-at-a-time and in isolation, students may never learn when to chose which one to use in particular situations; and (c) when instruction emphasizes only facts and skills, or content-independent problems solving processes, other exceedingly important goals of instruction are ignored.

p121: high scores on tests often are treated as if they went beyond being indicators of achievement toward actually being the goals of instruction. [JC: see Labaree and his discussion of credentialism]
p122: In past stages of history, educators have tended to think about the mind (and about the nature of mathematical knowledge) as if it were similar to the most sophisticated technology of the preceding age. For example, as civilizations evolved from the industrial revolution through the electronics revolution to the current age of biotechnologies, educators have shifted from machine metaphors (based on hardware) to computer metaphors (based on software) to organic metaphors (based on wetware of the type that characterizes the processes of the human brain). Yet, in the area of assessment, simple input-output models continue to dominate that are based on machine metaphors.

p122: (NCTM, 1989)
In spite of the best intentions of developers and implementors. it was unreasonable to expect that new products or programs would be used as intended in most schools and classrooms. The reason for this is that public schools as they now operate are integrated social systems. Tinkering with parts. such as changing textbooks or the number of required courses, fails to change other components of the system. The traditions of the system force new products to be used in old ways. Current educational practice is based on a coherent set of ideas about goals. knowledge. work. and technology that came from a set of "scientific management" principks growing out of the industrial revolution of the past century. These ideas about schooling need to be challenged and replaced with an equally coherent set of practices in light of the economic and social revolution in which we are now engaged. Current school mathematics operates within a coherent system; reform will happen only if an equally coherent system replaces it. (National Council of Teachers of Mathematics. 1989)

p126: Behavioral Objectives Involve Three Parts
Given {specified conditions} the student will exhibit {specified behaviors} with identifiable quality {perhaps specified as percents correct on relevant samples of tasks, or perhaps specified in terms of a correspondence with certain criteria for excellence}.

In mathematics and science education, the main problem with behavioral objectives is that not all forms of learning consist of rules (facts, skills, procedures); and, if attempts are made to reduce more complex conceptual systems to checklists of rules, the following sorts of difficulties arise.

For instance, it may be true that a great artist (or a great athlete) should be able to perform well on certain basic drills and exercises (calisthenics); nonetheless, a program of instruction (or assessment) that focuses on nothing more than these checklists of basic skills is unlikely to provide adequate approaches to excellence.

p127: Cognitive objectives function similarly to the ways cyclotrons, cloud chambers, and vats of heavy water are used in physics. That is, they are defined operationally by specifying: (a) situations that optimize the chances that the targeted construct will occur in an observable form; (b) observation tools that enable observers to sort out signal from noise in the results that occur; and (c) quality assessment criteria that allow meaningful comparisons to be made among alternative possibilities.

p130: In particular, in the case of conceptual systems that students develop during the solution of individual problem solving sessions: (i) model-eliciting activities put students in situations where they confront the need to produce a given type of construct, and where the products that they generate require them to reveal explicitly important characteristics of their undedying ways of thinking; (ii) ways of thinking sheets focus on ways of recognizing arxi describing the nature of the constructs that students produce; and (iii) guidelines for assessing the quality of students' work provide criteria that can be used to compare the usefulness of alternative ways of thinking.

p133: Three final characteristics should be mentioned that pertain to operational definitions involving the development of students, teachers, and programs. First, the development of these problem solvers tends to be highly interdependent. Second, when something (or someone) acts on anyone of these complex systems, they tend to act back. Third, researchers (as well as the instruments that they use) usually are integral parts of the systems that they are hoping to understand and explain.

p.135-6: Examples of such teacher-level activities include generating:
(a) observation sheets that colleagues could use to make significant observations about students as they are working in groups;
(b) "ways of thinking sheets" that colleagues could use to give feedback to students about the strengths or weaknesses of their work; and
(c) quality assessment procedures that colleagues could use to lead discussions with students aimed at assessing the quality of the products that students produce.

For the purposes of this chapter, three of the most important characteristics of multitiered teaching experiments are that:
1. They use formative feedback and consensus-building to provide conditions that make it possible for students, teachers, and/or programs to develop in directions that are continually "better" without basing the next steps on preconceived notions of "best."
2. They emphasize the use of self-documenting activities that encourage students, teachers, and/or programs to learn while simultaneously producing trails of documentation that reveal important characteristics about the nature of what is being learned.
3. The preceding trends allow inferences to be made about future developments that are likely to occur.

p.136: One of the most important assumptions underlying the teacher level of multi-tiered teaching experiments is that, to improve teachers' teaching practices, it is not enough to ensure familiarity with a checklist of behavioral objectives; teachers also need to develop productive ways thinking about their teaching experiences. In particular, teachers need to develop productive ways of thinking about their students' learning and problem solving experiences.

p.136: Cognitively guided instruction is a name that's been given to an approach to teacher development that focuses on helping teachers become "reflective practitioners" by becoming familiar with new insights about the nature of students' developing mathematical knowledge (T. Carpenter & Fennema, 1992).

p.137: rather than telling teachers about their students' ways of thinking, thought-revealing activities are used so that teachers can make firsthand observations about their students' ways of thinking.

p.138: in studies of developing expertise of teachers (Lesh & Kelly, 1998), it is not necessary for a given description of expertise to be locked in at the beginning of a study (and used as the basis for a pretest-posttest design). Instead, increasingly sophisticated descriptions can be refined and documented gradually over the course of the study; and, at the end of the study, the validity of the description can be based on the trajectory of development that is revealed. In fact, as teachers, programs, and schools develop, their notions of excellence in teaching are primary factors that change. Therefore, if their progress continues to be measured using conceptions of excellence that existed at the beginning of a study, this practice tends to have significant negative influences on development.

p.139: An alternative to conformance models for curriculum change might be called planned experimentation.

p.140 Contexts that elicit complex performances don't necessarily require the relevant participants to reveal observable ways of thinking; and, they also don't necessarily provide useful tools for comparing or assessing the quality of competing systems. Nonetheless, it often is not difficult to identify situations that require the relevant systems to be elicited and revealed, and it often is not difficult to identify ways to compare and assess the results that are produced. For example, we may not know how to define what makes Granny a great cook; however, it still may be easy to identify situations that will elicit and reveal her capabilities, and it also may be easy to compare and assess alternative results that are produced.

p.141: some important principles to keep in mind include the following:
1. Achievement usually needs to be assessed using something more than brief tests that reduce expertise to simplistic lists of condition-action rules.
2. Students' relevant products or performances usually should include more than those that can be interpreted and assessed easily by a machine.
3. Emphasis needs to shift beyond superficial coverage of a large number of small tasks to the comprehensive treatment of a small number of big ideas.
4. Quality ratings should not ignore the conditions under which complex performances occur, and complex profiles should not be collapsed into simplistic scores on a scale that recognizes only a single dimension along which progress can be made.

p.144: three-fold character of model-eliciting activities: situations, products, and quality assessment criteria This structure focuses attention (as classroom practitioners or as educational researchers, for instance) on activities that create situations conducive to the constructs to be studied, requires products or observation tools through which the construct is made manifest, and produces these products in a form amenable to assessment against specified criteria for quality.

p.144: perhaps the most contentious of the suggestions put forward in this chapter is advocacy of the use of assessment to change the observed system purposefully toward the desired outcomes. In this, the authors anticipate the development of assessment systems that both reveal valued constructs and deliberately prompt adaptations within the systems studied in directions advantageous to the system or to the individual whose performances are assessed.