Crooks, Kane & Cohen (2008) - Threats to the valid use of assessments

Crooks, T. J., Kane, M. T., & Cohen, A. S. (2008). Threats to the valid use of assessments. In H. Wynne (Ed.) Student assessment and testing: Vol. 2 (Chapter 21, pp. 151-171). Thousand Oaks, CA: Sage.

Main points:
1) Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment (Messick, 1989, p. 13) [p. 150]
2) Assessment can be depicted as a chain of eight linked stages (administration, scoring, aggregation, generalization, extrapolation, evaluation, decision and impact) which form an assessment system
3) Problems and issues in any of the links can threaten the confidence in an assessment system, and undermine any inferences and claims from this system
4) Validity estimation relies heavily on human judgment and is therefore harder to carry out, report and defend. [p. 150]
5) High reliability is necessary but not sufficient for high validity. Some degree of
reliability is essential for validity. Reliability establishes an upper limit for validity.

Theoretical Framework
Concept of validity argument: interpretations of assessments of performances involve a linked series of inferences and assumptions (Kane 1992; Shepard, 1993; Cronbach, 1988). If the inferences and assumptions can be identified, the plausibility can be examined by logical and empirical means, and the importance of each can be debated. [p.151]

Assessment validation model
Crooks, Kane and Cohen’s (2008) assessment validation model depicts assessment as involving eight linked stages:
(1) Administration of assessment tasks to the student.
(2) Scoring of the student’s performances on the tasks.
(3) Aggregation of the scores on individual tasks to produce one or more combined scores.
(4) Generalization from the particular tasks included in a combined score to the whole domain of similar tasks (the assessed domain).
(5) Extrapolation from the assessed domain to a target domain containing all tasks relevant to the proposed interpretation.
(6) Evaluation of the student’s performance, forming judgments.
(7) Decision on actions to be taken in light of the judgments.
(8) Impact on the student and other participants arising from the assessment processes, interpretations and decisions. [p. 153]

Gipps (2008) - Socio-cultural aspects of assessment

Gipps, C. (2008). Socio-cultural aspects of assessment. In H. Wynne (Ed.) Student assessment and testing: Vol. 1 (Chapter 8, pp. 252-291). Thousand Oaks, CA: Sage.

MAIN POINTS

1.     At all levels, assessment is a social activity and that we can understand it only by taking account of the social, cultural, economic, and political contexts in which it operates. (p. 252)

2.     Assessment plays an important role in cultural and social reproduction, in allocating educational and economic opportunities, and more recently, to control curriculum and teaching. (p. 264)

3.     Changes in assessment practice and design reflect changes in world view, a resulting change in epistemology, and new understandings of learning. (p. 273)

4.     There are complex interactions among students, teachers, and assessment. (p. 284)

5.     When designing assessment, there are trade-offs between reliability, validity, and assessment of higher order thinking skills. (p. 283)

6.     Theories about intelligence and learning have implications for assessment design (p.272)

7.     Although new approaches to assessment have the promise of being more equitable, performance assessments on their own will not enhance equity. (p. 283)

CLAIMS , EVIDENCE & REASONING

1.     CLAIM: Although IQ testing, objective testing, and external examinations were seen originally as equitable tools for selection and certification purposes, a sociological critique calls this into question. (p. 260)

EVIDENCE & REASONING:

a.     Performance at school may be affected by social and cultural background factors. Among these factors are poverty, poor resources at home and/or at school, absenteeism owing to work or domestic duties, mismatch between the language and culture of the home and the school, gender bias, and ethnic discrimination. As a result, examinations may be biased, and furthermore, because of their role in certification, they may institutionalize and legitimate social stratification. (p.361)

b.    Cultural capital argument: children from lower social groups are not less intelligent or less academically capable, but children from middle-class homes are better able to do well at school because of the correspondence of cultural factors between home and school. As a result, examinations have a legitimating role in that they allow the ruling classes to legitimate the power and prestige they already have. (p.361) [see Bourdieu & Passeron (1976)]

2.     CLAIM: It is possible to run a national assessment program that includes high-quality examinations and some performance assessment, and it is possible to design an assessment program with different features and purposes at different levels of the school system. (p.283)

EVIDENCE & REASONING: experience in England in the 1990s (Stobart & Gipps, 1997; James & Gipps, 1998) provides existence proofs that it is possible to implement new assessment systems at scale.

3.     CLAIM: Although performance assessment and evaluation of culturally sensitive classroom-based learning have the potential to foster multicultural inclusion and facilitate enhanced learning, performance assessment on its own will not enhance equity. (p. 283)

EVIDENCE & REASONING: Consideration must still be given to students' opportunity to learn (Linn, 1993), the knowledge and language demands of the task (Baker & O'Neil, 1995), and the criteria used for scoring (Linn, Baker, & Dunbar, 1991). Clearly, as with traditional forms of assessment, questions of fairness arise in the selection of tasks and in the scoring of responses. Furthermore, the more informal and open-ended such assessment becomes, the greater the reliance on the judgment of the teacher/assessor. Here we come again to the issue of power and control, a theme of this chapter. Alternative forms of assessment do not, of themselves, alter power relationships and cultural dominance in the classroom. (p.283)

FRAMEWORKS

Psychometric theory: Psychometric theory developed originally from work on intelligence and intelligence testing. The underlying notion was that intelligence was innate and fixed in the way that other inherited characteristics are, such as skin color. Intelligence could therefore be measured (since, like other characteristics, it was observable), and, on the basis of the outcome, individuals could be assigned to streams, groups, or schools that were appropriate to their intelligence (or "ability," as it came to be seen). … With the psychometric model comes an assumption of the primacy of technical issues, notably standardization and reliability (Goldstein, 1996). (p.263)

Constructivist learning theory: students learn by actively making sense of new knowledge, making meaning from it (Iran-Nejad, 1995), and mapping it into their existing knowledge map or schema. Shepard (1991) notes that "contemporary cognitive psychology has built on the very old idea that things are easier to learn if they make sense." (p.271)

Sociocultural learning theory: Socioculturalist assume human agency in the process of coming to know, but socioculturalists further argue that meaning derived from interactions is not exclusively a product of the person acting. They view the individual engaged in relational activities with others. Building on Vygotsky's arguments about the importance of interaction with more knowledgeable others and the role of society in providing a framework for the child's learning, sociocultural theorists thus describe learning in terms of apprenticeship (e.g., Brown et al., 1993; Glaser, 1990; Rogoff, 1990), legitimate peripheral participation (Lave & Wenger, 1991), or negotiation of meaning in the construction zone (Newman, Griffin, & Cole, 1989). (p. 271)

Tittle’s (1994) framework for an educational psychology of assessment: there are three dimensions: the epistemology and theories involved (both general and in relation to subject matter); the interpreter and user, whose presence, characteristics, needs, and values must be brought into the frame; and the characteristics of the assessment itself.

KEEPSAKES

1. Testing is now being used to control curriculum and teaching. (p. 283)
2. Developments in cognition and learning are telling us to assess more broadly, in context, and in depth. This requires methods of assessment that do not lend themselves readily to traditional reliability, highlighting the tension between types and purposes of assessment. (p. 283)
3. From an interpretivist viewpoint, it is important to acknowledge the complexity of interactions among students, teachers, and assessment. Factors such as students' perceptions of how testing affects them (Herman et al., 1997), student and teacher confidence in the veracity of test results, and differences in student and teacher perceptions of the goals of assessment all need to be considered. (p. 284)
4. We need to bring out into the open the nature of the power relationship in teaching and assessment and point out the possibility of reconstructing this relationship. Perhaps most important, we need to encourage teachers to bring pupils into the process of assessment, in order to recognize their social and cultural background, and into self-assessment, in order to develop their evaluative and metacognitive skills. (p.286)
5. A key direction for the future lies in the development of teachers' classroom assessment skills. It is evident from this chapter that some teachers are operating in collaborative, constructivist ways supported by portfolio work, for example, or as evidenced by their feedback to learners. Such practice is not common but clearly can become part of the teacher's repertoire. This implies the continued development of new assessment strategies for use by teachers, involving group and interactive assessment and interview and portfolio approaches. It will involve extending teachers' skills in observation and questioning while making them aware of social and cultural influences on the assessment process. (p. 286)

Gutiérrez & Orellana (2006) - The "Problem" of English Learners: Constructing Genres of Difference

Gutiérrez, K. & Orellana, M.F. (2006). The "Problem" of English Learners: Constructing Genres of Difference. Research in the Teaching of English, 40, 502-507.

Key terms:

selective exemplification: Taking a "slice of life" of English Learners in a way that does not show complexity and account for a fuller understanding of how that practice fits into a larger ecology of a student's life (or literacy repertoire) raises serious questions about the validity of claims concerning what counts as literacy for these students. We want to highlight a convention we find particularly troubling both methodologically and ethically, that is, the practice of "selective exemplification" in empirical work. Specifically, we refer to zooming in our analytical lens on topics that sensationalize, exoticize, and romanticize, or, conversely, zooming out to essentialize or homogenize English Learners in ways that blind us or our readers to the fuller, more complicated realities of these students' lives. (504)

"framing" English Learners: the way the "problem" of English Learners and other non-dominant students is typically framed. This framework begins with the statistical set-up to the problem that we described above; it is here that a group of non-dominant students are first isolated and identified as a distinct, unified group that is somehow different from an invisible and mostly unspecified norm. In much research, the grouping is based on racial or ethnic-group categorization, not language practices or other distinctions, even when the issue being explored is one of language. (505)

difference framework: white middle-class students are often considered the "norm" in terms home practices, values, beliefs, attitudes, identities, or skills. "The descriptors that researchers use can reveal an underlying set of assumptions about normativity and serve to construct the population as 'other' even when there is no direct comparison with another community or group." (505)

Key ideas:

In this work we have struggled against commonplace approaches to conceptualizing and reporting research that unwittingly create or reinforce deficit views of these students and their communities. (502)

Almost invariably, research reports contextualize the study with statistics about changing demographics in the U.S. as the rationale for the urgency of studying English Learners. Taken together, these data often function to paint pictures of poor, struggling students in schools and communities with limited resources. (502)

We, of course, are not discouraging the use of relevant descriptive statistics as they can serve to identify critical issues and inequities; instead, we call attention to the ways they can constitute deficit-oriented, uncomplicated, and uneven narratives about students for whom English is a second language. (503)

Some time ago, our colleague Elinor Ochs (1979) wrote that researchers engaged in transcribing their participants' talk and interaction are also engaged in the process of building theory about the participants and their practices. Similarly, we argue that the ways we marshal data to make generalizations about what is normative for English Learners, their language and literacy practices, and their home communities build theories about normativity, often without regard to students' existing repertoires of practice or the additional sets of challenges English Learners experience. (504)

There are a number of ways to denote normativity or regularity in English Learners, and one salient tendency in the genre of English Learner studies is to focus on a narrow range of what constitutes the students' literacy toolkit and repertoires. Selectively focusing on what we as researchers find as salient, fascinating, or unusual to us or the field - to the exclusion of the widely diverse range of practices, interests, and proclivities of English Learners- serves to reinforce a kind of analytical reductiveness too often associated with discussions of non-dominant students and communities. (504)

As we try to build a corpus of studies with English Learners, there is a critical need for more nuanced and complete analyses and depictions of students' literacy practices observed across a range of settings, tasks, and contexts over sustained periods of time. (505)

But when the issue is second- language learning, members of this student population should be identified by more than their membership in an ethnic category, and race/ethnicity should not be conflated with language abilities (just as it should not be conflated with social class).(505)

The difference framework has a long history, most perniciously in the overtly deficit-driven notion that some groups of children suffer from "cultural deprivation" or live in "cultures of poverty." The only slightly more benign version of a difference framework - that of "cultural mismatch theory," which spotlights presumed differences between school and home language practices - has had significant longevity in educational research, and, especially, in language and literacy studies. (506)

Heath (1982) - What no bedtime story means

Heath, S. B. (1982). What no bedtime story means: Narrative skills at home and school. Language in Society. 11(2):49-76.

Maintown [is a] white middle-class neighborhoods in a city of the Piedmont Carolina, families of fifteen primary-level school teachers (p.52)

Roadville is a white working-class community of families steeped for four generations in the life of the textile mill. (p57)

Trackton is a working-class black community whose older generations have been brought up on the land, either farming their own land or working for other landowners. However, in the past decade, they have found work in the textile mills. (p57)

Maintown: bedtime stories; children learn/taught what-explanations, classification and knowledge construction, decontextualization, affective comments, reason-explanations (p71)

Roadville: bedtime stories; children learn/taught what-explanations but little or no decontextualization, emphasis on personal experience; no analytic statements or universal truths (p71)

Trackton: no bedtime stories; few occassions for reading to or with children; good storytelling valued; talked about events they witness are rewarded

Learning how to take meaning from writing before one learns to read involves repeated practice in using and learning from language through appropriate participation in literacy events such as exhibitor/questioner and spectator/respondendt yads (Scollon and Scollon 1979) or group negotiation of the meaning of a written text. Children have to learn to select, hold, and retrieve content from books and other written or printed texts in accordance with their community's rules or "ways of taking," and the children's learning follows community paths of language socialization. In each society, certain kinds of childhood participation in literacy events may precedeo thers, as the developmental sequence builds toward the whole complex of home and community behaviors characteristic of the society. [p.70]

Roadville and Trackton tell us that the mainstream type of literacy orientation is not the only type even among Western societies. They also tell us that the mainstream ways of acquiring communicative competence do not offer a universally applicable model of development. They offer proof of Hymes' assertion a decade ago that "it is impossible to generalize validly about 'oral' vs. 'literate' cultures as uniform types" (Hymes 1973: 54). (p.73)

Yet in spite of such warnings and analyses of the uses and functions of writing in the specific proposals for comparative development and organization of cultural systems (cf. Basso 1974: 432), the majority of research on literacy has focused on differences in class, amount of education, and level of civilization among groups having different literacy characteristics. (p.73-74)

"We need, in short, a great deal of ethnography" (Hymes 1973: 57) to provide descriptions of the ways different social groups "take" knowledge from the environment. For written sources, these ways of taking may be analyzed in terms of types of literacy events, such as group negotiation of meaning from written texts, individual "looking things up" in reference books, writing family records in Bibles, and the dozens of other types of occasions when books or other written materials are integral to interpretation in an interaction. These must in turn be analyzed in terms of the specific features of literacy events, such as labelling, what-explanation, affective comments, reason-explanations, and many other possibilities. Literacy events must also be interpreted in relation to the larger sociocultural patterns which they may exemplify or reflect. (p.74)

The culture children learn as they grow up is, in fact, "'ways of taking" meaning from the environment around them. The means of making sense from books and relating their contents to knowledge about the real world is but one "'way of taking" that is often interpreted as "natural" rather than learned. The quote also reminds us that teachers (and researchers alike) have not recognized that ways of taking from books are as much a part of learned behavior as are ways of eating, sitting, playing games, and building houses. (p.49)

In some communities [the] ways of schools and institutions are very similar to the ways learned at home; in other communities the ways of school are merely an overlay on the home-taught ways and may be in conflict with them. (p.50)

Just how does what is frequently termed "the literate tradition" envelope the child in knowledge about interrelationshipsb etween oral and written language, between knowing something and knowing ways of labelling and displaying it? We have even less information about the variety of ways children from non-mainstream homes learn about reading, writing, and using oral language to display knowledge in their preschool environment. The general view has been that whatever it is that mainstream school-oriented homes have, these other homes do not have it; thus these children are not from the literate tradition and are not likely to succeed in school. (p.50)

A key concept for the empirical study of ways of taking meaning from written sources across communities is that of literacy events: occasions in which written language is integral to the nature of participants' interactions and their interpretive processes and strategies. Familiar literacy events for mainstream preschoolers are bedtime stories, reading cereal boxes, stop signs, and television ads, and interpreting instructions for commercial games and toys. In such literacy events, participants follow socially established rules for verbalizing what they know from and about the written material. Each community has rules for socially interacting and sharing knowledge in literacy events. (p.50)

The bedtime story is a major literacy event which helps set patterns of behavior that recur repeatedly through the life of mainstream children and adults. (p.51)

Before the age of two, the child is socialized into the "'initiation-reply-evaluation sequences" repeatedly described as the central structural feature of classroom lessons. (p.51) Training in ways of responding to this pattern begins very early in the labelling activities of mainstream parents and children. (p.52)

Reading for comprehension involves an internal replaying of the same types of questions adults ask children of bedtime stories. We seek what-explanations, asking what the topic is, establishing it as predictable and recognizing it in new situational contexts by classifying and categorizing it in our mind with other phenomena. (p54)

These various ways of taking [from books] are sometimes referred to as "cognitive styles" or "learning styles." It is generally accepted in the research literature that they are influenced by early socialization experiences and correlated with such features of the society in which the child is reared as social organization, reliance on authority, male-female roles, and so on. These styles are often seen as two contrasting types, most frequently termed "field independent-field dependent" (Witkin et al. 1966) or "analytic-relational" (Kagan, Sigel, and Moss I963; Cohen 1968, 1969, 1971). The analytic field-independent style is generally presented as that which correlates positively with high achievement and general academic and social success in school. Several studies discuss ways in which this style is played out in school - in preferred ways of responding to pictures and written text and selecting from among a choice of answers to test items. (p55)

In both [the Roadville and Trackton] communities, children go to school with certain expectancies of print and, in Trackton especially, children have a keen sense that reading is something one does to learn something one needs to know (Heath 1980). ... Roadville and Trackton view children's learning of language from two radically different perspectives: in Trackton, children "learn to talk," in Roadville, adults "teach them how to talk." (p57)

Nieto, S. (2005). Public education in the twentieth century and beyond

Nieto, S. (2005). Public education in the twentieth century and beyond: High hopes, broken promises, and an uncertain future. Harvard Educational Review 75(1):43-64.

Theories used explain the underachievement of students of diverse cultural, linguistic, and racial backgrounds.

1. Genetic and Cultural Inferiority - proponents assert that students of racial minority and economically poor backgrounds are genetically or culturally inferior

2. Economic and Social Reproduction Theories - schools tend to serve the interests of the dominant classes by reproducing the economic and social relations of society; schools help to create and maintain these inequalities.

3. Cultural Incompatibility Theory - school culture and home culture are often at odds, and the result is a “cultural clash” that gets in the way of student learning

4. Sociocultural Explanations for School Achievement - cultural practices of particular communities are linked with their students' learning in school settings. Shirley Brice Heath's (1983) classic research with a Black community that she called "Trackton" is a persuasive example of the power of aligning teaching to students' cultural practices.

5. Students as Castelike Minorities - According to Ogbu, given the long history of discrimination and racism in the schools, involuntary minority children and their families are often distrustful of the education system. It is not unusual for students from these groups to engage in what Ogbu called cultural inversion, that is, to resist acquiring and demonstrating the culture and cognitive styles identified with the dominant group.

6. Resistance theory, as articulated by scholars such as Henry Giroux (1983), Jim Cummins (1996), Herb Kohl (1994), and others, adds another layer to the explanation of school failure. According to this theory, not learning what schools teach can be interpreted as a form of political resistance.

7. Care, Student Achievement, and Social Capital - for Nel Noddings (1992), care is just as — and in some cases, even more — important than entrenched structural conditions that influence student learning. Valenzuela (1999): subtractive schooling concept; Ricardo Stanton-Salazar (1997): social capital networks framework

Newer theories: race, context of incorporation, and others - are also at work. Haitians, Nicaraguans, and Mexicans as economic refugees, who have significantly lower earnings than Cubans and Vietnamese, even after controlling for level of education, knowledge of English, and occupation. Alejandro Portes and Rubén Rumbaut (2001) found that immigrants fleeing from Communism are received more favorably than those fleeing economic exploitation.

Response to an OpEd by Nicholas Kristof

Below is my response to Response to an OpEd piece "How Mrs. Grady Transformed Olly Neal" by Nicholas Kristof (Published: January 21, 2012 in The New York Times)

I think we should celebrate teachers like Mildred Grady who made an extra effort to help a child and encourage his interest in reading, and Olly Neal who succeeded under challenging circumstances. Both sound like exceptional human beings.

However, I think Kristof's column and the simple narrative that a single great teacher transformed the life of an "at-risk" youth and was the key factor that enabled him to become educated, get a law degree and eventually become a court judge, is somewhat misleading. The column obscures the other positive influences in Olly Neal's life that were probably as or more important than Mrs. Grady - parents, family, mentors, etc. In addition, it is unlikely that Mr. Neal would have succeeded if he didn't have many competent and caring teachers through his life - before and after Mrs. Grady - and access to functional education systems.

While a great teacher can make a difference with some troubled, surly kids in a high-poverty environment, the odds of this happening are very low struggling in many inner-city schools today. The schools with the most troubled kids in high-poverty environments tend to have the toughest working conditions, worst funding, and weakest leadership that few good teachers would want to work or stay at.

What is Culture?

Excerpt from Michael Cole's Cultural Psychology (1996):
Culture, according to Hutchins, should be thought of as a process, not as "any collection of things, whether tangible or abstract." Culture "is a process and the 'things' that appear on list-like definitions of culture are residua of the process. Culture is an adaptive process that accumulates the partial solutions to frequently encountered problems." (Hutchins, p. 354)

---

Culture includes: artifacts, schemas, scripts, models, practices, heritages, history, activity systems

---

Excerpt from McDermott & Varenne (1995). Culture as Disability:

Anthropologists define culture as well-bound containers of coherence that mark off different kinds of people living in their various ways, each kind separated from the others by a particular version of coherence, a particular way of making sense and meaning. (p. 325)

The coherence of any culture is not given by members being the same, nor by members knowing the same things. Instead, the coherence of a culture is crafted from the partial and mutually dependent knowledge of each person caught in the process and depends, in the long run, on the work they do together. Life in culture, Bakhtin (1984[1940]) reminds us, is polyphonous and multivocalic; it is made of the voices of many, each one brought to life and made significant by the others, only sometimes by being the same, more often by being different, more dramatically by being contradictory. Culture is not so much a product of sharing as a product of people hammering each other into shape with the well-structured tools already available. We need to think of culture as this very process of hammering a world. When anthropologists instinctively celebrate the coherence of culture, they imply that all the people in the culture are the same, as if stereotyping is a worthy practice as long as it is done by professionals. Thick brush-stroke accounts of Samoans or Balinese, to stay with Margaret Mead, may give some hints as to what Samoans and Balinese must deal with in their daily life, but they can greatly distort the complexity of Samoans and Balinese as people. The coherence of culture is something many individuals, in multiple realities, manage to achieve together; it is never simply the property of individual persons. (p.326)

The anthropological instinct has been perhaps most destructive when applied to the divisions and inequalities that exist inside a presumed cultural container, that is, the culture “of which they are a member,” “to which they belong,” or “in which they participate.” The problem in assuming that there is one way to be in a culture encourages the misunderstanding that those who are different from perceived norms are missing something, that it is their doing, that they are locked out for a reason, that they are in fact, in reality, disabled. If it is distorting to describe Samoans and Balinese without an account of the full range of diversity to be found in Samoa or Bali, imagine how distorting it can be in complex divided fields like the United States.

When culture is understood as the knowledge that people need for living with each other, it is easy to focus on how some always appear to have more cultural knowledge than others, that some can be a part of everything and others not, that some are able and others not. Before entering the Country of the Blind, Nunez thought that sight was essential to being fully cultured and that having sight in a world of people who cannot see would net him the cultural capital of a king. The anthropological instinct teaches us that he was arrogant to think he knew better and foolish to not learn from his masterful subjects. The instinct gives us an essential insight, and we can be thankful that anthropology has taken its place in the human sciences.