Crooks, Kane & Cohen (2008) - Threats to the valid use of assessments

Crooks, T. J., Kane, M. T., & Cohen, A. S. (2008). Threats to the valid use of assessments. In H. Wynne (Ed.) Student assessment and testing: Vol. 2 (Chapter 21, pp. 151-171). Thousand Oaks, CA: Sage.

Main points:
1) Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment (Messick, 1989, p. 13) [p. 150]
2) Assessment can be depicted as a chain of eight linked stages (administration, scoring, aggregation, generalization, extrapolation, evaluation, decision and impact) which form an assessment system
3) Problems and issues in any of the links can threaten the confidence in an assessment system, and undermine any inferences and claims from this system
4) Validity estimation relies heavily on human judgment and is therefore harder to carry out, report and defend. [p. 150]
5) High reliability is necessary but not sufficient for high validity. Some degree of
reliability is essential for validity. Reliability establishes an upper limit for validity.

Theoretical Framework
Concept of validity argument: interpretations of assessments of performances involve a linked series of inferences and assumptions (Kane 1992; Shepard, 1993; Cronbach, 1988). If the inferences and assumptions can be identified, the plausibility can be examined by logical and empirical means, and the importance of each can be debated. [p.151]

Assessment validation model
Crooks, Kane and Cohen’s (2008) assessment validation model depicts assessment as involving eight linked stages:
(1) Administration of assessment tasks to the student.
(2) Scoring of the student’s performances on the tasks.
(3) Aggregation of the scores on individual tasks to produce one or more combined scores.
(4) Generalization from the particular tasks included in a combined score to the whole domain of similar tasks (the assessed domain).
(5) Extrapolation from the assessed domain to a target domain containing all tasks relevant to the proposed interpretation.
(6) Evaluation of the student’s performance, forming judgments.
(7) Decision on actions to be taken in light of the judgments.
(8) Impact on the student and other participants arising from the assessment processes, interpretations and decisions. [p. 153]