Validity: An Integrated Approach to Test Score Meaning and Use

Description

Validity is a clear, substantive introduction to the two most fundamental aspects of defensible testing practice: understanding test score meaning and justifying test score use. Driven by evidence-based and consensus-grounded measurement theory, principles, and terminology, this book addresses the most common questions of applied validation, the quality of test information, and the usefulness of test results. Concise yet comprehensive, this volume’s integrated framework is ideal for graduate courses on assessment, testing, psychometrics, and research methods as well as for credentialing organizations, licensure and certification entities, education agencies, and test publishers. Half Title Title Page Copyright Page Dedication Copyright Acknowledgment Table of Contents Preface 1 Introduction Foundational Measurement Concepts Underlying Validity What Is a Test? Tests as Samples Tests: Agnostic as to Format Standardized Tests Tests and Assessments Conclusions about Tests Inference: Always Required, Sometimes Risky Inferential “Leaps” Inferences: Necessarily Tentative Constructs: The Objects of Social Science Research and Development Tests, Inferences, and Constructs Validity Defined Validity Concerns Intended Score Meaning Validity Is a Property Validity Does Not Define a Construct Validity of Intended Inferences and Attention to Test Score Use Validity and Validation Why Is Validity So Important? Summary 2 Validity: The Consensus The Long Tradition of Professional Standards Related to Validity Areas of Consensus in Modern Validity Theory Principle 1: Validity Pertains to Test Score Inferences Principle 2: Validity Is Not a Characteristic of an Instrument Principle 3: Validity Is a Unitary Concept Principle 4: Validity Is a Matter of Degree Principle 5: Validation Involves Gathering and Evaluating Evidence Bearing on Intended Test Score Inferences Principle 6: Validation Is an Ongoing Endeavor Sources of Evidence for the Meaning of Test Scores Evidence Based on Test Content When and Why Is Validity Evidence Based on Test Content Important? Evidence Based on Internal Structure Internal Consistency Analysis Subscore Analysis Factor Analytic Methods Differential Item Functioning Structural Equation Modeling When and Why Is Validity Evidence Based on Internal Structure Important? Evidence Based on Response Processes What Is a Response Process? How Is Evidence Based on Response Processes Obtained? When and Why Is Validity Evidence Based on Response Processes Important? Evidence Based on Relationships to Other Variables Concurrent and Predictive Evidence Convergent and Discriminant Evidence Combining Convergent and Discriminant Evidence When and Why Is Validity Evidence Based on Relations to Other Variables Important? Conclusions 3 Validity and the Controversy of Consequences Roots of the Problem of Consequences in Validity Theory What Is Consequential Validity Anyway? Three Conceptual Problems with Consequential Validity 1. The Definitional Flaw 2. The Temporal Flaw 3. The Causal Flaw Three Practical Problems with Consequential Validity 1. The Problem of Delimitation 2. The Problem of Practice 3. The Problem of Location The Most Fundamental Problem with Consequential Validity Can Consequences of Test Use Ever Provide Validity Evidence? Conclusions and the Harm in the Status Quo 4 A Comprehensive Framework for Defensible Testing Part I: Validating the Intended Meaning of Test Scores A Comprehensive Framework for Defensible Testing Foundations of Validating Intended Test Score Inferences Validity and the Validation Process Major Threats to Test Score Meaning Construct-Relevant Variation and Construct-Irrelevant Variation Construct Misspecification and Construct Underrepresentation Reconsidering Sources of Validity Evidence Relationships among Variables Test Development and Administration Procedures A Revised Menu of Sources Summary and Conclusions 5 A Comprehensive Framework for Defensible Testing Part II: Justifying the Intended Uses of Test Scores Purposes of Testing Justification of Test Score Use A Foundation for Justifications of Intended Test Score Use Sources of Evidence for Justifying an Intended Test Score Use Evidence Based on Consequences of Testing Intended and Unintended Consequences Consequences of Classification Errors Conclusions about Consequences Evidence Based on Costs of Testing Cost-Benefit Analyses Cost-Effectiveness Analyses Cost-Utility Analyses Opportunity Costs Conclusions about Costs of Testing Evidence Based on Alternatives to Testing Evidence Based on Fairness in Testing. Stakeholder Input Opportunity to Learn Due Notice Disparate Impact Summary of Sources of Evidence for Justification of Test Use Comparing Validation and Justification Dimension 1: Rationale Dimension 2: Timing Dimension 3: Focus Dimensions 4 and 5: Traditions and Warrants Dimension 6: Duration Dimension 7: Responsibility Critical Commonalities Some Common Threats to Confident Use of Test Scores The Comprehensive Framework for Defensible Testing A Note on Consequences of Test Use as a Source of Validity Evidence Conclusions 6 How Much Is Enough? How Much Evidence is Enough to Support an Intended Test Score Meaning? 1. The Purposes of Testing 2. Quantity vs. Quantity 3. Quantity vs. Quality 4. Resources 5. Burden Conclusions about Validation How Much Evidence is Enough to Support an Intended Test Use? Validity Evidence Resources and Burden Input Need Consequences of Use Commonalities and Conclusions 7 Conclusions and Future Directions A Comprehensive Approach to Defensible Testing The Benefits of a Comprehensive Approach Future Research and Development in Validity Theory and Practice Justification of Test Use Validity and Classroom Assessment Group vs. Individual Validity Validation, Justification, and Theories of Action Toward a Truly Comprehensive Framework References Index