Causality in Policy Studies: a Pluralist Toolbox

Description

This volume provides a methodological toolbox for conducting policy research. Recognizing that policy research spans various academic disciplines, each of which takes a different view on causality, the volume introduces a methodologically pluralistic approach to policy studies. Each chapter clarifies the research question that each technique can answer, the research design and data treatment that each technique requires for its results to be sound, the validity domain of its results, and the actual deployment of the technique through a replicable example. Techniques covered include quasi-experimental designs, approaches to account for selection bias and observed imbalances, directed acyclic graphs and structural equation models, Qualitative Comparative Analysis, Bayesian case study and process tracing, and Agent-Based Modelling. By working through the volume, readers will understand how to learn from different techniques, apply them consciously, and triangulate them to make better sense of findings. This volume is intended for advanced academic courses, as well as scholars and practitioners in policy-related fields, such as political science, economics, sociology, and public administration. This is an open access book. Preface Acknowledgments Contents Chapter 1: Introduction: The Elephant of Causation and the Blind Sages 1.1 Policy Decisions and Causal Theories 1.2 The Elephant of Causation 1.2.1 Elephants by the Principle 1.2.2 Elephants by the Rules 1.2.2.1 Regularity 1.2.2.2 Counterfactual 1.3 The Blind Sages’ Portrayals as the Book’s Blueprint 1.3.1 Can this Single Factor Make Any Difference? 1.3.2 Through Which Structures? 1.3.3 Through Which Process? 1.3.4 Considerations and Extensions References Untitled Chapter 2: Causation in the Social Realm 2.1 Why Discuss the Ontology of Causation? 2.2 Scientific Realism About the Social World and Social Causation 2.2.1 Critical Realism 2.3 What Is Causation? 2.3.1 Causal Mechanisms 2.3.2 Causal Powers 2.3.3 Manipulability and Invariance 2.4 Pluralism About Causal Inquiry 2.4.1 Case Studies and Process Tracing 2.4.2 Quantitative Research Based on Observational Data 2.4.3 Randomized Controlled Trials and Quasi-experimental Research 2.4.4 Generative Models and Simulation Methods 2.5 Realism and Methodological Pluralism References Suggested Readings Chapter 3: Counterfactuals with Experimental and Quasi-Experimental Variation 3.1 Introduction 3.2 Causation and Counterfactual Impact Evaluation: The Jargon 3.2.1 Causes as Manipulable Treatments 3.2.2 Effects as Differences Between Factual and Counterfactual Outcomes 3.2.3 What the Data Tell (And When) 3.3 Shades of Validity 3.3.1 Internal Validity: The Ability to Make a Causal Claim from a Pattern Documented in the Data 3.3.2 Statistical Validity: Measuring Precisely the Relationship Between Causes and Outcomes in the Data 3.3.3 External Validity: The Ability to Extend Conclusions to a Larger Population, over Time and Across Contexts 3.4 Random Assignment Strengthens Internal Validity 3.5 Internally Valid Reasoning Without RCTs: Instrumental Variation 3.5.1 A Tale of Pervasive Manipulation 3.5.2 General Formulation of the Problem 3.5.3 Assumptions 3.5.3.1 The “Monotonicity” Assumption 3.5.3.2 The “As Good as Random” Assumption 3.5.3.3 The “Exclusion Restriction” 3.5.3.4 The “First-Stage” Requirement 3.5.4 Better LATE than Never 3.5.5 External Validity of Causal Conclusions 3.6 Causal Reasoning with Administrative Rules: The Case of Regression Discontinuity Designs 3.6.1 Larger Classes, Worse Outcomes? 3.6.2 Visual Interpretation 3.6.3 General Formulation of the Problem 3.6.3.1 The Sharp RD Design 3.6.3.2 The Fuzzy RD Design 3.6.4 Validating the Internal Validity of the Design 3.7 Conclusion References Suggested Readings Chapter 4: Correlation Is Not Causation, Yet… Matching and Weighting for Better Counterfactuals 4.1 Introduction 4.2 Not Just a Mantra: Correlation Is Not Causation Because… 4.2.1 Causal Inference Entails an Identification Problem 4.2.2 Each Identification Strategy Entails a Set of Assumptions 4.2.3 Last but not Least: Model Dependence 4.3 Preprocessing Data with Matching to Improve the Credibility of the Estimates 4.3.1 No Magic: What Matching Can and Cannot Do 4.3.2 Useful Starting Point: Exact Matching 4.3.3 Propensity Score Tautology 4.3.4 How to Choose Among Matching Procedures? 4.3.5 The End: The Parametric Outcome Analysis 4.4 Empirical Illustration 4.4.1 Entropy Balancing 4.4.2 Coarsened Exact Matching 4.5 Conclusion References Suggested Readings Chapter 5: Getting the Most Out of Surveys: Multilevel Regression and Poststratification 5.1 Introduction 5.2 How It Works 5.3 Running Example 5.3.1 Draw a Sample 5.3.1.1 Step 1: Fit a Model 5.3.1.2 Step 2: Construct the Poststratification Frame 5.3.1.3 Step 3: Predict and Poststratify 5.3.2 Beware Overfitting 5.3.3 Partial Pooling 5.3.4 Sample Size Is Critical 5.3.5 Stacked Regression and Poststratification (SRP) 5.3.6 Synthetic Poststratification 5.3.7 Best Performing 5.4 Conclusion References Chapter 6: Pathway Analysis, Causal Mediation, and the Identification of Causal Mechanisms 6.1 Introduction 6.2 Can Pathways Be Mechanisms? 6.3 Identifying Causal Mechanisms with Graphs 6.3.1 Closing the Backdoor 6.3.2 Closing the Front Door 6.4 Identifying Indirect Effects 6.4.1 Indirect Effect in Non-linear Systems 6.4.2 Indirect Effect When the Cause and the Mediator Interact 6.4.3 Wrapping Up 6.5 Applications 6.5.1 A Mechanistic View on the Worm Wars 6.5.2 A Mechanistic View on a Chicago School Reform 6.6 Thou Shall Not Raise Causal Illusions References Chapter 7: Testing Joint Sufficiency Twice: Explanatory Qualitative Comparative Analysis 7.1 Introduction 7.2 Interpretability 7.2.1 Mechanisms and Machines 7.2.2 Operationalizing Typological Theories 7.2.3 Assembling Configurational Hypotheses 7.3 Validity 7.3.1 QCA’s Algebra 7.3.1.1 Literals 7.3.1.2 Operators Negation Joint Occurrence Alternatives Necessity and Sufficiency 7.3.1.3 Truth Tables 7.3.2 Identifying Valid Inus Hypotheses 7.3.2.1 Rendering Hypotheses 7.3.2.2 Tackling Underspecification Decision 1: Frequency Cut-Off Decision 2: The Consistency Threshold Decision 3: The Coverage Cut-Off 7.3.2.3 Tackling Overspecification Irrelevant Components A Note on Ambiguity in Solutions Dealing with Trivial Factors 7.4 Soundness 7.4.1 Gauging for QCA: The Theoretical Side 7.4.1.1 The Starting Point 7.4.1.2 Ragin’s Reinvention 7.4.1.3 Fuzzy Sufficiency and Necessity 7.4.2 Gauging for QCA: The Empirical Side 7.4.2.1 Establishing the Universe of Reference 7.4.2.2 Operationalizing Intension Hyper-Specificity Hyper-Generality The Problem of Missing Values 7.4.2.3 Identifying Membership Thresholds 7.5 Summing Up References Suggested Readings Chapter 8: Causal Inference and Policy Evaluation from Case Studies Using Bayesian Process Tracing 8.1 Introduction 8.2 The Epistemic Foundations of Process Tracing 8.3 Process Tracing Best Practices and Examples from COVID Research 8.3.1 Definition of Process Tracing 8.3.2 How to Do Process Tracing 8.3.3 Best Practices in Process Tracing 8.3.3.1 Cast the Net Widely for Alternative Explanations 8.3.3.2 Be Equally Tough on the Alternative Explanations 8.3.3.3 Consider the Potential Biases of Evidentiary Sources 8.3.3.4 Consider Whether the Case Is Most or Least Likely for Alternative Explanations 8.3.3.5 Make a Justifiable Decision on When to Start 8.3.3.6 Be Relentless in Getting Diverse Evidence, but Make a Justifiable Decision on When to Stop 8.3.3.7 Combine PT with Case Comparisons if Relevant 8.3.3.8 Be Open to Inductive Insights 8.3.3.9 Use Deduction to Infer What Must Be True if a Hypothesis Is True 8.3.3.10 Remember Not All PT Is Conclusive 8.3.4 Examples from COVID Case Studies 8.4 The “Replication Crisis” and the Comparative Advantages of Process Tracing Case Studies 8.4.1 The Replication Crisis 8.4.2 Process Tracing on Complex Phenomena 8.4.3 Process Tracing in Multimethod Research 8.4.4 Process Tracing and Generalizing from Case Studies 8.4.5 Limitations of Process Tracing 8.5 New Developments in Process Tracing 8.5.1 Formal Bayesian Process Tracing 8.5.2 New Modes of Multimethod Research 8.6 Conclusions References Suggested Reading Chapter 9: Exploring Interventions on Social Outcomes with In Silico, Agent-Based Experiments 9.1 Introduction 9.2 Agent-Based Modeling 9.3 Exploring Artificial Policy Scenarios 9.3.1 Interventions to Increase Competition or Collaboration in Science 9.3.1.1 Example 1 9.3.1.2 Example 2 9.4 Conclusions References Chapter 10: The Many Threats from Mechanistic Heterogeneity That Can Spoil Multimethod Research 10.1 Introduction 10.2 Basic Ideas Behind MMR 10.3 The Problem of Mechanistic Heterogeneity for External Validity in MMR 10.4 Sources of Mechanistic Heterogeneity in MMR 10.4.1 Complex Concepts or Measures 10.4.2 Known and Unknown Omitted Conditions 10.4.3 Causal and Temporal Dynamics 10.5 Taking Mechanistic Heterogeneity in MMR More Seriously 10.6 Concluding Remarks References Chapter 11: Conclusions. Causality Between Plurality and Unity 11.1 Introduction 11.2 Two Tales About the Making of Science 11.2.1 The Viewpoint of the History of Science 11.2.2 The Perspective of the Philosophy of Science 11.3 Can We Learn from One Another? 11.3.1 Ontological Incommensurability? 11.3.2 Epistemic Incommensurability? 11.3.3 Methodological Incommensurability? 11.3.3.1 Design-Based Solutions 11.3.3.2 Model-Based Solutions 11.4 Wrapping Up and Looking Ahead References