The Methods Workshop took place on Wednesday, Jan. 22, 2014, from 2–5:30 p.m.
The objectives for the workshop included:
- Review with attendees advantages of experiments for causal inference, policy/program evaluation.
- Discuss data collection issues, including ethical considerations.
- Provide and discuss examples of experimental or RCT designs, basic analysis and interpretation of results.
Suleiman Asman, research manager at IPA-Kenya, provided an overview on RCTs and running experiments, including rationale for randomizing, the importance of developing a theory of change, and a step-by-step review of a completed RCT project.
Gisella Kagy, University of Colorado, described the application of “randomized controlled trials” to population-level research, noting how the conditions of randomization and control differ at the population-level from conditions in a laboratory or clinic. She described the design issues, noting some of the pitfalls, and used the Matlab site and the multi-generational effects of MCH/FP intervention to provide an example.
Willa Friedman, Center for Global Development, described the girls’ merit scholarship program (GSP) experiment in Kenya and the long-term effects of the scholarship program, focusing on the design of the observational study that followed the randomized experiment. She highlighted the importance of assessing the balance of the treatment and control groups across key covariates in follow-up and presented results with respect to impact on autonomy within the household.
Presentations:
- Why, When, and What of Evaluation – Suleiman Asman
- How to Randomize – Suleiman Asman
- Impact Evaluation: Why Randomize? – Suleiman Asman
- Implementing an Evaluation: Data Collection and Pitfalls – Suleiman Asman
- Multiple Outcomes and Inference: An Example from Analysis of Long-Term Effects of a Scholarship Program – Willa Friedman
- Randomized Controlled Trials in Population Level Research: An Example from a Quasi-Random Child Health and Family Planning Program in Bangladesh – Gisella Kagy
Required Readings:
- Case Study 1: Get Out the Vote
- Case Study 2: Learn to Read Evaluations
Resources:
- Willa Friedman et al., “Education As Liberation?” NBER Working Paper, No. w169392011 (April 2011).
- Donald Rubin, “For Objective Causal Analysis Design Trumps Analysis,” Annals of Applied Statistics 2, no. 3 (2008): 808-40.
- Angus Deaton vs. Abhijit Banerjee on RCTs.
- Martin Ravallion,“Fighting Poverty One Experiment at a Time: A Review of Abhijit Banerjee and Esther Duflo’s Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty,” Journal of Economic Literature 50, no. 1 (2012): 103-14.
- Angus Deaton, “Instruments of Development: Randomization in the Tropics and the Search for the Elusive Key to Economic Development,” NBER Working Paper No. 14690 (January 2009).
- Abhijit Banerjee and Esther Duflo, “The Experimental Approach to Development Economics,” Annual Review of Economics, Vol. 1: 151-178.
- SJL Edwards et al., “Ethical Issues in the Design and Conduct of Randomized Controlled Trials,” Health Technology Assessment 2, no. 15 (1998).
- Christine Paul, Robert Sanson-Fisher, and Mariko Carey, “Randomized Controlled Trials,” Oxford Bibliographies in Public Health, published online February 2011.