Discussion Papers 2022

CIRJE-F-1194	"Still Biased? A Remaining Classical Selection Problem of RCTs in Education"
Author Name	Kawarazaki, Hikaru, Minhaj Mahmud, Yasuyuki Sawada, Mai Seki, and Kazuma Takakura
Date	May 2022
Full Paper	PDF file
Remarks

Abstract
In the already very rich and crowded literature on education interventions, the use of test scores to capture students’cognitive abilities has been the norm when measuring the impact. We show that even in randomized controlled trials (RCTs), estimated treatment effects on the true latent abilities can still be biased towards zero, because test scores are often censored outside of zero and full marks. This paper employs sui generis data from a field experiment in Bangladesh as well as data sets from existing highly-cited studies in developing countries to illustrate theoretically and empirically that this remaining classical sample selection problem exists. We suggest three concrete ways to correct such bias: First, to employ the conventional sample selection correction methods; second, to use tests that are designed with an extensive set of questions from easy to challenging levels which allow students to answer the maximum they could; and third, to incorporate each student’s completion time in estimation.
Keywords:

Abstract

In the already very rich and crowded literature on education interventions, the use of test scores to capture students’cognitive abilities has been the norm when measuring the impact. We show that even in randomized controlled trials (RCTs), estimated treatment effects on the true latent abilities can still be biased towards zero, because test scores are often censored outside of zero and full marks. This paper employs sui generis data from a field experiment in Bangladesh as well as data sets from existing highly-cited studies in developing countries to illustrate theoretically and empirically that this remaining classical sample selection problem exists. We suggest three concrete ways to correct such bias: First, to employ the conventional sample selection correction methods; second, to use tests that are designed with an extensive set of questions from easy to challenging levels which allow students to answer the maximum they could; and third, to incorporate each student’s completion time in estimation.

Keywords: