Throwing the Baby Out With the Bathwater? The What Works Clearinghouse Criteria for Group Equivalence

Abstract

The What Works Clearinghouse (WWC) provides summary reports of the effectiveness of educational programs and practices, but only a small proportion of studies considered for review meet their selection criteria. A common reason for rejecting studies from consideration regards the WWC’s standard for equivalence of the intervention and comparison groups. This paper examines the criteria related to these decisions. Calculations based on the Central Limit Theorem illustrate how the probability of meeting the WWC criteria declines sharply when studies include multiple measures and/or comparisons (which is common in more sophisticated studies) and/or when sample sizes are smaller (which is common in highly controlled experimental designs). Descriptions of two welldesigned studies rejected for inclusion in recent WWC reports illustrate the nature of the problem. Implications and recommendations for policy change are discussed.