This articile summarizes findings from a QA review of published results for the National Evaluation of Upward Bound conducted for the US Department of Education by Mathematica Policy Research, the final report for which was published in 2009. The article also presents a summary of re-analysis results. The Mathematica impact estimates were found to suffer from inadequately controlled for bias in favor of the control group, and a sample with serious representational flaws and unequal weighting issues. In the interest of full government transparency this article is also written as an effort to comply with professional evaluation standards that state:
The formal parties to an evaluation should ensure that the full set of evaluation findings along with pertinent limitations are made accessible to the persons affected by the evaluation and any others with expressed legal rights to receive the results, (Standard P6 in Joint Committee on Education Evaluation Standards).
The re-analysis found that when identified study flaws are addressed using standard statistical procedures there are positive impacts for the UB program on the goals of the program—postsecondary entrance, award of financial aid, and degree or credential attainment. The paper is a summary of results published by the Council on Opportunity in Education (COE) last fall in a report entitled Addressing Study Error in the Random Assignment National Evaluation of Upward Bound: Do the Conclusions Change? (http://www.coenet.us/files/files-do_the_Conclusions_Change_2009.pdf.) (referred to here as the “re-analysis report”).