Importance of Studying the Selection Process

Cook, Shadish & Wong (2008) review within-study comparisons to identify the conditions under which observational studies can produce estimates comparable to experiments. It is a great article. One thing it made me realize is that my proposed study is a within-study comparison ... but without the comparison to an experimental design.

More importantly, however, is the finding that "[k]nowledge of the selection process can significantly reduce selection bias provided the selection process is valid and reliably measures" (p. 740). This is exactly what I am trying to investigate when it comes to measuring the effect of algebra vs. pre-algebra in 8th grade: what is the selection process and how do the available data accurately account for that process? Until we can reasonably understand the answer to this question, it's difficult to understand how biased the current course-taking studies are.

---

Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, Vol. 27, No. 4, 724-750.