Importance of Studying the Selection Process

Cook, Shadish & Wong (2008) review within-study comparisons to identify the conditions under which observational studies can produce estimates comparable to experiments. It is a great article. One thing it made me realize is that my proposed study is a within-study comparison ... but without the comparison to an experimental design.

More importantly, however, is the finding that "[k]nowledge of the selection process can significantly reduce selection bias provided the selection process is valid and reliably measures" (p. 740). This is exactly what I am trying to investigate when it comes to measuring the effect of algebra vs. pre-algebra in 8th grade: what is the selection process and how do the available data accurately account for that process? Until we can reasonably understand the answer to this question, it's difficult to understand how biased the current course-taking studies are.

---

Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, Vol. 27, No. 4, 724-750.

Choice and Potential Outcomes

I'm trying to read Manski's Identification Problems in the Social Sciences. In the section titled "Selection of the Treatment with the Larger/Smaller Outcome" he briefly talks about the Roy model of occupational choice (p. 45). The model states that a person selects the occupation that provides the higher wage for that individual. I believe the implication is that the observed wage for an individual is the highest potential wage outcome for that individual given multiple potential outcomes from different occupations.

Under this rational choice model, one could assume any observed outcome based on self-selection is greater, for a given individual, than the counterfactual outcome. I like the idea of starting an inquiry under the assumption that individuals do make rational decisions and seek out the most optimal outcomes available. I'd even like to think that selection decisions made on the behalf of others follow this model, say a school counselor selecting the math class that will maximize a student's academic success.

As with any traditional economic model, however, optimal choice requires perfect information for an individual to correctly gauge expected outcomes. I doubt the assumption of perfect information holds in most cases, and particularly not in the case when one individual (e.g., a counselor) is making selection decisions for multiple people (e.g., students).

Furthermore, economic models regarding selection and choice are more plausible when applied to individuals maximizing the more general concept of utility rather than a specific outcome like wages. For example, occupational choice is based on multiple factors that comprise an individuals utility including wages, hours, location, benefits, etc. Similarly, it's likely that course selection--even under the idealized Roy model--is based on multiple factors that comprise a student's academic utility including knowledge acquisition, motivation, etc. So any inquiry into causal effects that focus on a single outcome (e.g., wages or knowledge acquisition) may still find unobserved potential outcomes (the counterfactual) that exceed the observed outcome.