Hiatus

Not surprisingly, I have not been able to keep up this blog and a glance at my current to-do list suggest that won't change any time soon.

So in the meantime, here are some blogs and websites to follow the causal inference chatter:

• Andrew Gelman's blog: statistical modeling, causal inference, and social science
• Judea Pearl's blog: causality
• The mostly harmless econometrics blog (Joshua Angrist)
• Christopher Winship's website for counterfactual analysis
• Stephen Morgan's website
• Gary King's website for causal inference
• Jasjeet Sekhon's website

Choice and Potential Outcomes

I'm trying to read Manski's Identification Problems in the Social Sciences. In the section titled "Selection of the Treatment with the Larger/Smaller Outcome" he briefly talks about the Roy model of occupational choice (p. 45). The model states that a person selects the occupation that provides the higher wage for that individual. I believe the implication is that the observed wage for an individual is the highest potential wage outcome for that individual given multiple potential outcomes from different occupations.

Under this rational choice model, one could assume any observed outcome based on self-selection is greater, for a given individual, than the counterfactual outcome. I like the idea of starting an inquiry under the assumption that individuals do make rational decisions and seek out the most optimal outcomes available. I'd even like to think that selection decisions made on the behalf of others follow this model, say a school counselor selecting the math class that will maximize a student's academic success.

As with any traditional economic model, however, optimal choice requires perfect information for an individual to correctly gauge expected outcomes. I doubt the assumption of perfect information holds in most cases, and particularly not in the case when one individual (e.g., a counselor) is making selection decisions for multiple people (e.g., students).

Furthermore, economic models regarding selection and choice are more plausible when applied to individuals maximizing the more general concept of utility rather than a specific outcome like wages. For example, occupational choice is based on multiple factors that comprise an individuals utility including wages, hours, location, benefits, etc. Similarly, it's likely that course selection--even under the idealized Roy model--is based on multiple factors that comprise a student's academic utility including knowledge acquisition, motivation, etc. So any inquiry into causal effects that focus on a single outcome (e.g., wages or knowledge acquisition) may still find unobserved potential outcomes (the counterfactual) that exceed the observed outcome.

New Study on 8th Grade Math Placement Misses the Methodological Mark

The Brookings Institute released a new study this month that claims many students will suffer under a policy where all 8th graders are placed in Algebra (and the LA Times ran an article about it). The conclusion makes sense: placing low-achieving students in Algebra negatively affects them and may also negatively affect high-achieving students in the same class.

How the author of the report gets to the above conclusion, however, is riddled with problems. First, and most importantly, the author identifies the "misplaced" students as those who scored poorly on the 8th grade NAEP test. One well accepted rule for any inference about causation (such as misplacing students in Algebra causes them to do poorly in math) is that the cause must come before the effect. Yet, this study defines "misplacement" based on a test they take well into their 8th grade year, after placement and exposure to Algebra instruction. Under this tautology a misplaced student will always exhibit poor performance in 8th grade math. It's the same circular logic that leads people to conclude limited English proficient students always score poorly on English language arts tests ... if they didn't score poorly on those tests they wouldn't be classified as limited English proficient.

It's plausible that the measure of "misplacement" in this study is in fact measuring the effect of placement and not the cause. The author notes that "misplaced" students were more likely to have teachers with less experience and education. Assuming these teacher characteristics are associated with lower quality instruction, it's not surprising that students in classrooms with poor instruction would exhibit lower math proficiency. But this logic makes poor instruction, not poor placement, the cause of poor math performance.

Even if the the study used NAEP performance prior to 8th grade as the measure of "misplacement," the validity of this measure is still questionable. Who's to say NAEP performance is an accurate measure of who should take Algebra and who should take pre-Algebra? Perhaps other assessments, mathematics grades, or teacher recommendations provide better measures of Algebra "readiness."

There's another nagging problem with the study. It compares average NAEP performance among "misplaced" students to average NAEP performance among 4th graders to claim that the misplaced students do not even have 4th grade math skills. From what I understand of the NAEP scale scores, this is an invalid use of the scores. The NAEP tests are based on grade-level standards and scores are scaled within-grade and are not meant for comparisons across grades. If 4th graders have an average NAEP score of 238 and the average NAEP score for "misplaced" 8th graders is 211, it does not mean the 8th grade students know less math than the typical 4th grade student. The 8th graders are taking an 8th grade math test and the 4th graders are taking a 4th grade math test.

It's good to know people are trying to empirically look into whether the new Algebra-for-All California policy will benefit or hurt students, but I wish they'd be a little more mindful of the difficulties involved in actually producing empirically-sound conclusions.

News Flash: "Children Willing to Consume Gummy Bear Snacks Daily"

My 2 year-old is mildly obsessed with gummy bears. So when I came across a report titled Xylitol gummy bear snacks: a school-based randomized clinical trial I had to skim it. Given my experience with children and gummy bears, I was not really taken aback by the following finding: "Parents are accepting and children willing to consume gummy bear snacks daily."

But I hope some day I can publish a paper with a figure as great as their Figure 1:

Are Parachutes Effective?

I was cleaning up some of my files and came across a 2003 article in the British Medical Journal (BMJ) I forgot about. It's a brilliant satire of randomized experiment dogmatism.

You can find the article here: http://www.bmj.com/cgi/content/full/327/7429/1459

Here's the conclusion:

As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.

---

Smith G. C., Pell J. P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: Systematic review of randomised controlled trials, BMJ 327:1459-1461.

A Case Study of Causal Inference for Multilevel Observational Data

To evaluate the effect of retaining students in kindergarten instead of promoting them to first grade Hong & Raudenbush (2005, 2006) use a methodology that incorporates principal stratification, propensity scores, and hierarchical modeling.

I imagine a similar methodology is the way to go to evaluate the effect of placing students in pre-algebra instead of algebra 1. If only I could understand what it means.

---

Hong, G. & Raudenbush, S. W. (2005). Effects of kindergarten retention policy on children’s cognitive growth in reading and mathematics. Educational Evaluation and Policy Analysis, 27 (3), 205-224.

Hong, G. & Raudenbush, S. W. (2005). Evaluating kindergarten retention policy: A cause study of causal inference for multilevel observational data. Journal of the American Statistical Association, 101, 901-910.

Smoking, Lung Cancer, and Course-taking

In a 1996 article in the Journal of the American Statistical Association, Mitchell H. Gail quotes the 1964 Smoking and Health Surgeon General's Report regarding causal relationships:

Statistical methods cannot establish proof of a causal relationship in an association. The causal significance of an association is a matter of judgement which goes beyond any statement of statistical probability. To judge or evaluate the causal significance of the association between the attribute or agent and the disease, or effect upon health, a number of criteria must be utilized, no one of which is an all-sufficient basis for judgement.

As Gail wrote, the Surgeon General's Report defined those criteria as:

• consistency of the association in study after study
• strength of the association
• temporal pattern with exposure preceding disease
• coherence of the causal hypothesis with the body of evidence

While there's much to add to that list of criteria and much to comment on, the underlying notion that identifying a causal relationship is "a matter of judgement" is a salient one. And a list of criteria to guide that judgement is a powerful tool for anybody thinking about drawing causal inferences from a study.

Some time down the road it might be useful to compare the causal relationship discussion in the Surgeon General's Report to classic Cambell & Stanley description of internal and external validity published the year before (1963).

---

Gail, M. H. (1996). "Statistics in Action," Journal of the American Statistical Association, Vol. 91, No. 433, pp. 1-13.

U.S. Department of Health, Education and Welfare, Public Health Services (1964), Smoking and Health; Report of the Advisory Committee to the Surgeon General of the Public Health Service, Public Health Service Publication No. 1103, Washington, DC: U.S. Government Printing Office.