Hiatus

Not surprisingly, I have not been able to keep up this blog and a glance at my current to-do list suggest that won't change any time soon.

So in the meantime, here are some blogs and websites to follow the causal inference chatter:

• Andrew Gelman's blog: statistical modeling, causal inference, and social science
• Judea Pearl's blog: causality
• The mostly harmless econometrics blog (Joshua Angrist)
• Christopher Winship's website for counterfactual analysis
• Stephen Morgan's website
• Gary King's website for causal inference
• Jasjeet Sekhon's website

Importance of Studying the Selection Process

Cook, Shadish & Wong (2008) review within-study comparisons to identify the conditions under which observational studies can produce estimates comparable to experiments. It is a great article. One thing it made me realize is that my proposed study is a within-study comparison ... but without the comparison to an experimental design.

More importantly, however, is the finding that "[k]nowledge of the selection process can significantly reduce selection bias provided the selection process is valid and reliably measures" (p. 740). This is exactly what I am trying to investigate when it comes to measuring the effect of algebra vs. pre-algebra in 8th grade: what is the selection process and how do the available data accurately account for that process? Until we can reasonably understand the answer to this question, it's difficult to understand how biased the current course-taking studies are.

---

Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, Vol. 27, No. 4, 724-750.

Choice and Potential Outcomes

I'm trying to read Manski's Identification Problems in the Social Sciences. In the section titled "Selection of the Treatment with the Larger/Smaller Outcome" he briefly talks about the Roy model of occupational choice (p. 45). The model states that a person selects the occupation that provides the higher wage for that individual. I believe the implication is that the observed wage for an individual is the highest potential wage outcome for that individual given multiple potential outcomes from different occupations.

Under this rational choice model, one could assume any observed outcome based on self-selection is greater, for a given individual, than the counterfactual outcome. I like the idea of starting an inquiry under the assumption that individuals do make rational decisions and seek out the most optimal outcomes available. I'd even like to think that selection decisions made on the behalf of others follow this model, say a school counselor selecting the math class that will maximize a student's academic success.

As with any traditional economic model, however, optimal choice requires perfect information for an individual to correctly gauge expected outcomes. I doubt the assumption of perfect information holds in most cases, and particularly not in the case when one individual (e.g., a counselor) is making selection decisions for multiple people (e.g., students).

Furthermore, economic models regarding selection and choice are more plausible when applied to individuals maximizing the more general concept of utility rather than a specific outcome like wages. For example, occupational choice is based on multiple factors that comprise an individuals utility including wages, hours, location, benefits, etc. Similarly, it's likely that course selection--even under the idealized Roy model--is based on multiple factors that comprise a student's academic utility including knowledge acquisition, motivation, etc. So any inquiry into causal effects that focus on a single outcome (e.g., wages or knowledge acquisition) may still find unobserved potential outcomes (the counterfactual) that exceed the observed outcome.

Not So Fast My Mathematically Inclined Friend

The Califonia Board of Education's ruling to test all 8th graders on Algebra was put on hold by the state Supreme Court.

Summary of Thoughts Over the Past Months

It's been a while since I posted, mostly because I have been preoccupied with little things like getting through a quarter of classes and life. But here's a summary of some relevent thoughts/events over the past few months.

• I received IRB approval for my study just before Thanksgiving without having to make any major changes to the proposal. (Once I figure out how to link to pdf documents I'll include a copy of the proposal.)

• Came across another study of the effects of algebra in middle school. Based on my skimming of the article, Xin Ma (2005) used the Longitudinal Study of American Youth (LSAY) data for the 1987 cohort of 7th graders and hierarchical linear growth modeling to examine growth in mathematics achievement over time. Ma found that mathematics achievement among low achieving middle school students who took algebra “grew not only faster than low achieving students who were not accelerated into formal algebra but also faster than high achieving students who were not accelerated into formal algebra” (pp. 452-453). Like the other studies that look at this issue, selection into algebra is not addressed and selection bias is still a concern.

• For one of my classes I started using the National Education Longitudinal Study (NELS), which follows a 1988 cohort of 8th graders, to examine the effects of algebra in 8th grade. The dataset is less than ideal because the base year data collection occures during the second semester of 8th grade. This means it lacks good pre-selection measures and threfore any analysis of selection or effects of algebra in 8th grade could be confounded.

• I discovered that the Early Childhood Longitudinal Study (ECLS), which follows a 1998 cohort of kindergarteners, final wave of data collection was spring of 2007 ... when the cohort should be in 8th grade (first look at results here). I'm hoping the public use files will be available some time soon because this could be the best national data source available to examine the factors associated with 8th grade algebra selection. 

---

Ma, X. (2005). Early Acceleration of Students in Mathematics: Does it Promote Growth and Stability of Growth in Achievement across Mathematical Areas? Contemporary Educational Psychology, 30, 439-460.

New Study on 8th Grade Math Placement Misses the Methodological Mark

The Brookings Institute released a new study this month that claims many students will suffer under a policy where all 8th graders are placed in Algebra (and the LA Times ran an article about it). The conclusion makes sense: placing low-achieving students in Algebra negatively affects them and may also negatively affect high-achieving students in the same class.

How the author of the report gets to the above conclusion, however, is riddled with problems. First, and most importantly, the author identifies the "misplaced" students as those who scored poorly on the 8th grade NAEP test. One well accepted rule for any inference about causation (such as misplacing students in Algebra causes them to do poorly in math) is that the cause must come before the effect. Yet, this study defines "misplacement" based on a test they take well into their 8th grade year, after placement and exposure to Algebra instruction. Under this tautology a misplaced student will always exhibit poor performance in 8th grade math. It's the same circular logic that leads people to conclude limited English proficient students always score poorly on English language arts tests ... if they didn't score poorly on those tests they wouldn't be classified as limited English proficient.

It's plausible that the measure of "misplacement" in this study is in fact measuring the effect of placement and not the cause. The author notes that "misplaced" students were more likely to have teachers with less experience and education. Assuming these teacher characteristics are associated with lower quality instruction, it's not surprising that students in classrooms with poor instruction would exhibit lower math proficiency. But this logic makes poor instruction, not poor placement, the cause of poor math performance.

Even if the the study used NAEP performance prior to 8th grade as the measure of "misplacement," the validity of this measure is still questionable. Who's to say NAEP performance is an accurate measure of who should take Algebra and who should take pre-Algebra? Perhaps other assessments, mathematics grades, or teacher recommendations provide better measures of Algebra "readiness."

There's another nagging problem with the study. It compares average NAEP performance among "misplaced" students to average NAEP performance among 4th graders to claim that the misplaced students do not even have 4th grade math skills. From what I understand of the NAEP scale scores, this is an invalid use of the scores. The NAEP tests are based on grade-level standards and scores are scaled within-grade and are not meant for comparisons across grades. If 4th graders have an average NAEP score of 238 and the average NAEP score for "misplaced" 8th graders is 211, it does not mean the 8th grade students know less math than the typical 4th grade student. The 8th graders are taking an 8th grade math test and the 4th graders are taking a 4th grade math test.

It's good to know people are trying to empirically look into whether the new Algebra-for-All California policy will benefit or hurt students, but I wish they'd be a little more mindful of the difficulties involved in actually producing empirically-sound conclusions.

News Flash: "Children Willing to Consume Gummy Bear Snacks Daily"

My 2 year-old is mildly obsessed with gummy bears. So when I came across a report titled Xylitol gummy bear snacks: a school-based randomized clinical trial I had to skim it. Given my experience with children and gummy bears, I was not really taken aback by the following finding: "Parents are accepting and children willing to consume gummy bear snacks daily."

But I hope some day I can publish a paper with a figure as great as their Figure 1: