New Study on 8th Grade Math Placement Misses the Methodological Mark

The Brookings Institute released a new study this month that claims many students will suffer under a policy where all 8th graders are placed in Algebra (and the LA Times ran an article about it). The conclusion makes sense: placing low-achieving students in Algebra negatively affects them and may also negatively affect high-achieving students in the same class.

How the author of the report gets to the above conclusion, however, is riddled with problems. First, and most importantly, the author identifies the "misplaced" students as those who scored poorly on the 8th grade NAEP test. One well accepted rule for any inference about causation (such as misplacing students in Algebra causes them to do poorly in math) is that the cause must come before the effect. Yet, this study defines "misplacement" based on a test they take well into their 8th grade year, after placement and exposure to Algebra instruction. Under this tautology a misplaced student will always exhibit poor performance in 8th grade math. It's the same circular logic that leads people to conclude limited English proficient students always score poorly on English language arts tests ... if they didn't score poorly on those tests they wouldn't be classified as limited English proficient.

It's plausible that the measure of "misplacement" in this study is in fact measuring the effect of placement and not the cause. The author notes that "misplaced" students were more likely to have teachers with less experience and education. Assuming these teacher characteristics are associated with lower quality instruction, it's not surprising that students in classrooms with poor instruction would exhibit lower math proficiency. But this logic makes poor instruction, not poor placement, the cause of poor math performance.

Even if the the study used NAEP performance prior to 8th grade as the measure of "misplacement," the validity of this measure is still questionable. Who's to say NAEP performance is an accurate measure of who should take Algebra and who should take pre-Algebra? Perhaps other assessments, mathematics grades, or teacher recommendations provide better measures of Algebra "readiness."

There's another nagging problem with the study. It compares average NAEP performance among "misplaced" students to average NAEP performance among 4th graders to claim that the misplaced students do not even have 4th grade math skills. From what I understand of the NAEP scale scores, this is an invalid use of the scores. The NAEP tests are based on grade-level standards and scores are scaled within-grade and are not meant for comparisons across grades. If 4th graders have an average NAEP score of 238 and the average NAEP score for "misplaced" 8th graders is 211, it does not mean the 8th grade students know less math than the typical 4th grade student. The 8th graders are taking an 8th grade math test and the 4th graders are taking a 4th grade math test.

It's good to know people are trying to empirically look into whether the new Algebra-for-All California policy will benefit or hurt students, but I wish they'd be a little more mindful of the difficulties involved in actually producing empirically-sound conclusions.