Subjective Reflection on Objective Research

Hiatus

2009-07-30T16:35:00.000-07:00

Not surprisingly, I have not been able to keep up this blog and a glance at my current to-do list suggest that won't change any time soon.

So in the meantime, here are some blogs and websites to follow the causal inference chatter:

Andrew Gelman's blog: statistical modeling, causal inference, and social science
Judea Pearl's blog: causality
The mostly harmless econometrics blog (Joshua Angrist)
Christopher Winship's website for counterfactual analysis
Stephen Morgan's website
Gary King's website for causal inference
Jasjeet Sekhon's website

Importance of Studying the Selection Process

2009-01-04T22:14:00.000-08:00

Cook, Shadish & Wong (2008) review within-study comparisons to identify the conditions under which observational studies can produce estimates comparable to experiments. It is a great article. One thing it made me realize is that my proposed study is a within-study comparison ... but without the comparison to an experimental design.

More importantly, however, is the finding that "[k]nowledge of the selection process can significantly reduce selection bias provided the selection process is valid and reliably measures" (p. 740). This is exactly what I am trying to investigate when it comes to measuring the effect of algebra vs. pre-algebra in 8th grade: what is the selection process and how do the available data accurately account for that process? Until we can reasonably understand the answer to this question, it's difficult to understand how biased the current course-taking studies are.

---

Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, Vol. 27, No. 4, 724-750.

Choice and Potential Outcomes

2009-01-04T17:44:00.000-08:00

I'm trying to read Manski's Identification Problems in the Social Sciences. In the section titled "Selection of the Treatment with the Larger/Smaller Outcome" he briefly talks about the Roy model of occupational choice (p. 45). The model states that a person selects the occupation that provides the higher wage for that individual. I believe the implication is that the observed wage for an individual is the highest potential wage outcome for that individual given multiple potential outcomes from different occupations.

Under this rational choice model, one could assume any observed outcome based on self-selection is greater, for a given individual, than the counterfactual outcome. I like the idea of starting an inquiry under the assumption that individuals do make rational decisions and seek out the most optimal outcomes available. I'd even like to think that selection decisions made on the behalf of others follow this model, say a school counselor selecting the math class that will maximize a student's academic success.

As with any traditional economic model, however, optimal choice requires perfect information for an individual to correctly gauge expected outcomes. I doubt the assumption of perfect information holds in most cases, and particularly not in the case when one individual (e.g., a counselor) is making selection decisions for multiple people (e.g., students).

Furthermore, economic models regarding selection and choice are more plausible when applied to individuals maximizing the more general concept of utility rather than a specific outcome like wages. For example, occupational choice is based on multiple factors that comprise an individuals utility including wages, hours, location, benefits, etc. Similarly, it's likely that course selection--even under the idealized Roy model--is based on multiple factors that comprise a student's academic utility including knowledge acquisition, motivation, etc. So any inquiry into causal effects that focus on a single outcome (e.g., wages or knowledge acquisition) may still find unobserved potential outcomes (the counterfactual) that exceed the observed outcome.

Not So Fast My Mathematically Inclined Friend

2008-12-20T08:47:00.000-08:00

The Califonia Board of Education's ruling to test all 8th graders on Algebra was put on hold by the state Supreme Court.

Summary of Thoughts Over the Past Months

2008-12-02T06:29:00.000-08:00

It's been a while since I posted, mostly because I have been preoccupied with little things like getting through a quarter of classes and life. But here's a summary of some relevent thoughts/events over the past few months.

I received IRB approval for my study just before Thanksgiving without having to make any major changes to the proposal. (Once I figure out how to link to pdf documents I'll include a copy of the proposal.)

Came across another study of the effects of algebra in middle school. Based on my skimming of the article, Xin Ma (2005) used the Longitudinal Study of American Youth (LSAY) data for the 1987 cohort of 7th graders and hierarchical linear growth modeling to examine growth in mathematics achievement over time. Ma found that mathematics achievement among low achieving middle school students who took algebra “grew not only faster than low achieving students who were not accelerated into formal algebra but also faster than high achieving students who were not accelerated into formal algebra” (pp. 452-453). Like the other studies that look at this issue, selection into algebra is not addressed and selection bias is still a concern.

For one of my classes I started using the National Education Longitudinal Study (NELS), which follows a 1988 cohort of 8th graders, to examine the effects of algebra in 8th grade. The dataset is less than ideal because the base year data collection occures during the second semester of 8th grade. This means it lacks good pre-selection measures and threfore any analysis of selection or effects of algebra in 8th grade could be confounded.

I discovered that the Early Childhood Longitudinal Study (ECLS), which follows a 1998 cohort of kindergarteners, final wave of data collection was spring of 2007 ... when the cohort should be in 8th grade (first look at results here). I'm hoping the public use files will be available some time soon because this could be the best national data source available to examine the factors associated with 8th grade algebra selection.

---

Ma, X. (2005). Early Acceleration of Students in Mathematics: Does it Promote Growth and Stability of Growth in Achievement across Mathematical Areas? Contemporary Educational Psychology, 30, 439-460.

New Study on 8th Grade Math Placement Misses the Methodological Mark

2008-09-23T16:52:00.000-07:00

The Brookings Institute released a new study this month that claims many students will suffer under a policy where all 8th graders are placed in Algebra (and the LA Times ran an article about it). The conclusion makes sense: placing low-achieving students in Algebra negatively affects them and may also negatively affect high-achieving students in the same class.

How the author of the report gets to the above conclusion, however, is riddled with problems. First, and most importantly, the author identifies the "misplaced" students as those who scored poorly on the 8th grade NAEP test. One well accepted rule for any inference about causation (such as misplacing students in Algebra causes them to do poorly in math) is that the cause must come before the effect. Yet, this study defines "misplacement" based on a test they take well into their 8th grade year, after placement and exposure to Algebra instruction. Under this tautology a misplaced student will always exhibit poor performance in 8th grade math. It's the same circular logic that leads people to conclude limited English proficient students always score poorly on English language arts tests ... if they didn't score poorly on those tests they wouldn't be classified as limited English proficient.

It's plausible that the measure of "misplacement" in this study is in fact measuring the effect of placement and not the cause. The author notes that "misplaced" students were more likely to have teachers with less experience and education. Assuming these teacher characteristics are associated with lower quality instruction, it's not surprising that students in classrooms with poor instruction would exhibit lower math proficiency. But this logic makes poor instruction, not poor placement, the cause of poor math performance.

Even if the the study used NAEP performance prior to 8th grade as the measure of "misplacement," the validity of this measure is still questionable. Who's to say NAEP performance is an accurate measure of who should take Algebra and who should take pre-Algebra? Perhaps other assessments, mathematics grades, or teacher recommendations provide better measures of Algebra "readiness."

There's another nagging problem with the study. It compares average NAEP performance among "misplaced" students to average NAEP performance among 4th graders to claim that the misplaced students do not even have 4th grade math skills. From what I understand of the NAEP scale scores, this is an invalid use of the scores. The NAEP tests are based on grade-level standards and scores are scaled within-grade and are not meant for comparisons across grades. If 4th graders have an average NAEP score of 238 and the average NAEP score for "misplaced" 8th graders is 211, it does not mean the 8th grade students know less math than the typical 4th grade student. The 8th graders are taking an 8th grade math test and the 4th graders are taking a 4th grade math test.

It's good to know people are trying to empirically look into whether the new Algebra-for-All California policy will benefit or hurt students, but I wish they'd be a little more mindful of the difficulties involved in actually producing empirically-sound conclusions.

News Flash: "Children Willing to Consume Gummy Bear Snacks Daily"

2008-08-18T13:46:00.000-07:00

My 2 year-old is mildly obsessed with gummy bears. So when I came across a report titled Xylitol gummy bear snacks: a school-based randomized clinical trial I had to skim it. Given my experience with children and gummy bears, I was not really taken aback by the following finding: "Parents are accepting and children willing to consume gummy bear snacks daily."

But I hope some day I can publish a paper with a figure as great as their Figure 1:

More on the California Algebra 1 Testing Requirement

2008-08-14T15:02:00.001-07:00

The California Department of Education (CDE) set up a webpage with information on the new Algebra 1 testing requirement for 8th graders. It includes press releases, op-ed pieces, a research report, and more.

Coincidentally (?), the CDE just released the 2008 state testing results. Over 210,000 California 8th graders took the General Math CST in 2008 (meaning they were in a pre-Algebra course in 8th grade) and just under 250,000 took the Algebra 1 test. Of those who took the General Math test, only 31% scored proficient or advanced and 41% scored below basic or far below basic.

Would they have been better off taking Algebra 1 instead?

Policy Precedes (my) Research

2008-07-10T15:14:00.000-07:00

This week the California State Board of Education (SBE) voted to require all 8th graders get tested in algebra (L.A. Times article here). Currently 8th graders can take either the Algebra 1 test or the General Math test (geared to "pre-algebra" standards) depending on whether they actually took Algebra 1 or some pre-algebra course in 8th grade. So the SBE decision essentially forces all 8th graders into Algebra 1 whether they are ready for the course or not.

Proponents of algebra for 8th graders point to benefits from early access to algebra and issues of equality. Opponents argue that many 8th graders are not prepared to succeed in algebra and are being set up for failure.

This exemplifies the timing disconnect between research and policy. The big picture purpose behind my proposed research is to shed some quality empirical evidence on this debate. Do students, particularly low achieving students, have better high school success if they take algebra or pre-algebra in 8th grade? Well it looks like the decision makers went ahead and presumed an answer to that question before I got a chance to answer it ... but that doesn't mean I can't test their presumptive answer.

Methodological side note: will this decision influence responses from school staff if asked how they currently decide to place a student in algebra or pre-algebra? My plan was to interview school staff to figure out how they go through the decision making process for different types of students but will the fact that such a decision is now defunct limit their honesty?

Are Parachutes Effective?

2008-06-20T13:28:00.000-07:00

I was cleaning up some of my files and came across a 2003 article in the British Medical Journal (BMJ) I forgot about. It's a brilliant satire of randomized experiment dogmatism.

You can find the article here: http://www.bmj.com/cgi/content/full/327/7429/1459

Here's the conclusion:

As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.

---

Smith G. C., Pell J. P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: Systematic review of randomised controlled trials, BMJ 327:1459-1461.

The Art of Science

2008-06-09T11:48:00.000-07:00

Social science research is often criticized for not being as rigorous as "real" scientific research and there is a strong call, particularly in education, for "scientific-based" decisions and programs. The general perception is that practitioners in fields like medicine make decisions driven by scientific evidence while practitioners in fields like education make decisions driven by personal and anecdotal experience.

So when I heard my brother's medical school graduating class recite the Hippocratic oath a few weeks back I was somewhat surprised. The oath contains multiple references to practicing the "art" of medicine but no reference to the "science."

A Case Study of Causal Inference for Multilevel Observational Data

2008-06-09T11:36:00.000-07:00

To evaluate the effect of retaining students in kindergarten instead of promoting them to first grade Hong & Raudenbush (2005, 2006) use a methodology that incorporates principal stratification, propensity scores, and hierarchical modeling.

I imagine a similar methodology is the way to go to evaluate the effect of placing students in pre-algebra instead of algebra 1. If only I could understand what it means.

---

Hong, G. & Raudenbush, S. W. (2005). Effects of kindergarten retention policy on children’s cognitive growth in reading and mathematics. Educational Evaluation and Policy Analysis, 27 (3), 205-224.

Hong, G. & Raudenbush, S. W. (2005). Evaluating kindergarten retention policy: A cause study of causal inference for multilevel observational data. Journal of the American Statistical Association, 101, 901-910.

Smoking, Lung Cancer, and Course-taking

2008-05-10T22:13:00.000-07:00

In a 1996 article in the Journal of the American Statistical Association, Mitchell H. Gail quotes the 1964 Smoking and Health Surgeon General's Report regarding causal relationships:

Statistical methods cannot establish proof of a causal relationship in an association. The causal significance of an association is a matter of judgement which goes beyond any statement of statistical probability. To judge or evaluate the causal significance of the association between the attribute or agent and the disease, or effect upon health, a number of criteria must be utilized, no one of which is an all-sufficient basis for judgement.

As Gail wrote, the Surgeon General's Report defined those criteria as:

consistency of the association in study after study
strength of the association
temporal pattern with exposure preceding disease
coherence of the causal hypothesis with the body of evidence

While there's much to add to that list of criteria and much to comment on, the underlying notion that identifying a causal relationship is "a matter of judgement" is a salient one. And a list of criteria to guide that judgement is a powerful tool for anybody thinking about drawing causal inferences from a study.

Some time down the road it might be useful to compare the causal relationship discussion in the Surgeon General's Report to classic Cambell & Stanley description of internal and external validity published the year before (1963).

---

Gail, M. H. (1996). "Statistics in Action," Journal of the American Statistical Association, Vol. 91, No. 433, pp. 1-13.

U.S. Department of Health, Education and Welfare, Public Health Services (1964), Smoking and Health; Report of the Advisory Committee to the Surgeon General of the Public Health Service, Public Health Service Publication No. 1103, Washington, DC: U.S. Government Printing Office.

Reflective Practice

2008-05-10T20:37:00.000-07:00

A chapter by Michael Seltzer and Mike Rose in the Sage Handbook for Research in Education (2006) keyed my interest in formally reflecting and documenting the subjective aspects of the research process.

In that chapter Seltzer and Rose speak about reflective practice as "thinking about what we are doing, why we are doing it, what might be flawed about it, and how best to convey the important aspects of the process to others" (p. 477). They then go on to discuss:

the importance of context
attending to alternative explanations
the importance of getting close to the data

In regards to attending to alternative explanations is the "need to try to understand the selection process by which individuals wind up in the different groups that we wish to compare" (p. 487). Hence justification for investigating the policies and process schools utilize to place 8th graders in Algebra 1 or pre-Algebra.

---

Seltzer, M. & Rose, M. (2006). Constructing Analyses: The development of thoughtfulness in working with quantitative methods, in Conrad, C. F. & Serlin, R. C., Eds. The Sage Handbook for Research in Education. Thousand Oaks, CA: Sage Publications.

Developing a Proposal

2008-05-06T15:37:00.000-07:00

I've spent some time thinking about and developing a proposed research project. Now it's time to develop these thoughts into a formal proposal to receive IRB approval and get this thing off the ground.

But first, let's figure out where things stand. Here's a snippet of my thinking for the study ...

Title. Assessing the Implications of Different Quasi-Experimental Methods through the Impact of Different Course-Taking Pathways

Motivation. By the time students enter high school they are placed on different course-taking pathways. Long-term academic success for the student is typically the stated reason for differentiation in course-taking patterns. For example, a student may be placed in a pre-Algebra course instead of Algebra 1 if the school determines s/he does not have the appropriate mathematics foundation to successfully complete Algebra 1. Differentiated placement, however, raises questions of equal access to educational opportunities and may ultimately harm a student’s chances of completing the coursework necessary for high school graduation and college enrollment.

The short- and long-term impacts of differentiated course-taking are difficult to measure, however; primarily because one cannot simply randomly assign students to different pathways and various threats to internal validity arise. In the absence of random assignment, our ability to make valid causal inference about course-taking pathways hinges on our ability to develop sound and robust quasi-experimental designs.

Purpose. An investigation into the impacts of differentiated course-taking pathways provides an opportunity to address two critical issues: one methodologically relevant and one policy relevant. Methodologically, the goal is to identify some key implications for causal inferences in quasi-experimental research. I plan to focus the methods inquiry on the ability to account for initial differences between comparison groups (i.e., treatment and control groups) by better understanding the selection process. Operationally, I plan to focus the study on differences in high school progress and performance between students who took Algebra 1 in 8th grade and students who took "pre-Algebra" in 8th grade. The goal is to determine whether placing the average 8th grade student in Algebra 1 results in positive outcomes in high school relative to placing the student in pre-Algebra.

Design. I envision a general research design centered around a quantitative, quasi-experimental approach. Within the broad quasi-experimental framework, however, I plan to compare different methods and statistical procedures to estimate the counterfactual outcomes of students placed in pre-Algebra versus Algebra 1. For example, how do the findings differ if I use propensity score matching versus regression modeling? How do the findings differ if I use hierarchical linear modeling? I hope to acquire student-level data from an urban school district for a cohort of students who entered 9th grade in 2003 (graduating class of 2008).

While quantitative methods will drive the primary analyses, I plan to take a reflective, and more qualitative, approach to assess how the different procedures produce varying degrees of causal inference about the impact of differentiated course-taking. One question is whether a specific method might result in findings (descriptive and causal) that facilitate “qualification” of the data (e.g., a description of the differentiation process and/or a student’s progress through high school). For example, the propensity score approach might allow for a better understanding of the group selection process, but a path analysis approach might allow for a better understanding of direct and indirect effects as a student progresses through high school. Another line of investigation is whether the use of multiple quasi-experimental methods allows one to, in a qualitative sense, triangulate findings to improve causal inferences. Can one use multiple quasi-experimental and/or quantitative methods to offset the limitations of a single method, as one would try to do in a quantitative-qualitative mixed methods design?

Lastly, but perhaps most importantly, I plan to address the resulting information gaps, such as context, and how other methods (i.e., qualitative methods) could potentially fill those gaps to improve causal inferences. In particular, I will interview middle school administrators to identify subtle, yet key, factors that influence course placement. The interviews will primarily include open-ended questions about school policies and individual practices for assigning students to Algebra 1 or pre-Algebra. At the very least, the interviews will provide information on the types of important selection process data omitted from quantitative studies of course-taking pathways. I can then examine the sensitivity of my quasi-experimental results to omitted, confounding factors identified in the interviews.

Theorem #1: Productive Procrastination is not an Oxymoron

2008-05-06T15:15:00.000-07:00

Much like a research project needs a purpose, goals and objectives, so too does this blog. While procrastination is a worthy goal in itself (see previous post), it's time to think big.

Theorem #1: productive procrastination is not an oxymoron. Under this theory, a blog can simultaneously serve as an opportunity to procrastinate and a medium to compose a research project. As such, here are a few productive objectives for this blog:

brainstorm thoughts and reflections related to the research project at hand as they arise
document the subjective decisions behind a seemingly objective research project (hence the title of this blog)
document the research process from the first-person perspective of the researcher, similar to the way an ethnographer would keep a field journal during periods of observation

So let's start with that and see how things progress.

Procrastination: The First Law of Research

2008-05-04T22:08:00.000-07:00

The first step in any research project is to procrastinate. As a first year student in the Social Research Methodology (SRM) division at the UCLA Graduate School of Education & Information Science, it's a shame it has taken me this long to reach this first and crucial stage.

Students in our division are expected (i.e., required) to complete an independent research project some time within the first two or three years of the doctoral program, and definately prior to jumping into a full-blown dissertation. It's akin to a master's thesis but those with the proper institutionalized culture refer to it as the 299 project -- based on the 299 course sequence students take to prepare for the project.

It's now spring (not technically but academically), less than two months until the end of my first year, and it's time to plan for this research project. So the natural thing to do is to start a blog ...