Smoking, Lung Cancer, and Course-taking

In a 1996 article in the Journal of the American Statistical Association, Mitchell H. Gail quotes the 1964 Smoking and Health Surgeon General's Report regarding causal relationships:

Statistical methods cannot establish proof of a causal relationship in an association. The causal significance of an association is a matter of judgement which goes beyond any statement of statistical probability. To judge or evaluate the causal significance of the association between the attribute or agent and the disease, or effect upon health, a number of criteria must be utilized, no one of which is an all-sufficient basis for judgement.

As Gail wrote, the Surgeon General's Report defined those criteria as:

• consistency of the association in study after study
• strength of the association
• temporal pattern with exposure preceding disease
• coherence of the causal hypothesis with the body of evidence

While there's much to add to that list of criteria and much to comment on, the underlying notion that identifying a causal relationship is "a matter of judgement" is a salient one. And a list of criteria to guide that judgement is a powerful tool for anybody thinking about drawing causal inferences from a study.

Some time down the road it might be useful to compare the causal relationship discussion in the Surgeon General's Report to classic Cambell & Stanley description of internal and external validity published the year before (1963).

---

Gail, M. H. (1996). "Statistics in Action," Journal of the American Statistical Association, Vol. 91, No. 433, pp. 1-13.

U.S. Department of Health, Education and Welfare, Public Health Services (1964), Smoking and Health; Report of the Advisory Committee to the Surgeon General of the Public Health Service, Public Health Service Publication No. 1103, Washington, DC: U.S. Government Printing Office.

Reflective Practice

A chapter by Michael Seltzer and Mike Rose in the Sage Handbook for Research in Education (2006) keyed my interest in formally reflecting and documenting the subjective aspects of the research process.

In that chapter Seltzer and Rose speak about reflective practice as "thinking about what we are doing, why we are doing it, what might be flawed about it, and how best to convey the important aspects of the process to others" (p. 477). They then go on to discuss:

• the importance of context
• attending to alternative explanations
• the importance of getting close to the data

In regards to attending to alternative explanations is the "need to try to understand the selection process by which individuals wind up in the different groups that we wish to compare" (p. 487). Hence justification for investigating the policies and process schools utilize to place 8th graders in Algebra 1 or pre-Algebra.

---

Seltzer, M. & Rose, M. (2006). Constructing Analyses: The development of thoughtfulness in working with quantitative methods, in Conrad, C. F. & Serlin, R. C., Eds. The Sage Handbook for Research in Education. Thousand Oaks, CA: Sage Publications.

Developing a Proposal

I've spent some time thinking about and developing a proposed research project. Now it's time to develop these thoughts into a formal proposal to receive IRB approval and get this thing off the ground.

But first, let's figure out where things stand. Here's a snippet of my thinking for the study ...

Title. Assessing the Implications of Different Quasi-Experimental Methods through the Impact of Different Course-Taking Pathways

Motivation. By the time students enter high school they are placed on different course-taking pathways. Long-term academic success for the student is typically the stated reason for differentiation in course-taking patterns. For example, a student may be placed in a pre-Algebra course instead of Algebra 1 if the school determines s/he does not have the appropriate mathematics foundation to successfully complete Algebra 1. Differentiated placement, however, raises questions of equal access to educational opportunities and may ultimately harm a student’s chances of completing the coursework necessary for high school graduation and college enrollment.

The short- and long-term impacts of differentiated course-taking are difficult to measure, however; primarily because one cannot simply randomly assign students to different pathways and various threats to internal validity arise. In the absence of random assignment, our ability to make valid causal inference about course-taking pathways hinges on our ability to develop sound and robust quasi-experimental designs.

Purpose. An investigation into the impacts of differentiated course-taking pathways provides an opportunity to address two critical issues: one methodologically relevant and one policy relevant. Methodologically, the goal is to identify some key implications for causal inferences in quasi-experimental research. I plan to focus the methods inquiry on the ability to account for initial differences between comparison groups (i.e., treatment and control groups) by better understanding the selection process. Operationally, I plan to focus the study on differences in high school progress and performance between students who took Algebra 1 in 8th grade and students who took "pre-Algebra" in 8th grade. The goal is to determine whether placing the average 8th grade student in Algebra 1 results in positive outcomes in high school relative to placing the student in pre-Algebra.

Design. I envision a general research design centered around a quantitative, quasi-experimental approach. Within the broad quasi-experimental framework, however, I plan to compare different methods and statistical procedures to estimate the counterfactual outcomes of students placed in pre-Algebra versus Algebra 1. For example, how do the findings differ if I use propensity score matching versus regression modeling? How do the findings differ if I use hierarchical linear modeling? I hope to acquire student-level data from an urban school district for a cohort of students who entered 9th grade in 2003 (graduating class of 2008).

While quantitative methods will drive the primary analyses, I plan to take a reflective, and more qualitative, approach to assess how the different procedures produce varying degrees of causal inference about the impact of differentiated course-taking. One question is whether a specific method might result in findings (descriptive and causal) that facilitate “qualification” of the data (e.g., a description of the differentiation process and/or a student’s progress through high school). For example, the propensity score approach might allow for a better understanding of the group selection process, but a path analysis approach might allow for a better understanding of direct and indirect effects as a student progresses through high school. Another line of investigation is whether the use of multiple quasi-experimental methods allows one to, in a qualitative sense, triangulate findings to improve causal inferences. Can one use multiple quasi-experimental and/or quantitative methods to offset the limitations of a single method, as one would try to do in a quantitative-qualitative mixed methods design?

Lastly, but perhaps most importantly, I plan to address the resulting information gaps, such as context, and how other methods (i.e., qualitative methods) could potentially fill those gaps to improve causal inferences. In particular, I will interview middle school administrators to identify subtle, yet key, factors that influence course placement. The interviews will primarily include open-ended questions about school policies and individual practices for assigning students to Algebra 1 or pre-Algebra. At the very least, the interviews will provide information on the types of important selection process data omitted from quantitative studies of course-taking pathways. I can then examine the sensitivity of my quasi-experimental results to omitted, confounding factors identified in the interviews.

Theorem #1: Productive Procrastination is not an Oxymoron

Much like a research project needs a purpose, goals and objectives, so too does this blog. While procrastination is a worthy goal in itself (see previous post), it's time to think big.

Theorem #1: productive procrastination is not an oxymoron. Under this theory, a blog can simultaneously serve as an opportunity to procrastinate and a medium to compose a research project. As such, here are a few productive objectives for this blog:

• brainstorm thoughts and reflections related to the research project at hand as they arise
• document the subjective decisions behind a seemingly objective research project (hence the title of this blog)
• document the research process from the first-person perspective of the researcher, similar to the way an ethnographer would keep a field journal during periods of observation

So let's start with that and see how things progress.

Procrastination: The First Law of Research

The first step in any research project is to procrastinate. As a first year student in the Social Research Methodology (SRM) division at the UCLA Graduate School of Education & Information Science, it's a shame it has taken me this long to reach this first and crucial stage.

Students in our division are expected (i.e., required) to complete an independent research project some time within the first two or three years of the doctoral program, and definately prior to jumping into a full-blown dissertation. It's akin to a master's thesis but those with the proper institutionalized culture refer to it as the 299 project -- based on the 299 course sequence students take to prepare for the project.

It's now spring (not technically but academically), less than two months until the end of my first year, and it's time to plan for this research project. So the natural thing to do is to start a blog ...