Developing a Proposal

I've spent some time thinking about and developing a proposed research project. Now it's time to develop these thoughts into a formal proposal to receive IRB approval and get this thing off the ground.

But first, let's figure out where things stand. Here's a snippet of my thinking for the study ...

Title. Assessing the Implications of Different Quasi-Experimental Methods through the Impact of Different Course-Taking Pathways

Motivation. By the time students enter high school they are placed on different course-taking pathways. Long-term academic success for the student is typically the stated reason for differentiation in course-taking patterns. For example, a student may be placed in a pre-Algebra course instead of Algebra 1 if the school determines s/he does not have the appropriate mathematics foundation to successfully complete Algebra 1. Differentiated placement, however, raises questions of equal access to educational opportunities and may ultimately harm a student’s chances of completing the coursework necessary for high school graduation and college enrollment.

The short- and long-term impacts of differentiated course-taking are difficult to measure, however; primarily because one cannot simply randomly assign students to different pathways and various threats to internal validity arise. In the absence of random assignment, our ability to make valid causal inference about course-taking pathways hinges on our ability to develop sound and robust quasi-experimental designs.

Purpose. An investigation into the impacts of differentiated course-taking pathways provides an opportunity to address two critical issues: one methodologically relevant and one policy relevant. Methodologically, the goal is to identify some key implications for causal inferences in quasi-experimental research. I plan to focus the methods inquiry on the ability to account for initial differences between comparison groups (i.e., treatment and control groups) by better understanding the selection process. Operationally, I plan to focus the study on differences in high school progress and performance between students who took Algebra 1 in 8th grade and students who took "pre-Algebra" in 8th grade. The goal is to determine whether placing the average 8th grade student in Algebra 1 results in positive outcomes in high school relative to placing the student in pre-Algebra.

Design. I envision a general research design centered around a quantitative, quasi-experimental approach. Within the broad quasi-experimental framework, however, I plan to compare different methods and statistical procedures to estimate the counterfactual outcomes of students placed in pre-Algebra versus Algebra 1. For example, how do the findings differ if I use propensity score matching versus regression modeling? How do the findings differ if I use hierarchical linear modeling? I hope to acquire student-level data from an urban school district for a cohort of students who entered 9th grade in 2003 (graduating class of 2008).

While quantitative methods will drive the primary analyses, I plan to take a reflective, and more qualitative, approach to assess how the different procedures produce varying degrees of causal inference about the impact of differentiated course-taking. One question is whether a specific method might result in findings (descriptive and causal) that facilitate “qualification” of the data (e.g., a description of the differentiation process and/or a student’s progress through high school). For example, the propensity score approach might allow for a better understanding of the group selection process, but a path analysis approach might allow for a better understanding of direct and indirect effects as a student progresses through high school. Another line of investigation is whether the use of multiple quasi-experimental methods allows one to, in a qualitative sense, triangulate findings to improve causal inferences. Can one use multiple quasi-experimental and/or quantitative methods to offset the limitations of a single method, as one would try to do in a quantitative-qualitative mixed methods design?

Lastly, but perhaps most importantly, I plan to address the resulting information gaps, such as context, and how other methods (i.e., qualitative methods) could potentially fill those gaps to improve causal inferences. In particular, I will interview middle school administrators to identify subtle, yet key, factors that influence course placement. The interviews will primarily include open-ended questions about school policies and individual practices for assigning students to Algebra 1 or pre-Algebra. At the very least, the interviews will provide information on the types of important selection process data omitted from quantitative studies of course-taking pathways. I can then examine the sensitivity of my quasi-experimental results to omitted, confounding factors identified in the interviews.