# Research Methodology and Data Analysis

In this project, we are using qualitative methodologies to explore the sociocultural aspects of teachers learning to mediate the literacy demands of reform-based curricula in urban settings; we will use quantitative measures of mathematics and literacy performance to discern relative effectiveness of such changes in teaching. We first describe our adaptation of the methodology of the multi-tiered teaching experiment (Lesh & Kelly, 1999), followed by a discussion of the quantitative measures and analyses.

*Qualitative analyses of teacher learning*: To examine the development of teachers' understandings we have adapted the methodology of the multi-tiered teaching experiment. This approach is allowing us to examine the interactions among the researchers, students and teachers so that their interpretations, tools, and theories can be assessed, extended, refined or rejected in ways that are increasingly useful, shareable, and powerful. We are designing classroom-based experiences that seek to create an environment where researchers and teachers, as well as students, directly confront their needs to develop new theories for describing, interpreting and explaining their own experiences (see table below, modified from Lesh and Kelly [1999]). Formative feedback and consensus building are used at each tier to ensure that the theories develop in directions that are continually better without simply testing preconceived notions about “best practice.”

Three-Tiered Teaching Experiments |

Tier #3
Researchers |
With teachers’ and students’ help, researchers develop models to make sense of teachers’ and students’ learning and to re-interpret and extend their own theories. |

Tier #2
Teachers |
Teachers develop shared artifacts (such as literacy scaffolding tools, libraries of student work, and cases of practice) as they work with researchers to describe, explain, and make sense of student learning. |

Tier #1
Students |
Teams of students work, with teachers’ help, on conceptually rich and contextually embedded tasks in which they construct and re-construct their interpretations of the problem situation. |

The essence of this methodological approach is to develop tools and artifacts that will reveal how teachers and researchers are thinking about the issue of the interrelated development of mathematics and literacy. The process of using and sharing such tools and artifacts will potentially change the interpretations and explanations of the experiences of the teachers and researchers. As researchers, we are collecting an audit trail of documentation that allows us to carefully describe the development of interpretations and explanations that took place.

At the beginning of the project, our primary data sources will include the following:

- transcriptions of audiotaped and/or videotaped study-group meetings and project-wide, meetings at which teachers and researchers examine reform-based curricula for the literacy opportunities they offer,
- fieldnotes from classrooms where teachers are providing opportunities for students to read, write, listen, and speak algebraically,
- examples of student work,
- “ways of thinking” sheets (Doerr & Lesh, in press) created by teachers that record how they see students constructing meaning from and with mathematical texts.

Later in the project, we will continue to rely on the data sources listed above but we will also add the following:

- videotapes of classroom events that integrate literacy and mathematics,
- observational notes by teachers using a variety of approaches such as scripting, tallying, and sketching,
- teacher-generated annotations (marginal notes, Post-its attached to teachers’ guides, lesson plans) while teaching the algebraic units of reform-based curricula,
- formal interviews with teachers and students using semi-structured protocols to aid with comparability of data across sites and subjects,
- transcriptions of audiotaped and videotaped summer work sessions.

Analysis of these data will focus on such issues as teachers’ knowledge of comprehension strategies useful for mathematical texts, teachers’ perspectives on critical obstacles to student construction of meaning from mathematical texts, evidence of different kinds of participation in literacy-related tasks by students from various cultural groups, and students’ responses to teachers’ invitations to express their own ideas about the stories and situations in the reform-based curricula. Of central importance to this analysis will be those artifacts generated by the teachers themselves: literacy scaffolding tools, the library of student work, and cases of practice. Each of these will allow us to make inferences about the teachers’ ways of thinking about how best to support the development of students’ abilities to communicate algebraically. By examining changes in the shared artifacts and how they are used, we will also be able to trace how that thinking changed over time and how it influenced teachers’ daily practice. In this way, we will move beyond simply looking at student work to understand how teachers use the careful examination of student work to effect changes in their practice.

At the beginning of the project’s second year, researchers and teachers will identify key themes for the creation of a "case of practice" related to the teaching and learning of algebra in each of the four school sites. The first step in developing these cases will be to write a description of the events and perspectives that led to the selection of the identified theme. Using this starting point, the team will collect artifacts and rich records of practice such as lesson plans, student work, videotapes of teaching strategies, and teacher reflection that will be assembled into a multimedia text that can, in turn, be shared with the entire project. The creation of these multimedia texts will draw on and extend our prior work in the use of multimedia cases of practice to support the professional development of teachers (Bowers, Doerr, Masingila, & McClain, 1999, 2000; Chandler-Olcott, Heroux, & Bonilla, 2001). During the second and third years of the project, these cases will be the focus of study-group and work-session analyses by teachers and researchers that examine the interrelated development of literacy and mathematics.

The close collaboration of researchers in mathematics education and literacy education will yield complex understandings about the literacy demands of reform-based mathematics curricula that single-discipline research may not. Just as secondary mathematics teachers are underprepared to support students’ literacy learning, mathematics education researchers often lack a knowledge base for and experience with supporting students’ literacy development. Although literacy researchers sometimes pursue solo research projects in mathematics classrooms, they often do so without sophisticated understandings of the mathematical concepts teachers want students to construct. Interdisciplinary data analysis will help us to avoid both of these problems, as members of the research team will possess complex knowledge of mathematical concepts and literacy processes as a collective. Such an approach is consistent with sociocultural theories positing that knowledge and skills do not belong solely to individuals but rather are distributed across the social practices of a given community (Gee, 2000; Lave & Wenger, 1991).

- Analyses of student outcomes: A shared focus of the research team will be the careful examination of student outcomes throughout the project. As for many urban districts, the existing achievement gap has been a long-term, persistent difficulty. While it is not likely that a three-year project will produce dramatic changes in student outcomes, we will closely examine the changes in student outcomes that may occur. Furthermore, this project will provide an opportunity for the entire research team of university-based researchers in mathematics education and literacy education, mathematics teachers, and administrators, including principals and instructional leaders, to examine the relationship between changes in teaching practice and student outcomes. In order to partially separate the effect of the focus on the literacy demands when teaching a reform-based curriculum, we will collect and analyze achievement data using a 2x2 design with four cohorts. Two of the cohorts will consist of students in the study population: one from the three middles schools in grades 6 through 8, and the second from the high school in grades 9 and 10. The other two cohorts will consist of a group of students from three middle schools in grades 6 through 8 and from another high school in grades 9 and 10, all from the same district. This second set of cohorts will be comparable to the two cohorts in the study in terms of demographic make-up and will be using the same curricular materials as those in the study population. This design will allow us to make comparisons on measures of mathematical achievement and language arts achievement within the study group as the two cohorts move through their academic program across grades 6 through 8 and then again in grades 9 and 10. The within group analyses will compare the change in student achievement over a three year time span. We will also compare the achievement of these two cohorts of students to other students who were taught with the same curricula but without the benefit of the teachers' explicit instructional focus on the literacy demands of those curricula. We will use current school district measures of student achievement in mathematics and language arts for these analyses. In particular, we will compare the changes within each cohort group on the Terra Nova Multiple Assessment (CTB-McGraw Hill) group achievement test in reading, language and mathematics, which is given in the 6th and 7th grade, using the previous year's data at the 5th grade as a baseline. We will also compare the student achievement on three statewide achievement tests: the New York State Middle Level Mathematics Examination and the New York State Middle Level English Language Arts Examination, both of which are given in the 8th grade. We will compare achievement data on the New York State Math A Examination, ordinarily given in the 10th grade. This examination has both multiple choice items and extended response items that require a greater level of reading comprehension and written expression by the student.