Making Sense of Student Performance Data

Kim Marshall draws on his 44 years’ experience as a teacher, principal, central office administrator and writer to compile the Marshall Memo, a weekly summary of 64 publications that have articles of interest to busy educators. He shared one of my recent articles, co-authored with doctoral students Britnie Kane and Jonee Wilson, in his latest memo and gave me permission to post her succinct and useful summary.

In this American Educational Research Journal article, Ilana Seidel Horn, Britnie Delinger Kane, and Jonee Wilson (Vanderbilt University) report on their study of how seventh-grade math teams in two urban schools worked with their students’ interim assessment data. The teachers’ district, under pressure to improve test scores, paid teams of teachers and instructional coaches to write interim assessments. These tests, given every six weeks, were designed to measure student achievement and hold teachers accountable. The district also provided time for teacher teams to use the data to inform their instruction. Horn, Kane, and Wilson observed and videotaped seventh-grade data meetings in the two schools, visited classrooms, looked at a range of artifacts, and interviewed and surveyed teachers and district officials. They were struck by how different the team dynamics were in the two schools, which they called Creekside Middle School and Park Falls Middle School. Here’s some of what they found:

  • Creekside’s seventh-grade team operated under what the authors call an instructional management logic, focused primarily on improving the test scores of “bubble” students. The principal, who had been in the building for a number of years, was intensely involved at every level, attending team meetings and pushing hard for improvement on AYP proficiency targets. The school had a full-time data manager who produced displays of interim assessment and state test results. These were displayed (with students’ names) in classrooms and elsewhere around the school. The principal also organized Saturday Math Camps for students who needed improvement. He visited classrooms frequently and had the school’s full-time math coach work with teachers whose students needed improvement. Interestingly, the math coach had a more sophisticated knowledge of math instruction than the principal, but the principal dominated team meetings.

In one data meeting, the principal asked teachers to look at interim assessment data to predict how their African-American students (the school’s biggest subgroup in need of AYP improvement) would do on the upcoming state test. The main focus was on these “bubble” students. “I have 18% passing, 27% bubble, 55% growth,” reported one teacher. The team was urged to motivate the targeted students, especially quiet, borderline kids, to personalize instruction, get marginal students to tutorials, and send them to Math Camp. The meeting spent almost no time looking at item results to diagnose ways in which teaching was effective or ineffective. The outcome: providing attention and resources to identified students. A critique: the team didn’t have at its fingertips the kind of item-by-item analysis of student responses necessary to have a discussion about improving math instruction, and the principal’s priority of improving the scores of the “bubble” students prevented a broader discussion of improving teaching for all seventh graders. “The prospective work of engaging students,” conclude Horn, Kane, and Wilson, “predominantly addressed the problem of improving test scores without substantially re-thinking the work of teaching, thus providing teachers with learning opportunities about redirecting their attention – and very little about the instructional nature of that attention… The summative data scores simply represented whether students had passed: they did not point to troublesome topics… By excluding critical issues of mathematics learning, the majority of the conversation avoided some of the potentially richest sources of supporting African-American bubble kids – and all students… Finally, there was little attention to the underlying reasons that African-American students might be lagging in achievement scores or what it might mean for the mostly white teachers to build motivating rapport, marking this as a colorblind conversation.”

  • The Park Falls seventh-grade team, working in the same district with the same interim assessments and the same pressure to raise test scores, used what the authors call an instructional improvement logic. The school had a brand-new principal, who was rarely in classrooms and team meetings, and an unhelpful math coach who had conflicts with the principal. This meant that teachers were largely on their own when it came to interpreting the interim assessments. In one data meeting, teachers took a diagnostic approach to the test data, using a number of steps that were strikingly different from those at Creekside:
  • Teachers reviewed a spreadsheet of results from the latest interim assessment and identified items that many students missed.
  • One teacher took the test himself to understand what the test was asking of students mathematically.
  • In the meeting, teachers had three things in front of them: the actual test, a data display of students’ correct and incorrect responses, and the marked-up test the teacher had taken.
  • Teachers looked at the low-scoring items one at a time, examined students’ wrong answers, and tried to figure out what students might have been thinking and why they went for certain distractors.
  • The team moved briskly through 18 test items, discussing possible reasons students

missed each one – confusing notation, skipping lengthy questions, mixing up similar-sounding words, etc.

  • Teachers were quite critical of the quality of several test items – rightly so, say Horn, Kane, and Wilson – but this may have distracted them from the practical task of figuring out how to improve their students’ test-taking skills.

The outcome of the meeting: re-teaching topics with attention to sources of confusion. A critique: the team didn’t slow down and spend quality time on a few test items, followed by a more thoughtful discussion about successful and unsuccessful teaching approaches. “The tacit assumption,” conclude Horn, Kane, and Wilson, “seemed to be that understanding student thinking would support more-effective instruction… The Park Falls teachers’ conversation centered squarely on student thinking, with their analysis of frequently missed items and interpretations of student errors. This activity mobilized teachers to modify their instruction in response to identified confusion… Unlike the conversation at Creekside, then, this discussion uncovered many details of students’ mathematical thinking, from their limited grasp of certain topics to miscues resulting from the test’s format to misalignments with instruction.” However, the Park Falls teachers ran out of time and didn’t focus on next instruction steps. After a discussion about students’ confusion about the word “dimension,” for example, one teacher said, “Maybe we should hit that word.” [Creekside and Park Falls meetings each had their strong points, and an ideal team data-analysis process would combine elements from both: the principal providing overall leadership and direction but deferring to expert guidance from a math coach; facilitation to focus the team on a more-thorough analysis of a few items; and follow-up classroom observations and ongoing discussions of effective and less-effective instructional practices. In addition, it would be helpful to have higher-quality interim assessments and longer meetings to allow for fuller discussion. K.M.] “Making Sense of Student Performance Data: Data Use Logics and Mathematics Teachers’ Learning Opportunities” by Ilana Seidel Horn, Britnie Delinger Kane, and Jonee Wilson in American Educational Research Journal, April 2015 (Vol. 52, #2, p. 208-242

Advertisements

What I Notice and Wonder about Teaching Like a Champion

Last night, Chris Robinson shared an experience with an administrator who observed his math classroom. He had been doing an activity called Noticing and Wondering with his students, something that Max Ray of the Math Forum has written about extensively. Noticing and wondering is a great discussion starter. You share a mathematical object or situation with children and open up the floor to their curiosity. They can connect the mathematical thing with their own ideas, then a teacher can shape the conversation by building connections to formal math.

Here is the administrator’s feedback:

Now, I am not naive. I understand that our lack of consensus about good teaching leaves a lot of room for interpretation about what is working and what is not. The administrator was obviously perplexed by the wide berth Chris gave to his students to wonder about the math. Kids do say and think goofy things, as do all people. But sometimes our odd ideas need a good airing to connect to what we are learning.

Normally, seeing Chris’s tweet would frustrate me. What do we need to do to drive a wedge between people’s confusion about students being compliant and being engaged? What do we need to do to help educators understand that the path to deep understanding is often not a straight line, and that to connect ideas to our lives, our own thinking –– goofy or not –– needs a chance to come out?

Yesterday, however, the administrator’s problematic response did more than frustrate me. As I told Chris (and the others on the thread):

In my class Teaching as a Social Practice, we have been discussing the consequences of our lack of consensus on the nature of good teaching. We often examine what gets put out and circulated as good teaching and hold it against various research on things like  how kids learn or how teachers can teach responsively.

I showed this Doug Lemov video related to his best-selling book, Teach Like a Champion, with the intent to dissect the underlying assumptions about teaching and learning. The 100% technique is a way of managing students’ attention during instruction. Take two minutes to watch it.

What do you notice? What do you wonder?

I notice that these are all White teachers and that the students are nearly all Black.

fold hands

I wonder why the teacher (above) is signalling this boy to have his hands folded. I wonder if there is any research anywhere showing that folded hands will help with his learning.

Whisper to Jasmin

I notice that when this teacher reprimands this student for not having the answer to a question (1:11 on the video), she jumps immediately to the assumption that the girl needs to work harder. I wonder why the teacher doesn’t ask her if she has any questions about what was being asked or if everything is okay today.

Giving you a gift

I notice that this teacher says the following to his class as a motivational speech (1:44):

I can bring it to you but I can’t give it to you. You’ve got to reach for it. If they were free at Toys R Us you would reach. I’m giving you the same kind of gift, just not wrapped up. The gift of knowledge.

I wonder what is going on in this metaphor. I am wondering if I ever have seen wrapped up gifts at Toys R Us. I wonder if other overly analytical kids in this class also got lost down this rabbit hole of wondering.

I wonder if the kids would like the gift of being able to keep their hands unfolded and moving their bodies more freely more than the gift of repeating after the teacher in the name of “knowledge.”

____

What does all this have to do with Chris and his interaction with his administrator?

Teach Like a Champion has been a huge seller, especially in urban schools. It’s highly rated and ranked on Amazon and I have talked to numerous new teachers who report getting handed a copy by administrators. There is even a new edition Champion 2.0.

Activities like noticing and wondering open up classroom discussions and invite kids (goofy ideas and all) to think. Techniques like 100% in Teach Like a Champion limit permissible activity and thinking by students.  Contrasting the two is a productive microcosm on current debates about teaching. The issue is particularly urgent in urban classrooms, where methods like those promoted in Champion emphasize the control of Black and Brown bodies by White teachers instead of the celebration of children’s own ideas. This is especially troubling given what we know about disproportionate discipline of these children.

With this vision of teaching dominating the landscape, it becomes increasingly difficult for teachers like Chris Robinson to invite their children to think with him in the classroom without the risk of being reprimanded.