on the non-normativity of value-added analysis

As you are likely to have heard by now, the Los Angeles Times recently conducted and published a value-added analysis of some of the city’s elementary school teachers, using data that had been collected by the school district but never previously analyzed in this way. There was a nice summary of the value-added analysis and the ensuing controversy in the New York Times this week.

And here’s a quick but thoughtful critique of that summary over at the Quick and the Ed. Its author dispels two common criticisms of the value-added analysis:

  1. We shouldn’t criticize value-added analysis simply on the basis that it shows many teachers’ effectiveness as shifting substantially from year to year. It’s possible that teacher effectiveness *does* shift from year to year, for whatever reason.
  2. Because this particular method of value-added analysis uses individual students’ own previous scores as a baseline for measuring progress, it does not penalize teachers for having slower students in their classes (at least, the criticism must be more subtle, as a commenter suggests).

Still, the most interesting thing I’ve read on the subject remains cognitive scientist Daniel T. Willingham’s “3 key factors in teacher evaluation (beyond the hype of value added).” According to him, the key factors are:

  1. Figuring out the goals of schooling, and only then crafting assessments to measure teachers’ success at attaining those goals. It’s backwards to assume that whatever we are able to test or assess must be the goal of schooling.
  2. Taking into account the ages of students, recognizing that responsibility for students’ learning probably falls more fully on teachers of early elementary schoolers than on those of high schoolers.
  3. Remaining cognizant of the limitations of any evaluation system in thoughtfully choosing criteria for firing teachers that balance the costs of keeping bad teachers with the costs of firing good ones.

Each of Willingham’s three key factors gets at what is essentially the major problem with value-added analysis: its results, even if accurate, lead us away from philosophical questions about education and teaching, the answers to which have important practical and policy implications. So I want to elaborate on them, with that in mind.

The goals of schooling are far from noncontroversial. Although most people would agree that proficiency in math and reading are amongst its goals, there is significant disagreement as to what constitutes proficiency and how it ranks in importance as compared to other goals of schooling (character development, socialization, preparation for the workforce – which may or may not require proficiency in math and reading, etc). It’s a mistake to establish teacher evaluation policy based on value-added analyses without having clarified at least some of the goals of schooling and their relative importances. As should be obvious, this value judgment can’t be generated from the value-added analysis itself. Rather, it will be outcome of philosophical discussion regarding the moral value of character development/socialization/preparation for the workforce, the best kind of life for a human to lead, how these responsibilities should be shared between schools and families, etc.

Willingham’s second point, about the ages of students and their respective levels of responsibility for their own learning, also raises moral questions. The discourse surrounding value-added analysis has seemingly taken entirely for granted that teachers ought to be doing all that they can to raise students’ test scores, regardless of their ages. This stands in need of defense. While we may be reluctant to blame 7 or 8 year old students for failing to learn math and reading, it may be appropriate to blame 16 and 17 year olds for failing to progress in those subjects. Some high school teachers may manage to raise teenagers’ test scores significantly, and they will come out looking better than other teachers in the value-added analysis. But, even if raising students’ test scores were of the most pressing importance in their early years, other functions may be more important for teachers to engage in at the high school level – maybe helping students to think about their future educational and career plans, and taking a more laissez faire approach in order to begin acclimating them to the “real world.”

So this ties back into the previous point, about the goals of schooling. If raising math and reading test scores is, literally, the one and only proper goal of schooling, then all teachers should be expected to do so each year. However, there may be many other goals of schooling that are more difficult to test. Teachers will need to make tradeoffs in pursuing these various goals, depending not only on their relative importances but based on what their students are like. Maybe, in some particular class, some of the students are ok at math but have social difficulties. Assuming that social development is one of the goals of schooling, the teacher might reasonably decide to devote more time to group work than to math drills. As a result, the students might progress more slowly in math than in previous years, while having made strides socially that do not show up on any test.

The third key factor, about criteria for firing teachers, raises even more moral questions. There are costs associated both with keeping bad teachers and with firing good teachers. If you keep a bad teacher, many students in his or her classes will fail to learn as much as they could have learned with a better teacher, negatively impacting their future educational outcomes and maybe even significantly harming their life prospects. If you fire a good (or at least adequate) teacher, you unduly harm that teacher and demoralize her colleagues (and the replacement teacher might be an unknown quantity who turns out to be worse). We might privilege students’ well-being over teachers, erring on the side of firing, or we might privilege teachers’ well-being in order to show respect to what many consider one of the most important professions. The methods of economics may tell us how to set teacher firing criteria so as to be financially cost effective, but that’s not necessarily the end of the story from the moral perspective.

None of this to say that value-added analysis is “bad,” or has no legitimate purpose. Its results might be quite accurate and useful to some degree, as is perhaps the case in this Los Angeles situation. But we need to realize how it does – or doesn’t – square with our conception of what education, and teaching, ought to be (and, if we lack such a conception, we need first to develop one). At the end of the day, value-added analysis is a descriptive/evaluative tool, and not a normative one.

2 Comments

  • You raise several critical points that have been overlooked in the public conversation on the report. The reason this story has gained the traction that it has is that there are figures testifying to the fact that an LA teacher with tenure is about as immovable as the Rock of Gibraltar these days. My familiarity with the Ed Lit is admittedly weaker than yours. However, I’ve seen evidence documenting that good teaching can in fact raise test scores. How one would test this is, of course, open to debate. What strikes many as offensive however is that teachers’ unions reactionary objection to the mention of anything smacking of oversight.

  • Yep, you’re definitely right about the unions’ “reactionary objection to the mention of anything smacking of oversight” (although the tides may be turning on this as testing-based accountability gains bipartisan support).

    I am additionally suspicious of their agreeing to be held more accountable — but only by regulating *themselves*, without external involvement. Where are the incentives for them actually to do so?

Leave a Reply