Value Added Measures: Fail (Shanker Blog): Updated

Much of the criticism of value-added (VA) focuses on systematic bias, such as that stemming from non-random classroom assignment (also here). But the truth is that most of the imprecision of value-added estimates stems from random error. Months ago, I lamented the fact that most states and districts incorporating value-added estimates into their teacher evaluations were not making any effort to account for this error. Everyone knows that there is a great deal of imprecision in value-added ratings, but few policymakers seem to realize that there are relatively easy ways to mitigate the problem.

This is the height of foolishness. Policy is details. The manner in which one uses value-added estimates is just as important – perhaps even more so – than the properties of the models themselves. By ignoring error when incorporating these estimates into evaluation systems, policymakers virtually guarantee that most teachers will receive incorrect ratings. Let me explain.

Each teacher’s value-added estimate has an error margin (e.g., plus or minus X points). Just like a political poll, this error margin tells us the range within which that teacher’s “real” effect (which we cannot know for certain) falls. Unlike political polls, which rely on large random samples to get accurate estimates, VA error margins tend to be gigantic. One concrete example is from New York City, where the average margin of error was plus or minus 30 percentile points. This means that a New York City teacher with a rating at the 60th percentile might “actually” be anywhere between the 30th and 90th percentiles. We cannot even say with confidence whether this teacher is above or below average.

...
[Update! I forgot the following paragraph!!]

Now, here’s the problem: In virtually every new evaluation system that incorporates a value-added model, the teachers whose scores are not significantly different from the average are being treated as if they are. For example, some new systems sort teachers by their value-added scores, and place them into categories – e.g., the top 25 percent are “highly effective,” the next 25 percent are “effective,” the next 25 percent are “needs improvement,” and the bottom 25 percent are “ineffective.”

Read the whole thing @Shanker Blog

L.A. Times: Should We Sue Them?

These are my students' CST scores (the TEST scores) from a couple years ago. Don't bother looking at how well they did compared to the rest of the state or district (go ahead). Look instead at the red box under the red arrow near the upper right. See that statement? It says that the report, which is the report of my students' scores on the high-stakes test and therefore a report about my teaching, is not to be used in any form of teacher evaluation. I don't think that has changed, yet.

The L.A. Times saw fit to discuss teachers, by name, in an article today. It sure smacks of scores being used as a form of teacher evaluation.

How did they get the information? Why do they think they get to publish it? To the 6 people left subscribing to the L.A. Times, maybe now would be a good time to cancel that subscription.

Problems With Student/Teacher Performance Linkage

Linking student performance to teachers is a dubious undertaking. Sherman Dorn explains why:

Longitudinal data systems, good; unique teacher linkage, bad

Diane Ravitch's blog entry this morning seriously disparages the value of longitudinal data systems, including the linking of teachers to students, and John Thompson's entry discusses the abuse of data by administrators. Essentially, both Ravitch and Thompson fear the brain-dead or conscious abuse of data to judge teachers out of context. That's also the reason why NYSUT (the New York state joint NEA-AFT affiliate) worked hard to convince the legislature to put a moratorium on using test scores to make tenure decisions; Joel Klein was moving very quickly, and I think UFT and NYSUT had good reason to believe that without the moratorium, there would be substantial abuses of test data in NYC (and elsewhere) in tenure decisions.

My take: longitudinal data systems are a good thing, but linking teachers to students is a much more fragile undertaking.

Florida has a longitudinal data system that began in the early 1990s and has been used for 10 years to judge schools based on test data. Approximately ten years ago, I sat in a windowless room in Tallahassee as a Florida DOE member discussed the new A-plus system and a variety of technical decisions tied to it, and for which he had brought stakeholders and a few yahoos from around the state to give advice. I was one of the unpaid yahoos who had the great joy of flying in tiny airplanes several hundred miles a few times a year to give advice on the matters.

We had so many matters to discuss that one minor conversation was almost overlooked: a state mandate that required that the FDOE link each student to a teacher primarily responsible for reading and math. One state official showed us a draft form and then explained the concerns he had about it: in his view, the state that had tried that a few years earlier (Tennessee) had multiple conceptual difficulties connecting individual teachers to individual students. But they had run roughshod over those concerns, and he anticipated that Florida would do the same.

It wasn't a matter of letting teachers off the hook (this now-retired professional staffer is what I think of as an accountability hawk) but logic and sense. How many physics and chemistry teachers help students understand algebra better? How many history teachers help students with writing or reading? For students receiving special education services in a pull-out system, do you want only the special educator to be responsible for a subject, or do you want both the general-ed classroom teacher and the special educator to have responsibility? This spring, my wife (a math major and special educator) is tutoring a local child in math on weekends or evenings; so who should get credit for how he performed on testing in the last week, his teachers in school or my wife? Today, you can add NCLB supplemental educational services (or after-school tutoring) to the mix.

The larger point: even if you decide to wave away the concerns of Richard Rothstein and others, even if you focus entirely on what happens in academic environments, it is fallacious to link every student performance with a single teacher. If we are providing the appropriate supports for children, then the students with the lowest performance are the ones for whom such unique linkage assumptions are the least justifiable, because they may be receiving academic support from general education classroom teachers, from special educators, from after-school tutors, and maybe mentors or other providers in neighborhood support organizations (such as Geoffrey Canada's). Today, I do not think one can parcel out responsibility without making assumptions that have no basis in empirical research. Those who support individual teacher linkage have the burden to demonstrate otherwise.

Teacher Evaluations: A Crock

Teacher evaluations are dubious for a couple reasons. The first is the fact that principals, in my district at least, evaluate based on 3 observations in the classroom. These observations do not really provide enough evidence of teaching or not-teaching, unless the principal is predisposed to evaluate you in a certain way--positively or negatively. In that case, they can point to just about anything they want, and leave out anything want, to make the evaluation appear any way they want.

The second problem, related to the first, is the fact that the principal does this alone. There is no other person there, observing the same thing, to compare notes with. Perception is not always reality, and 2 people looking at something may just see things very differently.

Because of the way teacher evaluations are performed, teachers, unless they are on good personal terms with the principal, must live in fear of the baseless negative evaluation for which there is no recourse because it is the principal's word over the teacher's. As we know, management tends to win these battles.

I have many, many letters of appreciation from parents of students, and have had master teachers and principals tell me I am a fantastic teacher. I have been asked--by my current principal!--to be math leader (I said no), disaster coordinator (I am currently), tech leader (was until we got a new teacher who used to be tech guy for his district--he knows way more than me!), and am in charge of whole-school outings. Yet, because I push back, my principal gives me negative evaluations. This year is the first year I have ever received a negative evaluation. My teaching has not changed; indeed I feel I have gotten better having been left in the same grade for 4 years in a row (something I have written about before).

I would like to see teacher evaluations done by fellow teachers in conjunction with principals, and I would like the written notes of all evaluators to be included, with equal weight, in any final evaluation. This would be more informative and much more fair than the way it is done now.

For all you "unions protect the bad teachers" folks out there, the union can do nothing about a principal hell bent on firing a teacher; we teachers don't have any mechanism to expose the nonsense; union negotiations see to that (we lose, always).

More Proof They Don't Know What They're Doing

Last year I was evaluated. My students' state scores were not taken into account, but why would they be? In a teacher evaluation, the teacher, as well as the principal, set vapid, purely lip-service-to-the-man "goals" for the teacher, because as we all know, teachers need improving and the best way to do that is to point out what they don't do well.

Anyhoo, one of the goals was to have the second grade classes work toward a system where the teacher that felt most comfortable with, and was most successful at, a certain subject would teach that subject to the 3 second grade classes. For background, because my students did well on math assessments, and the principal was impressed with a math lesson she witnessed in my class, she wanted to get me to as many kids as she could, and this was a way to do that. I had mentioned it as a possibility in a conference or something during a brainstorm, and never thought about it again. She did, and suggested it as a goal. There are problems with the idea from the gitgo:

1. The other teachers have to want to teach science and literacy. What if they don't? Do I fail in my goal?

2. Parents might not want the lack of continuity for their children.

3. Kids may not like the lack of continuity.

4. I may get sick of teaching math all day every day (you upper school teachers have it rough!)

And who knows what other issues could come up?

So, that is the background and setup for what I realized after the meeting, tonight, when I got home, regarding my post below, about today's meeting, before I wrote it:

If we are to align our instructional blocks (literacy at 10am, math at 1pm, etc) so all second grade teachers teach the same thing at the same time, my goal (above) would be impossible. Not only would it be impossible, it seems to be antithetical to the pedagogy revealed by the new alignment of instructional blocks. Which pedagogy, or curricular delivery system, is best? Clearly the principal has no fucking idea.

I always blow off the goals, so my forgetting about it makes sense. But for the principal to espouse one way to deliver curriculum to kids as a worthy goal, then to tell us we are to do this aligning thing--which is the complete opposite--not only smacks of stupidity, but a bifurcated and incompatible-with-itself pedagogical view(s?). She is not compartmentalizing. She is confused. Sunni? Shia?

Prove Your Premise!!

I read a snippet of a story over at Schools Matter that just bolsters the correct notion that NCLB is a policy with a head-up-its-ass problem. Here is the mythbusting story from NewsReview, Reno!

Policy myths cause a lot of government’s problems, and education is especially damaged
Can a school system built on imaginary premises possibly succeed?

By Dennis Myers

Richard Rothstein spent three years as education columnist for the New York Times, giving a popular audience an unaccustomed look at the work of a scholar in sharp, to-the-point essays that challenged conventional wisdom and corrected public policy assumptions. Reading them and his current work as an associate at the Economic Policy Institute in Washington, D.C., one gets the impression of a very patient man. Although he must deal constantly with public policy myths—things we all know to be “true” but are not—he keeps plugging away trying to dispel them."All I can do is keep on telling the truth as I see it, and others have to do the same,” Rothstein says. “In the long run, I have to hope the truth wins out.”

The number of myths on which so many public policies are built raise serious questions about whether those policies have any hope of succeeding. In education, there are a number of myths that have been repeated so incessantly by press and politicians that they have become “true” in the public’s mind. Samples:

• Business uses performance pay, so schools should do the same

• Schools are violent

• Parents and students are fleeing public for private schools

• Schools should use the kind of numerical goals that business uses

• Charter schools outperform public schools

In fact, business generally avoids performance pay, schools are the safest places children frequent, private school enrollment is declining, business recommends against numerical goals, and public schools generally perform better than charter schools.

Take just one of them—numerical goals, which were actually written into the No Child Left Behind Act in the belief that they are an accepted business practice. In fact, it reflects a practice that has long since passed out of fashion in the business world, which found that it focused workers on process instead of outcome and generated fear in the workplace, low productivity, and customer alienation.

Using modern research methods, the business community discovered that when a worker must have numbers to show, she or he will crank them out by some means, at the expense of careful workmanship and productivity. Legendary statistician W. Edwards Deming, who in the postwar years gave Japanese management the methods of design, product quality and sales that brought that nation to commercial dominance and later swept the United States, wrote in his book Out Of the Crisis, “A numerical goal leads to distortion and faking, especially when the system is not capable of meeting the goal. Anybody will meet the quota (goal) allotted to him. He is not responsible for the losses so generated. Sears Roebuck waded into trouble in 1992 by allotting goals to their Auto Service Centers. Agents tried to meet the goals set for them. They did, to the detriment of the customer and of the reputation of the company.”

Insurance consultant John Pryor tells the companies he advises, “Focus first on the underwriting or claims or audit processes—and the quality of their delivery from the customer’s perspective—and then it can be determined if productivity is optimum.”

Yet numerical goals—what Rothstein calls “goals distortion” are a basic part of U.S. education policy.

Our peaceful schools
In 1998 during the spate of heavily publicized school shootings around the nation, the Center on Juvenile and Criminal Justice checked some figures on school violence against figures provided by the National Climatic Data Center. “To give the reader a sense of the idiosyncratic nature of these [school violence] events,” the CJCJ reported, “the number of children killed by gun violence in schools is about half the number of Americans killed annually by lightning strikes.”

Nothing has changed since then. There were more deaths on school grounds 32 years ago, when there were 80 million fewer people in the United States than there are today—usually less than 50 annually. Of home, the 7-Eleven, the park, school is their safest environ—far safer than the home, where as many children are killed in family violence every three days as died at Columbine. But the myth lives on.

Rothstein has been a one-person myth-buster in the education field by doing what reporters are supposed to do—checking the facts and statistics before reporting a “trend” (which may be the most abused word in journalism).

When in the early years of the Bush administration members of Congress were arguing that the private sector uses performance pay, so schools should do likewise, Rothstein went looking for companies that did so. He talked to firms like Wal-Mart and Cisco Systems, consulted Harvard Business School, called private and commercials schools. All told him the same thing. “The private sector does nothing of the sort,” he said. He even called John Chubb at Edison Schools Inc., the largest firm that tries to get contracts to commercially operate public schools. Chubb said using test scores to influence pay was a mistake. (Rothstein does acknowledge that “stockbrokers and sales clerks are paid on commission” but says the hardball sales tactics the practice fosters “should be intolerable where children are concerned.")

What George Bush, in his 2000 campaign, called parents voting with their feet—taking kids out of public and into private schools—did not escape Rothstein’s notice. He found that the actual number showed enrollment falling in private schools at every income level.

The charter school myth has been examined in depth, with Rothstein a part of it. In their book The Charter School Dust-Up, authors Martin Carnoy, Rebecca Jacobsen, Lawrence Mishel, and Rothstein examined 19 studies in 11 states and the District of Columbia, and then folded in data from the 2003 National Assessment of Educational Progress (NAEP) tests. Their conclusion: “There is no evidence that, on average, charter schools outperform regular public schools. In fact, there is evidence that the average impact of charter schools is negative.” (Their study also found that a claim that charter schools serve more disadvantaged students than public schools was false.)

In some cases, legislators have enacted legislation knowing full well that it was flawed. Rothstein recalls that during the debates over No Child Left Behind, economists Thomas Kane and Douglas Staiger produced a paper showing that, under the legislation, schools would be rewarded or penalized entirely because of dubious statistics produced by inadequate testing. Months went by while members of Congress wrestled with the problem. Then, they just gave up and passed the bill anyway.

“It’s a mistake to adopt policies that you know, based on the science, cannot work effectively,” Rothstein said.

The cost
None of this would matter much if it was harmless, but it is very harmful. Though the pendulum is now swinging back, for a decade precious education dollars were diverted to expensive high tech security gear, more police and weapons, expansion of juvenile jails. Various panaceas were legislated. In Virginia, the governor proposed eliminating after-school programs, an established violence preventive.

No Child Left Behind has exacerbated long-standing education problems, such as teacher shortages.

Can a school system succeed when its policy premises are false?

Former Nevada school superintendent Eugene Paslov says policy myths are complicated—he says they are like onions, with merit to be found in some levels as they are peeled—"Some of it has merit and much of it is mythology.” It’s hard to imagine hard pressed school administrators having the time or resources to sort things out.

Washoe County School District spokesperson Steve Mulvenon says he has to spend unnecessary amounts of time helping reporters do stories about school violence in the hope that they’ll get it right, though at times he has become so exasperated that he has considered cutting off assistance to such stories. “I guess I need to keep putting the message out there on each one of these incidents, that it is an aberration, that it is unusual.”

Principals: God's Gift To Teachers?

Over at LeaderTalk there is a post by Chris Hitch (not Hitchens) on how to handle the prima donna teacher. Basically, if a principal has a teacher who's students consistently do well, but rubs the principal the wrong way, the principal should fire the teacher. Hmmm. I think principals, for the good teacher, are a hindrance, not a help. Read this ridiculous screed with the 4 comments here, and the article only below the fold.

Handling Prima Dons (and Donnas)

I had a colleague in a school ask me about how to handle a prima don on his staff. He feels he is being held hostage by this individual. The individual is loved by parents, does great things with his students, is creative in the classroom, and get his students (all of their students) to perform at their best. He's also also cranky, arrogant, and think the rules apply to everybody else but him. This summer, the prima don may implicitly (or explicitly) say that if things don’t go his way, he will leave the school.

I suggested two filters to use at the outset-I had my suggestions, which you'll see below but I am quite curious about what readers of this blog suggest instead of or in addition to what I've suggested.

1. Ethics and Policy: If the person is violating any ethical canons or school board policy, you have no choice but to confront and quickly use whatever disciplinary measures you have to in accordance with your local and state policies and procedures. It’s one thing for a teacher to care about the children under her supervision or for a technology teacher to show how streaming video websites work, it is an entirely different matter for her to dispense pain medications or temporarily disable your system’s website filtering system without following your district policies. Whether the person is teacher of the year or a family member of the school board chair, you have to handle this person in the exact same manner as any other employee.

2. Consistent High Performance: All three of these words matter. If the person is not giving you consistent high performance, then you have a much easier decision-cut them no slack and look for removal, dismissal or nonrenewal. If you have a person who is rude to everybody and doesn’t do a good job, you are under no obligation to keep them around. We’d suggest that each time you work with a prima donna, you think through whether the person is still exhibiting consistent high performance.

A. Find the combination: While many prima donnas exhibit the same annoying and destructive behavior, they generally have different sets of needs. Some may need additional “air time” to demonstrate their superior knowledge and intellect. Others may want public recognition for the work they have done. Still others may want to be perceived as a “power player” by being asked to serve on a district level committee. By looking for what you see that they want, you have a tool that you can use to help leverage better behavior from them.

B Build a wall: Prima donnas know they are good and some may simply want to be left alone. If your prima donna simply wants to be left alone, put their expertise to use on a lone ranger project that aligns with your strategic goals and they don’t have to spend time in what they consider wasted time in endless meetings.

OR

Build a fence: If your prima donna loves the limelight, and wants to be perceived as a leader, have them lead a committee, but with conditions. Your conversation may go like this. “I’m glad you’ve agreed to lead this committee. Remember that one of the key aspects of this committee is to get a set of recommendations that that everybody will buy in to. You’ve got a lot of skill and talent and your perspective is valuable. One of the areas that will be important for you to focus upon is building upon the ideas of everybody in the group. What ideas do you have to make sure that everybody is listened to and heard?” When the prima donna pushes back, you have an opening to discuss how their skills in this committee can help increase their credibility with the others on the team not only for her specific skills but also being perceived as someone who listens to others.

3. Ensure accountability: You can certainly include high performance in the interpersonal realm as well. If one of your key values is client service and you have individuals who are rude and inconsiderate to others (internally and externally), you have to have this conversation with the prima donna. Point out that he runs the risk of alienating and distancing himself from others with his actions. Serving as a mirror to his behaviors serves as a start to demonstrate the linkage between his behavior and the actions and behaviors you want from him.

4. Check the “will”: You have to make the determination whether the prima donna is acting the way he is because of his interest in improving your school or if he is simply castigating you because he doesn’t like anybody telling him what to do or actively undercutting what you are trying to achieve. If, in your conversations, you find out that the individual thinks there is a better way to do this, you should certainly listen. On the other hand, if he is doing end runs, generating rumors, and generally trying to undermine you and what you are doing, you have an entirely different (and we hope, quick) resolution to the issue.

What other suggestions do you have in place of or in addition to what I have noted?
Chris

What a tool! This moron's first mistake is thinking that principals help teachers teach. They don't. They don't have time, and many, mine included, have no teaching experience in the grade level they are principal for (my principal was a middle school English teacher, now she is principal of an elementary school, and has to be taught what younger children need).

The second mistake Chris the principal makes is thinking that teachers want principal input. We don't (unless the principal is a great teacher, which we wouldn't know)! And that does not mean we don't want our students, school, colleagues, or district to succeed. We just want to be the professionals that we are, and do our job without hindrance, unnecessary committee responsibilities, and fear of reprisal from power-hungry, career-minded principals.

This screed by principal Hitch should be looked at as part of the problem in education; professional teachers being reviewed, retained, or fired by folks whose mission it is to foster conformity to a norm not yet affirmed, in a desire to wrest control of education away from those who provide it--teachers! Reminds me of HMO's power over doctors, which is waning. When will principal overwroughtness wane?

Principals Are "Instructional Experts" Don't Ya Know!

This is beautiful. Over at Principal's Policy Blog they have a link to a private Yahoo group where they are voting on the efficacy of online teacher instruction. Read this quote:

The U.S. Department of Education recently released Evaluating Online Learning: Challenges and Strategies for Success, a guide that addresses challenges faced by school leaders with the implementation of online learning such as how to meet the needs of various stakeholders, how to solve data collection problems, and how to translate evaluation findings into action.

With the rapid increase in students' taking courses online, the release of such a document is timely.

But principals, while not Internet technologists, are instructional experts and likely have already formed educated opinions about the quality of the online courses available to their students. [emphasis mine]

That's right. Even though principals (or anyone, really) have no business opining on stuff they have no expertise in, they are being asked to opine about stuff they have no expertise in!

This myth that principals are "instructional experts" flies in the face of reality. They have the same level of teacher education as any other teacher, at least in California. They had to take a couple extra MANAGEMENT and LEADERSHIP courses to get their supplemental Administrative credential. But, they have no more instructional expertise than me, or the first year teacher down the hall. Indeed, most principals have been out of the classroom so long, many of them have forgotten how to teach, and are pretty unfamiliar with lots of the problems teachers encounter on a daily basis.

Jeebus I wish leaders would defer to the expertise of the experts (teachers) and stop tooting their own leadership horns and inviting only a select few--the wrong few--to make policy.

(I have taken 2 online teacher courses. They were ridiculous. They took about an hour or two of real work--research, writing--and I got my A. They are useless. They make money for the online university. I know this not because I am an expert, but because I have taken one, or two. Oh, and I don't think much of Teacher Certification programs either. They suck too--mine did!)

The Correct Type of "Performance Review": Be f'n Amazing!

This little gem, and the graphic that goes with it (shown here) is a very smart take-down of the "improve your weaknesses" argument in any performance review. Instead, we should focus on our strengths and how to fill our weaknesses by using others' strengths. Read it!

What Can Other Professions Teach Us about Evaluation and Accountability in Education?

Check this out, from eduwonkette...

Now updated with article and links

What Can Other Professions Teach Us about Evaluation and Accountability in Education?

In a very productive exchange, Dean Millot and Corey Bower have been contemplating the professional status of education. Dean's most recent post, "Why Legally Recognized Professionalism is Necessary to Reasonable Teacher Accountability," is one of the best think pieces I've read in some time. Read the whole thing, but here's the central theme of the post:

Lawyers and doctors are not punished for undesired outcomes; they are accountable for doing what professionals should do given their client’s circumstances....As a legally recognized profession, teacher conduct would be judged by teachers, according to standards of educational care devised by teachers, applied to the client circumstances in question.

Dean's post links well with AEFA conference talks by Randi Weingarten and Richard Rothstein last weekend. Weingarten also drew on the medical metaphor to argue that "teachers are physicians of the mind." In her view, there is a difference between the most skilled physician and a miracle worker. Just as the best hospitals can't solve public health crises on their own, Weingarten argued that, "schools cannot beat back all personal, social, and economic challenges that kids have." In an op-ed last week, she also endorsed a professional standard similar to that proposed by Dean:

[Teachers] should be assessed on how they use test scores and other data to adjust their teaching to help students improve....The approach is akin to judging doctors on how they use the results of blood tests, X-rays, and the like to prescribe a course of treatment.

In his talk, Rothstein drew on the experience of more fields than I can name (business, medicine, public works, etc). Despite many leaders' calls for education to mimic the private sector, Rothstein's review concluded that "private sector performance incentives rely primarily on subjective evaluations, not easily corrupted quantitative measurements." The central theme of the talk was that systems of measurement distort the processes they are intended to measure. The paper on which the talk is based - "Holding Accountability to Account: How Scholarship and Experience in Other Fields Inform Exploration of Performance Incentives in Education" - is a comparative/historical tour de force, and a must read if you're interested in the evaluation question.

Blog posts without positions generally fall on their face, but I still have more questions than answers about Dean's proposal. Here are the two questions I'm pondering:

* How do the processes of diagnosis, inference, and treatment in education differ from those in medicine and law, and what are the implications of these differences for "professional accountability?"

* How does the state of our knowledge about educational diagnosis and treatment differ from that in other professions?

I Know they Don't Matter, But...

So, the CST scores are not to be used in any evaluation of teacher performance. Too bad for me.