If there's one thing we know about teacher evaluations, it's that they're only as effective as the people doing the evaluating. That's why smart districts, in addition to selecting good evaluation instruments, try to guarantee school administrators have what it takes to provide teachers with high-quality feedback.

However, a lesson from Boston Public Schools, as explored in a new study from Matthew Kraft and Alvin Christian (Brown University), shows that training evaluators is easier said than done. In 2013, BPS rolled out an evaluator training program aimed at helping school administrators build trusting relationships with teachers, establish a shared vision for effective teaching, and support teachers to set goals for improvement. The district worked with a national expert on teacher professional development and relied on high-performing school administrators to help tailor the evaluator training series.

Despite making some smart implementation choices, the training series mostly fell flat. The two-year study of the program found basically no impact on student achievement, teacher evaluation ratings, or teacher retention.

Kraft and Christian point to a few shortcomings of the evaluator training program. For starters, attendance at the training sessions was a bit spotty, with only 40 percent of administrators who had been assigned to receive the training in the first year showing up to every session. Then, and perhaps most importantly, there is the nature of the training itself, which provided no functional tools for improving administrators' knowledge of what good teaching looks like.

There's a lot that goes into being a good mentor—including the sort of qualities Boston was looking to build—but to help teachers become better instructors, you've got to have concrete knowledge about classroom practice. For teachers, feedback on how to encourage good behavior, use visual aids, or ask formative questions can go a long way. Training administrators to have these kinds of conversations might go a long way too.