Evaluation of Effectiveness: Massachusetts

Analysis of Massachusetts's policies

Although the state requires student performance data to be a factor, Massachusetts does not require that objective evidence of student learning be the preponderant criterion of its teacher evaluations. The state requires districts to either adopt the model system or develop one of their own that is consistent with the state's framework. 

Massachusetts requires its teacher evaluations to include "multiple measures of student learning, growth and achievement" as one category of evidence in teacher evaluations. The state defines these measures as student progress on classroom assessments that are aligned with the state's Curriculum Frameworks; student progress on learning goals; statewide growth measures, including the MCAS Student Growth Percentile and the Massachusetts English Proficiency Assessment (MEPA); and district-determined measures of student learning across grade or subject. Student feedback is also required. 

The summative evaluation includes the evaluator's judgment of the teacher's performance against performance standards and the teacher's attainment of goals set forth in the teacher's plan. Four rating categories must be used: exemplary, proficient, needs improvement and unsatisfactory. To be rated proficient overall, teachers must at least be rated proficient on the "Curriculum, Planning and Assessment" and "Teaching All Students" standards. 

In addition to the summative performance rating, an impact rating of high, moderate or low is also determined based on at least two state or districtwide measures of student learning: the MCAS Student Growth Percentile and the Massachusetts English Proficiency Assessment (MEPA), when available, as well as additional district-determined measures. The impact rating is discrete from the summative performance rating. 

Classroom observations are required. 

Citation

603 CMR 35.00

Recommendations for Massachusetts

Require instructional effectiveness to be the preponderant criterion of any teacher evaluation. 
Massachusetts falls short by failing to require that evidence of student learning be the most significant criterion. By keeping the impact measure wholly separate from the performance rating, it isn't clear that it is really a factor at all. 

The state should either require a common evaluation instrument in which evidence of student learning is the most significant criterion, or it should specifically require that student learning be the preponderant criterion in local evaluation processes. This can be accomplished by requiring objective evidence to count for at least half of the evaluation score or through other scoring mechanisms, such as a matrix, that ensure that nothing affects the overall score more. Whether state or locally developed, a teacher should not be able to receive an effective rating if found ineffective in the classroom. 

Ensure that evaluations also include classroom observations that specifically focus on and document the effectiveness of instruction. 
Although Massachusetts requires classroom observations, the state should articulate guidelines that ensure that the observations focus on effectiveness of instruction. The primary component of a classroom observation should be the quality of instruction, as measured by student time on task, student grasp or mastery of the lesson objective and efficient use of class time.

State response to our analysis

Massachusetts asserted that its evaluation framework is more nuanced than the comments included in this analysis suggest. The framework consists of two separate, but linked, ratings: the summative performance rating and the student impact rating. Instructional effectiveness is at the center of both ratings. 

Massachusetts pointed out that each educator is assigned a summative performance rating at the end of the five-step evaluation cycle. This rating assesses an educator's practice against four statewide Standards of Effective Teaching, as well as an educator's progress toward attainment of her or his student learning and professional practice goals. The evaluator classifies the teacher's “professional practice” into one of four performance levels: exemplary, proficient, needs improvement or unsatisfactory, and uses her or his professional judgment to determine this rating based on multiple categories of evidence related to the four standards and the educator’s progress toward meeting her or his goals. Evidence includes classroom observations and artifacts of instruction; multiple measures of student learning, growth and achievement; and student feedback. Instructional effectiveness plays a significant role in the summative performance rating in two ways. First, multiple measures of student learning, growth and achievement are a required source of evidence. An evaluator will review outcomes from student measures to make judgments about the effectiveness of the educator’s practice related to one or more of the four standards. Such evidence may be from classroom assessments, projects, portfolios, or district or state assessments. Second, evaluators must consider progress toward attainment of the educator’s student learning goal when determining the summative performance rating.

Massachusetts further noted that each educator is assigned a student impact rating, which is separate but complementary to the summative performance rating. This rating is informed by trends (at least two years) and patterns (at least two measures) in student learning, growth and achievement as measured by statewide growth measures (student growth percentiles, or SGPs), where available, and district-determined measures (DDMs). With the student impact rating, the evaluator applies her or his professional judgment and analyzes trends and patterns of student learning, growth and achievement to determine whether the educator’s impact on student learning is high, moderate or low. Each educator is matched with at least two measures each year to generate the data necessary for evaluators to determine student impact ratings. MCAS student growth percentiles must be used where available. Districts identify all other measures locally. Instructional effectiveness is a significant factor in the student impact rating, as the rating is wholly derived from the evaluator’s judgment of student outcomes from multiple measures of learning, growth and achievement.

Massachusetts contended that it eschews weights and algorithms in favor of professional judgment because that provides a more holistic and accurate understanding of educator effectiveness than systems that rely on formulas and “scores.” The state also believes that its framework guarantees, through its reliance on professional judgment, that educators who are not making expected levels of instructional impact will be identified and provided the targeted support necessary for rapid improvement. In cases where an educator is not able to improve quickly, the framework provides him or her with sufficient evidence to recommend dismissal. Massachusetts cautions NCTQ from advocating for evaluation systems that include weights or formulas that result in test scores or other measures of student outcomes comprising 50 percent of an educator’s rating without regard for the professional judgment of evaluators. As we have seen now in several states with such systems, the error introduced into an evaluation system when the underlying measures have significant year-to-year variation (error that is not equally distributed across all educators) can result in misaligned conclusions that are then collapsed into a single score. This misalignment crushes educator morale and shakes confidence in the evaluation process. The Massachusetts framework, with its two ratings, allows evaluators and educators to examine discrepancies in practice and impact, which is impossible in most systems.

Massachusetts added that its framework requires frequent observations to derive evidence of practice related to the standards, which are comprised of elements of practice that are largely instructional.

Last word

In its response, Massachusetts "cautions NCTQ from advocating for evaluation systems that include weights or formulas that result in test scores or other measures of student outcomes comprising 50 percent of an educator’s rating without regard for the professional judgment of evaluators." However, NCTQ does not advocate for a particular weight or formula when it comes to the incorporation of student growth into a teacher's evaluation score. By requiring student growth to be the preponderant criterion in local evaluation processes, however, states are able to ensure that teachers cannot receive an effective overall rating if found ineffective in the classroom.  This is achieved by states in a variety of ways, many of which lack a particular weight or formula.  

Further, NCTQ is not opposed to a two-rating system. Oklahoma recently passed legislation that now requires teachers  to get two evaluation scores: a qualitative one and a quantitative one. Even though an overall summative score is no longer provided, the state maintains the significance of student growth by explicitly addressing the quantitative scores in its requirements regarding personnel decisions. For example, teachers must earn both qualitative and quantitative scores of superior for two of three years to earn tenure, and they are eligible for dismissal if they receive a qualitative or quantitative rating of ineffective for two consecutive years.