An Application of Hierarchical
Linear Modeling to the Estimation
of School and Teacher Effect

William J. Webster, Robert L. Mendro,
Timothy H. Orsak, and Dash Weerasinghe

Dallas Independent School District

This paper provides a concise summary of ten years of research into defining appropriate statistical models for estimating school and teacher effect on student learning and other important educational outcomes. It discusses criteria for judging models and presents the formulae for the two-stage, two-level student-school HLM model that is the model of choice for estimating school effect and the two-stage, two-level student-teacher HLM model that is the model of choice for estimating teacher effect. Finally, a brief discussion is provided on criterion variables and the methods by which they are weighted, predicted, and aggregated in the school and teacher effectiveness system.

The need for instructional improvement in the Dallas Independent School District had been thoroughly documented over a period of twenty years. After a period of rapid achievement growth in the early and mid 1980's, student achievement in the Dallas schools had leveled out. In 1990, responding to this need, the District's Board of Education appointed a citizen's task force, the Commission on Educational Excellence, to formulate recommendations to accelerate the needed improvement. After a year of community hearings and extensive study, the Commission recommended a six point plan for massive educational reform. At the heart of the Commission's recommendations was an accountability system that fairly and accurately evaluated schools and teachers on their contributions to accelerating student growth in a number of important and valued outcomes schooling. This was coupled with a movement to give schools more decision-making authority about personnel, curriculum, and most other aspects of schooling. In exchange for this authority, school staffs were to be held accountable for their actions. As part of this recommendation, $2.4 million was set aside as an incentive award to reward effective schools, and their professional and support staffs.

It then became the task of the District's Research, Planning, and Evaluation Department to develop, pilot test, and implement an evaluation system to accomplish the goals of the Commission. The first step in accomplishing this task was the appointment of an Accountability Task Force to oversee the process. This task force, consisting of teachers, principals, parents, members of the business community, and central office administrators, was charged with the responsibility of advising the General Superintendent concerning the implementation of a performance incentive plan, working with the administration to ensure the validity of the selection procedure and subsequent results of the incentive plan, and serving as a review committee to examine any issues raised by personnel concerning questions of equity and fairness of the procedures. During a year of exhaustive deliberations, a number of requirements for the methodology associated with this plan were developed. Among these were:

1. It must be value-added.

2. It must include multiple outcome variables.

Performance indicators for 1997-1998 include Iowa Tests of Basic Skills and Tests of Achievement and Proficiency reading and mathematics, grades 1-9; Spanish Assessment of Basic Education, grades 1-6; Texas Assessment of Academic Skills, reading and mathematics, grades 3-8 and 10; writing, grades 4, 8, and 10; science and social studies, grades 4 and 8; Texas Assessment of Academic Skills, Spanish version grades 3-6; 143 standardized final examinations in language, mathematics, social studies, science, ESOL, reading, and world languages, grades 9-12; promotion rate, grades 1-8; student attendance, grades 1-12; graduation rate, grades 9-12; Scholastic Aptitude Test percent tested and scores, grades 9-12; dropout rate, grades 7-12; student enrollment in prehonors/honors courses, grades 7-12; student enrollment in advanced diploma plans, grades 9-12; students enrolled in advanced placement courses, grades 11-12; Preliminary Scholastic Aptitude Test percent tested and scores, grades 9-12; and percent passing Advance Placement Exams, grades 11-12. The system is operated with only continuously enrolled students and includes staff attendance incentives, minimum percent eligible tested requirements, and requirements that at least one-half of a school's cohorts must outgrow the national norm group on the ITBS and TAP in reading and mathematics.

3. Schools must only be held accountable for students who have been exposed to their instructional program (continuously enrolled students).

4. It must be fair. Schools must derive no particular advantage by starting with high-scoring or low-scoring students, minority or white students, high or low socioeconomic level students, or limited English proficient or non-limited English proficient students. In addition such factors as student mobility, school overcrowding, and staffing patterns over which the schools have no control must be taken into consideration.

5. It must be based on cohorts of students, not cross-sectional data.

Within the five aforementioned parameters, a number of statistical models are possible. This study examines alternative methodologies for determining school effect then extends these studies to the determination of teacher effect. These models are designed to isolate the effect of a school's or teacher's practices on important student outcomes. The school effect can be conceptualized as the difference between a given student's performance in a particular school and the performance that would have been expected if that student had attended a school with similar context but with practice of average effectiveness. The teacher effect can be conceptualized similarly at the teacher level.