Background

In Ontario, standardized assessment of student achievement has for the past nine years been the responsibility of EQAO (Education Quality and Accountability Office), an independent Crown corporation. There are four different assessments:

- Grade 3 Reading, Writing and Math (administered to all rade 3 students);

- Grade 6 Reading, Writing and Math (administered to all Grade 6 students);

- Grade 9 Math (administered to students participating in Grade 9 Math courses in the Academic and Applied programs of study);

- Grade 10 Literacy Test, the Ontario Secondary School Literacy Test, or OSSLT, administered for the first time to students in Grade 10, or the second year of secondary study.

In 1998-1999, student-level data of all board students was, for the first time, electronically provided back to local school authorities, called District School Boards.  This data served as the base for an eight-year cohort evaluation, going from Grade 3 through Grade 8.

Significance of the Dataset

The importance of cohort studies in school effectiveness and accountability research has been recognized in the literature (e.g. Raudenbush and Willms, 1991. Because Ontario did not have, until recently, common Ontario student ID numbers (OEN's) EQAO itself could not track how the same students would achieve on different tests. This is the first examination of a cohort participating in all four assessments, across eight years, going from the elementary into the secondary panel.

The Diversity of Toronto

The Toronto District School Board is considered one of the most diverse in the world. It is the largest school district in Canada and the fifth largest in North America. The complexity is not so much with its size as its complexity. Nearly a third (31%) were born outside Canada in more than 175 countries. Over 70 non-English languages are spoken at home by about half the population. The Board has close to 100,000 students whose families live in poverty. The proportion of lone parent families in the city is significantly greater than the Ontario population in general. Perhaps not surprisingly, in Toronto there has been a tradition of establishing social programmes to assist students in which is presumed to be an ultimate social and educational benefit.

The Importance of Grade 10 OSSLT

We have followed students who wrote the literacy test (OSSLT). Those who wrote the test and passed it the first time had the lowest dropout rate and highest graduation rate, while those who did not pass had much lower graduation rates and much higher dropout rates. Therefore, as a predictor of high school success, the OSSLT is a very powerful variable.

Preliminary Findings—An Example

There were 11,141 students who participated in both the Grade 3 Math Assessment in 1998-1999, and who were in Grade 9 in the TDSB during 2004-5, or 58% of the original assessment population. Much of the 42% non-match could be attributed to the high mobility of the TDSB population.

 

In looking at Grade 3 and Grade 9 Math assessment results, two key trends were observed:

1. Most students who were at Levels 3 and 4 in Grade 3 Math (the two higher-performance levels) were also at Levels 3 and 4 in Grade 9 Math. However, in looking at Level 4—the highest level of performance—less than a fifth of students in Grade 3 were also at Level 4 in Grade 9. Thus, early elementary Math performance is a general-- but by no means precise-- indicator of future Math performance.

2. Performance in Grade 3 Reading appeared to be a slightly better predictor of Grade 9 Math performance, than performance in Grade 3 Math (that is, 72% of students at Levels 3/4 in Grade 3 English were at Levels 3/4 in Grade 9 Math, while only 69% of students at Levels 3/4 in Grade 3 Math were at Levels 3/4 in Grade 9 Math). This indicates that the difference between subject specific assessments may not be as great as generally perceived.

 

Directions for More Detailed Analysis

Method

Our study modeled the school and district effects in the Literacy and Math scores of the Education Quality and Accountability Testing program using Hierarchical Linear Modeling. Principal Component Analysis (PCA) of the test scores from the EQAO Reading and Writing were conducted to derive a composite score as the outcome measure of overall student literacy achievement. Three-level hierarchical models were fitted to estimate school and district effects in EQAO Literacy and Math scores, and to examine the school and district variance with variables assessing student characteristics.

Model specifications and estimation procedures

Three-level linear regression models were applied to the four data sets described above. The dependent variables were the literacy and Math achievement levels, reported on a scale that runs from 0 to 4.

Variables of student background information were specified as independent variables at each level. At the student level, dummy variables were created to indicate country of birth, ESL, gender, and special education status. Grade 3 literacy and math scores in 1998-99 were used as an estimate of student previous literacy and mathematics achievement.

It was also known from preliminary analyses that approximately 20% changed schools while in the elementary panel. One more dummy variable was thus created to indicate whether a student had made changes in his/her registrations between grade 3 and grade 8.

The variables specified at school level included the size of school in terms of the number of students, and the proportion of the student in a school who were born in Canada, female, male or identified as special education students. The school’s average SES status was assessed by the percentage of students in that school who have low average family income.

At the district level, the variables included the district size (the number of students), the proportions of students in the district who were born in Canada, female, male or identified as special education students, and low SES.

Reference:

Raudenbush, S.W., Willms, J.D. (1991), Schools, Classrooms, and Pupils: International Studies of Schooling from a Multilevel Perspective, Academic Press, San Diego, CA.