Proposal view
| Proposal Type: | Individual Paper |
|---|---|
| Domain: | Teaching and Teacher Education |
| SIG: | Teaching and Teacher Education |
| Type | Submitted Paper |
| Equipment |
PC and projector |
| Paper Details |
|---|
| Title | Teachers’ judgments of students’ computational strategies and skill: How accurate, how well calibrated, and how important are they in determining instructional effectiveness? |
|---|---|
| Abstract | In this study we examined: (1) the accuracy of teachers’ judgments of students’ computational proficiency, (2) whether the confidence teachers have in their judgments is well calibrated with the accuracy of their judgments, and (3) whether the accuracy and calibration of teacher judgments is associated with classroom learning. Nineteen 1st and 2nd grade teachers were asked to make predictions about the current computational performance of a sample of randomly selected students from their classrooms. Data on the computational performance of each teacher’s entire class was concurrently collected. Results revealed that on average, teachers’ judgments were accurate and well calibrated. However, only calibration of teachers’ judgments predicted student learning. The theoretical significance and practical implications of this work will be discussed. |
| Summary | Aims: The accuracy with which teachers judge their students’ academic performance is likely to play a significant role in teacher instructional decision-making. It is generally assumed that the more accurate a teacher’s judgments are, the more likely they are to make appropriate selection, grouping, as well as interactive instructional decisions such as pacing of a lesson. Previous research that has examined the accuracy of teachers’ judgments of students’ concurrent academic performance indicates that teacher judgments are significantly correlated with their students’ performance (Hoge & Coladarci, 1989). However, several methodological concerns have been raised with these studies. For example, most studies infer judgment accuracy from the size of correlation coefficients. However correlations indicate the degree of correspondence between the relative position of two sets of values, not the degree of similarity and therefore do not necessarily provide a measure of teacher judgment accuracy. Others have voiced concerns regarding reliance on standardized norm-referenced tests, arguing that test items may not align well with the curriculum, therefore compromising the content validity needed to properly evaluate teacher judgment accuracy. Studies attempting to address one or both of these concerns paint different pictures of the accuracy of teacher judgments. Coladarci (1986) asked teachers to predict student performance on each item of a standardized achievement test and found that teachers’ judgments were on average 75% accurate in predicting student performance on those items. Other studies, using curriculum-based measures, report reasonably high correlations between teacher judgments and student performance but large discrepancies between teachers’ estimates and students’ actual performance (Bates & Nettlebeck, 2001; Eckert, et. al., 2006; and Feinberg & Shapiro, 2003). However, in these studies teachers were asked to make predictions in metrics that are often unfamiliar (e.g., words read correctly per minute) and therefore may underestimate teacher judgment accuracy by using metrics not routinely used for making instructional decisions. Finally, studies have rarely investigated the relationship between teacher judgment accuracy and measures of instructional processes or effectiveness. One notable exception was a study conducted by Carpenter et al., (1988). Teachers were asked to predict the performance of a sample of students on six arithmetic word problems. Results showed that teachers were fairly accurate (75% correct) at judging students’ performance. More importantly, they reported that teacher judgment accuracy was significantly correlated (although weakly, r=.32) with classroom achievement. In our study, we extend this line of work. We looked at teachers’ judgments about students’ computational performance and relate these to measures of instructional effectiveness. In addition, we explored an issue that has been ignored: the degree to which teachers’ judgments are well calibrated. Calibration, which has been generally defined as the degree to which confidence in judgments of performance accurately reflect actual performance, is a metacognitive measure that has been used to index how well an individual knows what they know as well as what they do not know (Stone, 2000). Although calibration research has primarily explored the relationship between an individual’s self-judgments of performance and their own actual performance, there is no reason why the same technique could not be used to evaluate the judgments teachers make of their students. We speculate that being well calibrated should provide teachers with advantages in regulating their instructional decisions about students and taking adaptive action to improve student performance. We therefore expect that measures of the calibration of teacher judgments should be better predictors of classroom performance than teacher judgment accuracy. To summarize, in this study we examine: (1) the accuracy of teachers’ judgments of students computational proficiency, (2) whether the confidence teachers have in their judgments is well calibrated with the accuracy of their judgments, and (3) whether the accuracy and calibration of teacher judgments is associated with overall classroom learning. Methods: Nineteen 1st and 2nd grade teachers were interviewed and asked to make predictions about the current computational performance of 5 randomly selected students from their classroom. Teachers were shown four sets of 10 basic fact problems (i.e., ‘easy’ and ‘hard’ addition and subtraction basic facts) and asked to predict how many problems a specific student would answer correctly as well as how confident they were in their prediction. Teachers were also asked to predict and rate the confidence of their predictions with respect to the number of times the same student would use: their fingers, retrieval, counting and derived-fact strategies on the same set of problems. To assess the accuracy and calibration of teachers’ judgments as well as the overall computational performance of each teacher’s class, all students in each teacher’s class were individually interviewed within a week of the teacher interview to assess their actual competences and strategy use using the same 40 problems shown to teachers. From these teacher and student data we created two types of scores: teacher prediction accuracy scores and calibration accuracy scores. Findings: Space limitations preclude a full description of our analyses. In brief, we found that: teachers’ judgment accuracy scores were relatively high for predicting both student performance (82%, SD=4.3) and strategy use (82%, SD=5.2); teachers were well-calibrated in their judgments about student performance (84%, SD=5.7) and strategy use (81%, SD=5.5); and calibration scores but not prediction accuracy scores were significantly correlated with classroom average residual gain scores (r=0.58, p=.01). Step-wise regression analyses revealed that calibration score was the best predictor of student learning among other predictor variables (prediction accuracy, years of teaching experience, and class size). Finally, teachers’ calibration and prediction accuracy scores were not significantly related (r=.22, n.s.), suggesting that teachers who were more accurate in judging their students were not necessarily better calibrated. Educational Significance: Our results suggest that with respect to instructional effectiveness, accuracy of teachers’ judgments may not be as important as the extent to which a teacher’s confidence in their judgments about student performance are aligned with the accuracy of this judgment. The theoretical significance and practical implications of this work will be discussed. Future work should seek to replicate these findings, explore how it relates to instructional processes, and study how it can be enhanced through professional development. |
| Keywords | Mathematics education Metacognition Teacher thinking |
| Appendices | |
| Authors | ||||||
|---|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | Presenting | |
| Anthony | Gabriele | University of Northern Iowa | United States | gabriele@uni.edu | * | |
| Kim | Knesting | University of Northern Iowa | United States | knesting@uni.edu | ||
| Shawna | Feldman | University of Northern Iowa | United States | sfeldman@aea1.k12.ia.us | ||
