This symposium is devoted to the relationship between metacognition and mathematical problem solving skills. Despite all the emphasis on metacognition, researchers currently use different concepts for overlapping phenomena. The purpose of this contribution is to help to clarify some of the issues on the conceptualization, the assessment and the training of metacognition.

Research on mathematics teaching and learning has recently moved away from purely cognitive variables. Metacognition and many of its dimensions such as self-representation, self-awareness, self-evaluation and self-regulation have been receiving increased attention in cognitive psychology and mathematics education. The present study concentrated on the impact of pupils’ self-representation on the metacognitive strategies they use in order to self-regulate their cognitive performance while trying to solve mathematical problems and on their real mathematical performance on different domains such as counting, geometry and statistics. Participants were 114 pupils (5^th grade). Three inventories were developed for measuring pupils’ self-representation in mathematics, their performance and the metacognitive strategies they use in problem solving. Results indicated that pupils with high self-representation in mathematics have high performance in specific domains of mathematics, they are more autonomous in the learning procedure and they insist in encountering difficulties.

Recent research in mathematics education has focused on the phenomenon of metacognition, its main dimensions such as self-representation, self-regulation, self-awareness, self-evaluation and their role in learning behavior, mathematical performance, and especially problem solving (Lerch, 2004). Learning mathematics, as an active and constructive process, implies that the learner assumes control and agency over her own learning and problem solving activities (DeCorte, Verschaffel & Op´t Eynde, 2000). The present study concentrates on self-representation in mathematics and its impact on mathematical performance. “Self-representation refers to how the individual perceives himself in regard to a given disposition, style, type of activity or dimension of ability” (Demetriou & Kazi, 2001, p.33).

Many researchers have argued that the development of metacognitive awareness and the precise self-representation are key factors to successful learning. Those characteristics guide the quality of interactions children have with the intellectual and social activities they encounter (Davis & Carr, 2002). For instance, Jacobson (1998) concludes that metacognition is vital to the renovation of a current educational system, while Montalvo and Torres (2004) advise that education should help students develop awareness of their own modes of thinking, to be strategic, and to direct their motivation towards valuable goals. Especially in problem solving tasks a balance between cognitive and metacognitive processes is necessary. The metacognitive aspect of problem solving needs to be expanded in order to include the problem solver´ s self-representation, as a mathematical being (Lerch, 2004). Lack of confidence and previous lack of success combined to prompt swift decisions to stop working.

Although there are a lot of studies dealing with pupils’ self-representation in relation to their school performance (Demetriou & Kazi, 2001; Demetriou, Kazi, & Georgiou, 1999), there is more to be explored, particularly on the role of these processes at specific domains, such as mathematics. The interest is concentrated on the impact of self-representation on the strategies pupils use in order to self-regulate their cognitive performance while trying to solve mathematical problems and on their real mathematical performance on different domains such as counting, geometry and statistics. The present study is a part of a project concerning the development of pupils’ performance on problem solving by enhancing their ability to concentrate on problems’ similarities in the underlying structure or principles involved in the problem and the development of their self-regulation. At this presentation we concentrated only on the impact of self-representation on mathematical performance and on the metacognitive strategies they use in order to solve problems. We believe that the results could be useful in two ways: On the theoretical level it would contribute to deeper understanding of important interconnection between metacognitive processes and on the practical side it might be useful in developing interventions for the improvement of self-representation, self-awareness and self-regulation in respect to mathematics. 

Participants were 114 pupils (5^th grade). Three questionnaires were developed: The first one measured pupils’ self-representation. It comprised of 40 Likert type items, of five points reflecting pupils’ behavior regarding mathematical learning. Two specimen items were: “I prefer solving problems that present the data with diagrams or tables” and “I can easily imagine the picture which is on a deflated balloon”. The second questionnaire comprised of 20 mathematical tasks on counting, geometry and statistics. The third questionnaire comprised of ten couples of sentences and pupils had to choose which one expressed better their cognitive behavior. An example of a couple of sentences is the following: (a) When I encounter a difficulty while I am solving a problem I ask for teacher’s help, or (b) When I encounter a difficulty while I am solving a problem I insist until to overcome it.

The data about self-representation were first subjected to exploratory factor analysis. This analysis resulted in 6 factors concerning pupils’ self-representation:

F1: self-representation in solving specific mathematical tasks

F2: general self-representation in mathematics

F3: self-representation in rethinking a solution of a problem

F4: preferences for the use of diagrams and representations in mathematics

F5: self-representation in their concentration

F6: self-representation in the impact of teacher at the learning

In order to examine the impact of pupils’ self-representation on their performance and the style of thinking they prefer or the self-regulatory strategies they use, pupils were cluster according to their self-representation regarding the abovementioned factors. Analysis of variance (ANOVA) indicated that pupils with high self-representation in solving mathematical tasks (F1) had at the same time high performance in statistics tasks (F(2,113)=9.332, p<0.001). Almost the same were the results in the case of factors F2 and F3. In the case of the F4, results indicated that pupils who preferred to use diagrams in mathematics had higher performance on the geometrical tasks (F(2,113)=9.902, p<0.001) and on the statistics tasks (F(2,113)=3.871, p<0.05) than other pupils. There were no differences in the case of the factors F5 and F6. At the same time analysis of variance indicated that pupils with high self-representation in solving mathematics tasks (F1) used more strategies for self-regulating their cognitive behavior (F(2,113)=9.158, p<0.001). Similar were the results for pupils with high performance on F2, F3 and F6.

            Results confirmed that pupils with high self-representation in the cognitive and metacognitive processes have high performance in mathematics. However, they have high performance in specific domains of mathematics, such as statistics and in some cases in geometry, where the tasks, which were used at the inventory, were more difficult than the counting tasks. The most important result concerns the strategies they use in order to regulate their performance on mathematics. Pupils with high self-representation tend to be more autonomous in the learning procedure, they insist in encountering difficulties and they use the strategies teachers usually learn them for problem solving. 

Effects of two online inquiry based learning in mathematics are compared: Online inquiry based either to meta-cognitive feedback guidance (MFG) or to no such guidance (NG). The MFG students were exposed to IMPROVE meta-cognitive questioning that serve as cues for solving the problem and features of providing feedback (Kramartski & Mevarech, 2003).

A total of 79 eight-grade Israeli students participated in the study. Students were  asked to solve online a real-life task and provide feedback to their peers on the solution process. Results indicated that the MFG students significantly outperformed the NG students on online   problem solving task, and using conceptual arguments. In addition, the MFG students provided more often mathematical and meta-cognitive feedback by referring to various measures as: Providing mathematical terms, and representations, identifying errors, and clarity of mathematical communication. Theoretical and practical implications of the study are discussed.

Rapid advances in computer technologies have facilitated the development of electronic tools that have in turn, expanded the opportunities to empower inquiry learning. Inquiry learning is the process of students being engaged in learning in which they pose questions and construct solutions (e.g., Schraw, Crippen and Hartley, 2006 ).  Although at face value the potential of these opportunities is compelling, research shows that students are usually not in fact “mindfully engaged” when it comes to learning with advanced computer technologies. The purpose of this study is to evaluate the effectiveness of a self-regulatory support called IMPROVE in assisting students’ problem solving and mathematical discourse with online collaborative inquiry learning environment.

IMPROVE meta-cognitive guidance

Azevado, Guthrie & Seibert (2004) summarized three commonalities in models of self-regulation with online stating: (a) define the specific processes and strategies used by students to improve academic achievement, (b) is a cyclical and recursive process with utilizes feedback mechanisms for students to monitor their learning and adjust accordingly, and (c) including a description of why and how students select a specific self regulatory strategy, approach or response within learning. In addition, planning, monitoring, and evaluation are  specify within the cyclical process of self regulation.

Research indicated that students unsupported self-regulation performed very few of the  activities discussed in the framework above. This result taken with the complexity of tasks students are expected to perform in inquiry learning environments indicates the need for explicit meta-cognitive guidance as self-questioning and providing feedback.

The IMPROVE meta-cognitive guidance (Mevarech and Kramarski, 1997; Kramarski, & Mevarech, 2003) encourages students to be involved in mathematical collaborative self-regulated learning by focusing on questioning about comprehending the problem (e.g., “What is the problem/task?”); constructing connections between previous and new knowledge (e.g., “How is this problem/task different/similar to what you have already solved?, Explain why”); using strategies for solving the problem (e.g., “What strategy/tactic/principle can be used in order to solve the problem/task?" and why”), and reflecting on the processes and the solution (e.g., “What am I doing?”; “Does the solution make sense?”). Students referred to these guidance during their turn to solve the task, the discussion about the solution, and in providing feedback regarding peers’ solutions.

The purpose of the present study is twofold: To investigate the ability to solve online real-life tasks of students’ who were exposed either to meta-cognitive feedback guidance  (MFG) or with no such guidance NG; and  (b) to examine the online discourse of students’ who were exposed to these instructional guidance with regard to mathematical and meta-cognitive aspects.

Method

Participants

Participants were 79 (boys and girls) ninth-grade students who studied in two  classes within one junior high school. Each instructional approach was assigned randomly to one of the classes. No statistical differences on a mathematical pre-knowledge were found between the two groups (M=83.30; SD=16. 80; M=80; SD=15.70; t(78)= 2.01;p>.05).

Measurements

(a) An online real life task: A real-life task was administrated in online discussion  environment adapted from PISA (2003). Students are asked to investigate patterns in change and relationships by comparing the growth of apple trees planted in a square pattern and conifers trees planted around the orchard and to explain their reasoning.

Scoring: Correct answers were translated to percents. Mathematical explanations were analyzed on two criteria: Conceptual arguments (e.g., logic-formal arguments); and Procedural arguments (e.g., calculation example).

(b)  Online inquiry discourse:  Students’ discourse was analysed by two criteria: mathematical feedback (number of statements, mathematical terms, representations ,and final solution)  and meta-cognitive feedback (errors identification, process description and clarity of mathematical communication).

Scoring: Sum of references provided to each category during the online discussion was calculated.

Instructions:

General online inquiry discussion: Students from both groups (MFG & NG) practiced online problem solving in pairs for a  four-week period once a week in the computer lab (45 min). The teacher encouraged students to be engaged in the discussion by providing mathematical explanations and feedback.

Meta-cognitive feedback guidance (MFG): The MFG focused on the IMPROVE self-questioning method for problem solving. In addition, a discussion regarding the questions: “How to be engaged in online inquiry discourse in mathematics, and what does it mean to provide feedback” was held in the entire class.

Results

We performed a one way ANOVA on real-life task scores. Results indicated that the online MFG students significantly outperformed their counterparts (NG) on mathematical problem solving (M= 86.68; SD= 19.9; M=74.46; SD=26.87; F(1,77)=7.98, p< .001 respectively) and providing mathematical explanations (M=39.93; SD=21.09; M=27.90; SD=21.17; F(1,77)=9.94, p<.001). In addition, we found that at the end of the study more MFG students provided conceptual arguments than the NG students (72%; 50%, t(77) = 3.97, p<.05).

In addition, we performed a MANOVA and an ANOVA on each criteria of mathematical and meta-cognitive feedback. Results indicated that the MFG students significantly outperformed their counterparts (NG) on mathematical feedback regarding three criteria: Using number of statements  (M= 5.44; SD= 3.38; M= 0.94; SD= 1.28; F(1,77)= 29.8, p< .001), mathematical terms (M=1.39; SD= 0.90; M=0.44; SD= 0.50; F=17.82, p< 0.001), and reference to mathematical representations (M= 11.04; SD= 15.71; M= 0.15; SD= 0.02; F (1,77) = 11.15, p<0.001).

Similarly, findings  indicated that the MFG students significantly outperformed their counterparts (NG) on meta-cognitive feedback regarding two criteria: Errors identification  (M= 2.4; SD= 6.11; M= 0.13; SD= 0.04; F(1,77)= 5.39, p< .001), and clarity of  mathematical communication (M= 16.64; SD= 19.18; M= 4.7; SD= 12.68; F(1,77)= 5.53, p< .001).

Discussion and conclusions

Our findings indicated that meta-cognitive feedback guidance (MFG) in an online inquiry learning environment might be a vehicle for students’ mathematical problem solving and discourse. It seems, that IMPROVE guidance integrating with discussion about providing feedback might help students: Access and interact with the content functionality, and  think about the deeper concepts and structure of disciplinary relations. Our study brings a focus on questions such as how does students learn mathematics with technology and what mathematics does they learn. 

Hypermedia environments lately become increasingly popular in educational contexts. Such environments offer a high degree of learner control, which includes the selection and combination of different representational formats on the one hand and the possibility to access information in a linear as well as in a nonlinear fashion on the other hand. While enabling learners to engage in active, constructive, and self-regulated learning processes, hypermedia environments also hold the risk of leading to the assembly of suboptimal information diets, disorientation and accordingly to cognitive overload when learners are overstrained by the decisions they have to make with regard to the contents they want to access and the rate and sequence for retrieving these contents. The question thus is, how hypermedia environments should be designed to foster learning in an optimal way and at the same time avoid the abovementioned disadvantages. Research suggests that the relationship between instructional design and learning outcomes is not a direct one when self-regulated learning is involved but that it is moderated by working memory capacities and other learner characteristics. In line with this reasoning, Gerjets & Scheiter (2003) proposed an extension of the Cognitive Load Theory (Sweller, van Merriënboer & Paas, 1998) that allows applying the theory to hypermedia learning and describes learner activities in terms of goals and strategies as moderators between instructional design, cognitive load, and learning outcomes. It is assumed that learner activities, in turn, are influenced by conceptions and expertise of learners (Gerjets & Hesse, 2004). The focus of the current study is to investigate metacognition, epistemological beliefs, and attitudes as specific aspects of these learner conceptions.

Research suggests that epistemological beliefs and metacognition are related to learner activities especially in the context of solving ill-defined problems, whereas there is only a minor influence when well-defined problems are presented (Schraw, Dunkle & Bendixen, 1995). Hypermedia learning environments, especially those that offer a high degree of learner control, share several features with ill-defined problem structures. Authors like Bendixen & Hartley (2003) thus assume that a high amount of learner control will be especially beneficial for learners with sophisticated beliefs and with a higher level of metacognitive awareness.

Our study investigates the interrelation between these individual characteristics, learner activities, and learning outcomes using a hypermedia learning environment on probability theory. This environment conveys the basic probability principles by means of eight worked-out examples. For each example, learners can choose whether they want to retrieve it with arithmetical information only, enrich it with written or spoken text or animations or use any combination of these. We are using a 2*2 design varying the following factors: metacognitive tool / no metacognitive tool, prompting of representational choices / no prompting of representational choices. The metacognitive tool that is used in two of the four conditions is a video that is displayed at the beginning of the learning phase and aims at supporting the metacognitive monitoring and evaluation of learners. When prompting of representational choices takes place, learners receive explanations of the advantages and disadvantages of the respective representational formats directly before they choose a format for the respective worked-out example they are about to have a look at. Learner behaviour (e.g., navigational choices) can be studied by means of log files, and performance and knowledge gains are measured with both a pretest and a posttest that contain parallel items. Learner characteristics are assessed with a comprehensive questionnaire that is administered before learners start working with the environment. This questionnaire includes items from the following scales: EBI (Schraw, Bendixen & Dunkle, 1995), EBI (Jacobson & Jehng, 1999), Mathematical Beliefs Questionnaire (Schoenfeld, 1989), Cognitive and Metacognitive Strategy Use (Wolters, 2004), Metacognitive Activity Scale (Schmidt & Ford, 2003), ATMI (Tapia & Marsh, 2004), and CAS (Nickell & Pinto, 1986).

The study is currently being conducted and final results will be presented at the EARLI. We assume that learners with more sophisticated beliefs, higher metacognitive awareness and more positive attitudes towards the domain (i.e., mathematics) and the educational technology involved (i.e., computers) will engage more deeply in activities like nonlinear navigation and extensive utilization of different representations. Therefore, they should benefit from a high level of learner control that provides ample opportunities for these activities. Contrary to this, learners with simpler beliefs and less metacognitive awareness might be better off with environments that provide structural help and recommendations for representational choices, that is, they should benefit from the metacognitive tool and the representational prompting. Pilot results from the first 37 participants who were all highschool students with an average age of 16 years only partly confirmed our expectations. Learners generally do not seem to engage in nonlinear navigation and extensive representational choices by themselves, independently of their metacognitive sophistication or epistemological beliefs. However, these missing relationships might also be due to the very small cell sizes. On the other hand, learner characteristics relate to performance. Learners with positive attitudes towards mathematics and more sophisticated beliefs achieve higher knowledge gains and performance scores. For metacognition, however, we surprisingly found that participants scoring lower on the Metacognitive Activity Scale had significantly higher knowledge gains. The question remains why learners who don’t monitor their learning progress and don’t reflect what they’re doing seem to be more successful than those who display more elaborated strategies. During the symposium, the issue will be raised whether this pattern of results reflects a misleading tendency to answer questionnaires, or whether the scales have been suitable to assess metacognition at all. Finally, it was impossible to investigate the influence of negative versus positive computer attitudes on hypermedia learning, because among our participants, there simply weren’t any negative attitudes towards computers.

Taken together, these preliminary results show that not only the design of a learning environment itself is important, but that learning and learning success also depends on individual characteristics and that not all learners benefit from complete navigational and representational freedom. These considerations should be taken into account when developing hypermedia environments.

The purpose of this exploratory study was to examine socially shared metacognition in pre-service teachers' collaborative mathematical problem solving supported by Workmates (WM) networked learning environment. Three matched and one lateral group of three students solved mathematical problems in a two hour session four times during the one-month period. The participants solved open and closed problems requiring proportional or algebraic thinking. A stimulated recall group interview was performed immediately after the problem solving situation. Participants' socially shared metacognition during the process was examined in the discussion forum data by using cognitive, metacognitive and social levels of analysis. The analyses were synthesized in a graph of the joint problem solving process as a function of time and compared with the transcripts of the group interviews. From the surface to deeper level, the qualitative content analysis of the participants' computer notes was carried out and the frequencies were calculated. At the cognitive level, the phases of the groups' mathematical problem solving process analysis, exploration, implementation and verifying were examined. In order to examine participants' metacognition, the computer notes were analysed to find the notes where metacognitive knowledge or metacognitive skills were evident. Further, the social processes of joint problem solving were described using the stages of perspective taking consistent with the mathematical problem solving processes: the subjective role taking, reciprocal perspective taking, and mutual perspective taking. In the transcripts of the group interviews the interviewees' utterances I and we were used to diverging individual and shared metacognitive processes.

Aim and theoretical background

Metacognition refers to an individual’s knowledge about and regulation of cognitive processes in problem solving (Flavell, 1979; Brown, 1987, Veenman, Van Hout-Wolters & Afflerbach, 2006). The possibilities to use technology-based learning environments as tools for enhancing metacognition (Azevedo, 2005) has challenged researchers to focus attention not only to individual’s metacognition (Flavell, 1979; Brown 1987) but also metacognition as a part of the collaborative learning situation (Goos, Gailbraith & Renshaw, 2000; Choi, Land & Turgeon, 2005) and networked learning (Hurme, Palonen & Järvelä, 2006). Although there have been a few attempts to describe socially shared metacognition (Tindale & Kameda, 2000; Vauras, Iiskala, Kajamies, Kinnunen & Lehtinen, 2003; Iiskala, Vauras & Lehtinen, 2004) in reciprocal interaction with peers, more research is needed to examine the complexity of cognitive, metacognitive and social processes especially in networked learning. Consequently, the aim of this exploratory study is to examine socially shared metacognition in pre-service teachers’ (N=12) collaborative mathematical problem solving supported by Workmates (WM) networked learning environment. The joint problem solving process as a function of time was analysed using qualitative content analysis (Chi, 1997) of the participants’ computer notes at cognitive and metacognitive levels with the stages of perspective taking (Järvelä & Häkkinen, 2002). Furthermore, the participants’ own interpretations of the networked problem solving situations were also analysed to reveal the processes possible to be considered as socially shared metacognition.

Method

Participants and procedure

The participants of the study were pre-service teachers having their first year at the university. The participants filled in the self-report questionnaires of their metacognition in the mathematical problem solving and Students Appraisals of Group Assesment, SAGA, (Volet, 1998). They were matched in groups of three students with the assistance of factor, cluster and variance analysis. Each group had one male and two female students.

One lateral and three matched and groups solved open and closed mathematical problems requiring proportional or algebraic thinking in a two hour session four times during four weeks period, January-February 2006. The joint problem solving session took place at the appointed time in the university's computer classroom. The group had their own folders in discussion forum to which the participants logged in by user account and password but they did not have access to view the other groups' discussions. Each joint problem solving session lasted for two hours and the participants communicated only through the computers. Immediately, the problem solving session was finished the participants were gathered around the computer in the interview room for a stimulated recall group interview which was video recorded. One of the group members logged in and the interviewer asked the group to describe their working in WM as precisely as possible.

Data collection and data analysis

The data of this study consist of discussion forum data, that is the participants’ posted computer notes saved in Workmates, WM learning environment database and the transcripts of the video-recorded stimulated recall group interviews.

Socially shared metacognition was examined in the discussion forum data by using cognitive, metacognitive and social levels of analysis. From the surface to deeper level, the qualitative content analysis (Chi, 1997) of the participants' computer notes was carried out and the frequencies of the sent notes were calculated. The unit of the analysis was a meaningful section of the computer note. One computer note could include several sections and it was possible to characterize the computer note in more ways than one. First, at the cognitive level, the computer notes were classified to mathematical knowledge, mathematical question, solution effort, suggestion and comment. Secondly, the phases of the groups' mathematical problem solving process (Schoenfeld, 1987), analysis, exploration, implementation and verifying were examined. In order to examine participants' metacognition, the computer notes were analysed to find the notes where metacognitive knowledge (person, task and strategy variables) or metacognitive skills (planning, monitoring and evaluating) (Flavell, 1979; Brown, 1987) were evident.

Further, the social processes of joint problem solving were described using the stages of perspective taking (Järvelä & Häkkinen, 2002; Selman, 1980) modified to describe networked mathematical problem solving processes: subjective role taking, reciprocal perspective taking, and mutual perspective taking. The networked discussions were partitioned into episodes where the participants tried different approaches to solve the problem.

The cognitive, metacognitive and social levels of analysis were synthesized in a graph where the phases of the group’s problem solving were drawn as a function of time. The graphical display was compared with the transcripts of the group interviews. In the transcripts of the group interviews the interviewees' utterances I and we were used to diverging individual and shared metacognitive processes.

Results and conclusion

Results of the study suggest that the participants shared their mathematical knowledge and jointly constructed solutions together although the minority of the computer notes included utterances identified as metacognitive. With the participants' interpretation of the joint problem solving, the results indicate that in networked discussions there is some metacognitive activity, which could be identified as socially shared metacognition.

In the future research it would be important to describe more detailed the mechanisms of socially shared metacognition and to develop mixed methods to enable to link socially shared metacognition to learning outcomes.

This paper focuses on the role of teacher ratings and other assessment techniques on metacognitive skills in mathematics in elementary school children. The skills measured by prospective and retrospective questionnaires and on-line techniques of above average, average and below average mathematical problem solvers were contrasted as parallel measures of metacognition. Child questionnaires seem attractive but not reliable as alternative to picture metacognitive or mathematics skills. Children think they act skillfulness, although they don’t. Experienced teachers have a better picture of the metacognitive skills of their pupils. In our dataset metacognitive skillfulness accounted for between 15 and 51% of the mathematics performances, depending on how it is assessed. The choice of diagnostic instruments highly determined the predicted percentage. How you test was what you got. Prediction on-line measured with EPA2000, planning measured with teacher ratings, monitoring on-line measured with think aloud protocols and evaluation skills on-line assessed with EPA2000 account for 32.4%, 46.2%, 20% and 34.1% respectively of the variance in mathematics performance. Especially planning was closely related to mathematical problem solving in third grade children. Educational implications of the study are discussed.

Introduction

First, the present study aims to add some data on the value of teacher ratings on metacognitive skills of elementary school children. The second aim of this study is to investigate whether concurrent and prospective and retrospective child-questionnaires differ in the taxation of metacognitive skills in young children. Third, the objective is to compare the prediction, planning, monitoring and evaluation skills in order to see if the degree that skills are acquired depend on the skill itself.

Method

Twenty children in their third year of elementary school from an urban big town in Flanders, participated in this study. All subjects completed three mathematics tests, namely the KRT-R (CAR Overleie, 2006), TTR (de Vos, 1992) and the CDR (Desoete & Roeyers, 2006b). The CDR measured their mathematics and calibration skills. The regular teacher completed a teacher survey on metacognitive skills in the same period. In a counterbalanced design, participants either the EPA2000 (a computer test on mathematics, prediction and evaluation skills) or they performed the think-aloud task first. Thinking-aloud protocols were transcribed verbatim and analyzed according to a metacognitive coding scheme on the presence of metacognitive skills. Moreover children answered on a prospective (PAC) and retrospective (RAC) questionnaire on prediction, planning, monitoring and evaluation skills. On the PAC they had to indicate before solving any mathematical problem on a 7 point Likert-type of scale to what extent a statement (e.g., ‘I control exercises I make’) is representative of their behaviour during mathematical problem solving. The RAC asked to indicate on a 7 point Likert-type of scale to what extent a statement (e.g., ‘working according to plan’) was representative of their mathematical problem solving behaviour on the past task.

Results

Significant positive correlations were demonstrated between prospective (PAC) and retrospective measures (RAC) of prediction. In addition significant correlations were shown between the concurrent EPA 2000 prediction score and the teacher rating of prediction skills in pupils. Children with high prediction skills according to EPA2000 also performed good at prediction behaviour in the Thinking Aloud, although the correlation was not significant.

Significant positive correlations were also demonstrated between the concurrent evaluation score (EPA2000), the calibration score (CDR), the Thinking Aloud Protocol analyses of evaluation behaviour (TAP), the teacher rating of evaluation skills in pupils and the prospective assessment of evaluation skills by children (PAC) but not with the retrospective assessment of evaluation skills by children (RAC). Children with good prospective evaluation skills (PAC) rated themselves also as good evaluators retrospectively (RAC), although the correlation was not significant.

Given the high intercorrelations between the mathematics subtestscores  the internal structure of the mathematical data was first analyzed with a Principal Components Analysis, to account for all the variance.  

A multiple regression analysis was conducted on the mathematics component as outcome variable with the teacher rating on prediction, planning, monitoring and evaluation simultaneously as predictor variables. R² was .428 and F (4, 16) = 3.988, p < .05 Planning skills predicted 39.6% of the mathematics component variance and could be regarded as the best estimates of variance. In the same vein, separate analyses revealed that teacher ratings of prediction skills contributed for 31.4%, planning for 46.2%, monitoring for 31.4% and evaluation for 31.2%  to the variance in mathematics performances of third grade children.

Moreover, the mathematics outcome was predicted for 25.7% (F (4, 18)=2.556; p ≤ .08) by the prospective child questionnaire. The unexpected outcome was that children rating they often used their prediction skills did worse in mathematics compared with peers who rated themselves bad on prediction skilfulness. The child retrospective questionnaire ratings scores predicted 51.1% (F (4, 18)=5.689; p ≤ .01) of the mathematics component. Again children rating they often used their prediction and planning skills did worse in mathematics compared with peers who rated themselves not often using  prediction or planning skills.

Analyses revealed that prediction skills of EPA2000 contributed for 34.1% (F (1,18)=10.323; p ≤.005) and the evaluation skills for 34.2% (F (1,18)=10.373; p ≤.005) to the variance in mathematics performances of third grade children. Children with high prediction and evaluation skilfulness measured by EPA2000 did better results on the mathematics component.

Analyses revealed that think aloud protocol assessing monitoring skills distributed for 20% (F (1,18)=5.503; p =.031; b=.495) to the variance in mathematics learning. However prediction; planning and evaluation skillfulness assessed by Think Aloud Protocols in third grade children were no significant predictors for mathematics performances of these children.

Finally MANOVA’s revealed that above-average performers exhibited more prediction and evaluation skills than below-average performers, while no differences were found between below-average and average mathematical problem solvers. All three performance groups differed on the teacher rating of metacognitive skills. Moreover, all children overestimated their performances, but above-average performers outperformed below-average performers on calibration. The groups did not differ significantly on thinking aloud protocols, although the scores were in line with our expectations. Furthermore, above-average mathematical problem solvers differed on the prospective and retrospective questionnaires from average and below-average mathematical problem solvers, but not in the expected way. Below-average performers were not only inaccurate in estimating their test performance (see calibration scores), but also often wildly overestimated their metacognitive performance (see prospective and retrospective scores). Above-average performers had a more accurate prospective and retrospective score than below-average performers.

Discussion

These results should be interpreted with care, since there are

some limitations to the present study. It should be

acknowledged that sample size is a serious limitation of

the present study. Additional research whith larger groups of

children see indicated. Such studies are recently being

planned. However, reflecting on the results of the present

study, there is evidence that not ‘what’ test but ‘how’ you test

is what you get. We suggest that researchers who are

interested in skilfulness in young children use multiple-method

designs, including teacher questionnaires. In addition, our

dataset revealed that planning measured by teacher

questionnaires was closely related to mathematics

performance. It might be possible that with more time

allocated to metacognitive planning instruction, some

mathematics problems may become less pervasive.