Proposal view
| Proposal Type: | Symposium |
|---|---|
| Domain: | Learning and Cognitive Science |
| SIG: | Metacognition |
| Type | Invited SIG Symposium |
| Title | The assessment of metacognition. |
| Abstract | In educational research a serious problem pertains to the disparity amongst definitions of concepts, and, hence, to the lack of congruence in operationalizations of concepts and their assessment methods. This is particularly the case for the field of metacognition, i.e., the knowledge about our cognitive system and the skills for regulating that system. Even as definitions appear to converge to some extent, until recently a proliferation of assessment methods occurred without thorough validation studies. Questionnaires administered either prior to or retrospective to task performance, observations and thinking aloud during task performance, eye-movement or computer-logfile registrations during task performance, and post hoc interviews or video-stimulated recall afterwards, belong to the broad range of assessment tools for metacognition (Veenman, Van Hout-Wolters, & Afflerbach, 2006). All methods suffer from a trade-off between advantages and disadvantages. Some may be easy to administer, but raise doubts about what is actually being measured; others are hard to obtain, but do reflect on-line metacognitive activities during task performance. Quite often, only the internal consistency of metacognition measures (reliability) is reported, rather than the convergent validity with other assessment methods that intend to assess the same construct (Veenman, 2005). Contributions to this symposium will show that multi-method designs are essential for establishing construct validity. Moreover, a second criterion for evaluating assessment methods is their external validity: Do assessment methods of metacognition converge in their prediction of learning performance to the same extent as expected? These validity issues will reoccur during the symposium. |
| Equipment |
Overhead projector PC and projector |
| Keywords | Assessment methods Metacognition Self-regulation |
| Chair list | |||||
|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | |
| Marcel V.J. | Veenman | University of Amsterdam/ Leiden University | Netherlands | Veenman@fsw.leidenuniv.nl | |
| Organiser list | |||||
|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | |
| Marcel V.J. | Veenman | University of Amsterdam/ Leiden University | Netherlands | Veenman@fsw.leidenuniv.nl | |
| Discussant list | |||||
|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | |
| Christa | van Kraayenoord | The University of Queensland, School of Education | Australia | c.vankraayenoord@uq.edu.au | |
| Paper Details |
|---|
| Title | An overview of assessment methods for metacognitive skills: Their internal consistency, concurrent validity, and external validity. |
|---|---|
| Abstract | Marcel V.J. Veenman Graduate School of Teaching and Learning, University of Amsterdam, Dept. of Developmental & Educational Psychology, Leiden University, Wassenaarseweg 52, 2333 AK Leiden, The Netherlands. Veenman@fsw.leidenuniv.nl and Bernadette H.A.M. Van Hout-Wolters Graduate School of Teaching and Learning, University of Amsterdam, The Netherlands. Metacognitive skillfulness refers to the repertoire of skills and strategies for the regulation of and control over one’s learning behavior. Task analysis, activating prior knowledge, planning, monitoring one’s activities, evaluating outcomes, and reflecting on one’s learning processes are activities representative of metacognitive skillfulness. Moreover, metacognition appears to be one of the most profound predictors of learning outcomes (Wang, Haertel, & Walberg, 1991). In educational research a variety of methods for assessing metacognitive skills, self-regulation, and metacognitive strategy usage is employed. Questionnaires, interviews, systematical observation, protocol analyses, log-file registrations, and stimulated recall are the most frequently used methods. The validity of assessment methods, however, is rarely investigated in a systematical way (Veenman, Van Hout-Wolters, & Afflerbach, 2006). Drawing from the literature, an up-to-date review of the pros and cons of assessment methods will be presented. A distinction will be made between off-line and on-line assessments: Off-line methods are administered either prior or retrospective to task performance, whereas on-line measures are registered during actual task performance. Apart from the off-line/on-line dimension, assessment methods will be evaluated against three validity criteria: Internal consistency (reliability), concurrent validity by mutual relations with other assessment methods, and external validity for the prediction of learning outcomes. Although we are still in the process of gathering literature data for this review study, some tentative conclusions can be drawn. On-line measures should probably be preferred over off-line-measures, as the latter seem to lack concurrent and external validity. Moreover, there is a need for research using multi-method designs. |
| Summary | Marcel V.J. Veenman Graduate School of Teaching and Learning, University of Amsterdam, Dept. of Developmental & Educational Psychology, Leiden University, Wassenaarseweg 52, 2333 AK Leiden, The Netherlands. Veenman@fsw.leidenuniv.nl and Bernadette H.A.M. Van Hout-Wolters Graduate School of Teaching and Learning, University of Amsterdam, The Netherlands. Metacognitive skillfulness refers to the repertoire of skills and strategies for the regulation of and control over one’s learning behavior. Task analysis, activating prior knowledge, planning, monitoring one’s activities, evaluating outcomes, and reflecting on one’s learning processes are activities representative of metacognitive skillfulness. Moreover, metacognition appears to be one of the most profound predictors of learning outcomes (Wang, Haertel, & Walberg, 1991). In educational research a variety of methods for assessing metacognitive skills, self-regulation, and metacognitive strategy usage is employed. Questionnaires, interviews, systematical observation, protocol analyses, log-file registrations, and stimulated recall are the most frequently used methods. The validity of assessment methods, however, is rarely investigated in a systematical way (Veenman, Van Hout-Wolters, & Afflerbach, 2006). Drawing from the literature, an up-to-date review of the pros and cons of assessment methods will be presented. A distinction will be made between off-line and on-line assessments: Off-line methods are administered either prior or retrospective to task performance, whereas on-line measures are registered during actual task performance. Apart from the off-line/on-line dimension, assessment methods will be evaluated against three validity criteria: Internal consistency (reliability), concurrent validity by mutual relations with other assessment methods, and external validity for the prediction of learning outcomes. Although we are still in the process of gathering literature data for this review study, some tentative conclusions can be drawn. On-line measures should probably be preferred over off-line-measures, as the latter seem to lack concurrent and external validity. Moreover, there is a need for research using multi-method designs. Marcel V.J. Veenman Graduate School of Teaching and Learning, University of Amsterdam, Dept. of Developmental & Educational Psychology, Leiden University, Wassenaarseweg 52, 2333 AK Leiden, The Netherlands. Veenman@fsw.leidenuniv.nl and Bernadette H.A.M. Van Hout-Wolters Graduate School of Teaching and Learning, University of Amsterdam, The Netherlands. Metacognitive skillfulness refers to the repertoire of skills and strategies for the regulation of and control over one’s learning behavior. Task analysis, activating prior knowledge, planning, monitoring one’s activities, evaluating outcomes, and reflecting on one’s learning processes are activities representative of metacognitive skillfulness. Moreover, metacognition appears to be one of the most profound predictors of learning outcomes (Wang, Haertel, & Walberg, 1991). In educational research a variety of methods for assessing metacognitive skills, self-regulation, and metacognitive strategy usage is employed. Questionnaires, interviews, systematical observation, protocol analyses, log-file registrations, and stimulated recall are the most frequently used methods. The validity of assessment methods, however, is rarely investigated in a systematical way (Veenman, Van Hout-Wolters, & Afflerbach, 2006). Drawing from the literature, an up-to-date review of the pros and cons of assessment methods will be presented. A distinction will be made between off-line and on-line assessments: Off-line methods are administered either prior or retrospective to task performance, whereas on-line measures are registered during actual task performance. Apart from the off-line/on-line dimension, assessment methods will be evaluated against three validity criteria: Internal consistency (reliability), concurrent validity by mutual relations with other assessment methods, and external validity for the prediction of learning outcomes. Although we are still in the process of gathering literature data for this review study, some tentative conclusions can be drawn. On-line measures should probably be preferred over off-line-measures, as the latter seem to lack concurrent and external validity. Moreover, there is a need for research using multi-method designs. Marcel V.J. Veenman Graduate School of Teaching and Learning, University of Amsterdam, Dept. of Developmental & Educational Psychology, Leiden University, Wassenaarseweg 52, 2333 AK Leiden, The Netherlands. Veenman@fsw.leidenuniv.nl and Bernadette H.A.M. Van Hout-Wolters Graduate School of Teaching and Learning, University of Amsterdam, The Netherlands. Metacognitive skillfulness refers to the repertoire of skills and strategies for the regulation of and control over one’s learning behavior. Task analysis, activating prior knowledge, planning, monitoring one’s activities, evaluating outcomes, and reflecting on one’s learning processes are activities representative of metacognitive skillfulness. Moreover, metacognition appears to be one of the most profound predictors of learning outcomes (Wang, Haertel, & Walberg, 1991). In educational research a variety of methods for assessing metacognitive skills, self-regulation, and metacognitive strategy usage is employed. Questionnaires, interviews, systematical observation, protocol analyses, log-file registrations, and stimulated recall are the most frequently used methods. The validity of assessment methods, however, is rarely investigated in a systematical way (Veenman, Van Hout-Wolters, & Afflerbach, 2006). Drawing from the literature, an up-to-date review of the pros and cons of assessment methods will be presented. A distinction will be made between off-line and on-line assessments: Off-line methods are administered either prior or retrospective to task performance, whereas on-line measures are registered during actual task performance. Apart from the off-line/on-line dimension, assessment methods will be evaluated against three validity criteria: Internal consistency (reliability), concurrent validity by mutual relations with other assessment methods, and external validity for the prediction of learning outcomes. Although we are still in the process of gathering literature data for this review study, some tentative conclusions can be drawn. On-line measures should probably be preferred over off-line-measures, as the latter seem to lack concurrent and external validity. Moreover, there is a need for research using multi-method designs. |
| Keywords | Assessment methods Metacognition Self-regulation |
| Appendices | |
| Authors | ||||||
|---|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | Presenting | |
| Marcel V.J. | Veenman | University of Amsterdam/ Leiden University | Netherlands | Veenman@fsw.leidenuniv.nl | * | |
| Bernadette H.A.M. | Van Hout-Wolters | University of Amsterdam | Netherlands | B.H.A.M.vanHout-Wolters@uva.nl | ||
| Title | Assessing the use of on-line trace methodologies for analyzing the deployment |
|---|---|
| Abstract | Assessing the use of on-line trace methodologies for analyzing metacognitve processes during learning with hypermedia Roger Azevedo, Amy Witherspoon, Jeremiah R. Sullins, and Shanna E. Baker University of Memphis, Department of Psychology, Institute for Intelligent Systems, 3693 Norriswood Avenue, Memphis, TN, 38152, USA razevedo@memphis.edu Learning with hypermedia environments involves the deployment of key cognitive and metacognitive self-regulatory processes. A key to assessing these processes during learning involves the use of several on-line trace methodologies such as concurrent think-aloud protocols, log files and video analyses. We have accumulated an extensive knowledge base from laboratory and classroom studies examining the role of adolescents’ and college students’ self-regulated learning with hypermedia-based environments (see Azevedo, 2005). Our current understanding of the dynamic and recursive nature of use of these processes during learning is somewhat limited by the capabilities of current technological tools which are augmented by the use of multiple converging on-line trace methodologies. In this presentation, we will assess the conceptual, methodological, and empirical issues related to using on-line trace methods to assess learners’ understanding about complex science topics (e.g., circulatory system) with hypermedia. Conceptually, we will (1) present results illustrating the advantages and disadvantages to using on-line trace methodologies in collecting cognitive and other non-cognitive processes during learning; (2) describe current limitations in the use of log file analyses to examine self-regulatory processes; and, (3) argue for the integration of other self-report measures to augment our theoretical understanding of the complex nature of self-regulatory processes. Methodologically, we will discuss ways in which current learning technologies and cognitive methodologies can be combined with computational and other advances in computer science and AI to collect, detect, trace, and model learners’ self-regulatory processes during learning. Empirically, we will discuss our existing approaches to converging process and product and assess its advantages and disadvantages. |
| Summary | Assessing the use of on-line trace methodologies for analyzing the deployment of metacognitve processes during learning with hypermedia Roger Azevedo, Amy Witherspoon, Jeremiah R. Sullins, and Shanna E. Baker University of Memphis, Department of Psychology, Institute for Intelligent Systems, 3693 Norriswood Avenue, Memphis, TN, 38152, USA razevedo@memphis.edu Learning with hypermedia environments involves the deployment of key cognitive and metacognitive self-regulatory processes. A key to assessing and measuring the deployment of these processes during learning involves the use of several on-line trace methodologies such as concurrent think-aloud protocols, log files and video analyses. We have accumulated an extensive knowledge base from laboratory and classroom studies examining the role of adolescents’ and college students’ self-regulated learning with hypermedia-based environments (see Azevedo, 2005). Our current understanding of the dynamic and recursive nature of use of these processes during learning is somewhat limited by the capabilities of current technological tools which are augmented by the use of multiple converging on-line trace methodologies. In this presentation, we will assess the conceptual, methodological, and empirical issues related to using on-line trace methods to assess learners’ understanding about complex science topics (e.g., circulatory system and ecology) with hypermedia. Conceptually, we will (1) describe and present results illustrating the advantages and disadvantages to using on-line trace methodologies (e.g., think-aloud protocols) in collecting cognitive and other non-cognitive processes during learning; (2) describe current limitations in the use of log file analyses to examine the deployment of self-regulatory processes; and, (3) argue for the integration of other self-report measures to augment our theoretical understanding of the complex nature of self-regulatory processes. Methodologically, we will discuss ways in which current learning technologies and cognitive methodologies can be combined with computational and other advances in computer science and AI to collect, detect, trace, and model learners’ self-regulatory processes during learning. Empirically, we will discuss our existing approaches to converging process and product and assess its advantages and disadvantages. Assessing the use of on-line trace methodologies for analyzing the deployment of metacognitve processes during learning with hypermedia Roger Azevedo, Amy Witherspoon, Jeremiah R. Sullins, and Shanna E. Baker University of Memphis, Department of Psychology, Institute for Intelligent Systems, 3693 Norriswood Avenue, Memphis, TN, 38152, USA razevedo@memphis.edu Learning with hypermedia environments involves the deployment of key cognitive and metacognitive self-regulatory processes. A key to assessing and measuring the deployment of these processes during learning involves the use of several on-line trace methodologies such as concurrent think-aloud protocols, log files and video analyses. We have accumulated an extensive knowledge base from laboratory and classroom studies examining the role of adolescents’ and college students’ self-regulated learning with hypermedia-based environments (see Azevedo, 2005). Our current understanding of the dynamic and recursive nature of use of these processes during learning is somewhat limited by the capabilities of current technological tools which are augmented by the use of multiple converging on-line trace methodologies. In this presentation, we will assess the conceptual, methodological, and empirical issues related to using on-line trace methods to assess learners’ understanding about complex science topics (e.g., circulatory system and ecology) with hypermedia. Conceptually, we will (1) describe and present results illustrating the advantages and disadvantages to using on-line trace methodologies (e.g., think-aloud protocols) in collecting cognitive and other non-cognitive processes during learning; (2) describe current limitations in the use of log file analyses to examine the deployment of self-regulatory processes; and, (3) argue for the integration of other self-report measures to augment our theoretical understanding of the complex nature of self-regulatory processes. Methodologically, we will discuss ways in which current learning technologies and cognitive methodologies can be combined with computational and other advances in computer science and AI to collect, detect, trace, and model learners’ self-regulatory processes during learning. Empirically, we will discuss our existing approaches to converging process and product and assess its advantages and disadvantages. Assessing the use of on-line trace methodologies for analyzing the deployment of metacognitve processes during learning with hypermedia Roger Azevedo, Amy Witherspoon, Jeremiah R. Sullins, and Shanna E. Baker University of Memphis, Department of Psychology, Institute for Intelligent Systems, 3693 Norriswood Avenue, Memphis, TN, 38152, USA razevedo@memphis.edu Learning with hypermedia environments involves the deployment of key cognitive and metacognitive self-regulatory processes. A key to assessing and measuring the deployment of these processes during learning involves the use of several on-line trace methodologies such as concurrent think-aloud protocols, log files and video analyses. We have accumulated an extensive knowledge base from laboratory and classroom studies examining the role of adolescents’ and college students’ self-regulated learning with hypermedia-based environments (see Azevedo, 2005). Our current understanding of the dynamic and recursive nature of use of these processes during learning is somewhat limited by the capabilities of current technological tools which are augmented by the use of multiple converging on-line trace methodologies. In this presentation, we will assess the conceptual, methodological, and empirical issues related to using on-line trace methods to assess learners’ understanding about complex science topics (e.g., circulatory system and ecology) with hypermedia. Conceptually, we will (1) describe and present results illustrating the advantages and disadvantages to using on-line trace methodologies (e.g., think-aloud protocols) in collecting cognitive and other non-cognitive processes during learning; (2) describe current limitations in the use of log file analyses to examine the deployment of self-regulatory processes; and, (3) argue for the integration of other self-report measures to augment our theoretical understanding of the complex nature of self-regulatory processes. Methodologically, we will discuss ways in which current learning technologies and cognitive methodologies can be combined with computational and other advances in computer science and AI to collect, detect, trace, and model learners’ self-regulatory processes during learning. Empirically, we will discuss our existing approaches to converging process and product and assess its advantages and disadvantages. |
| Keywords | Assessment methods Metacognition Self-regulation |
| Appendices | |
| Authors | ||||||
|---|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | Presenting | |
| Roger | Azevedo | University of Memphis, Department of Psychology | United States | razevedo@memphis.edu | * | |
| Amy | Witherspoon | University of Memphis, Department of Psychology | United States | razevedo@memphis.edu | ||
| Jeremiah R. | Sullins | University of Memphis, Department of Psychology | United States | razevedo@memphis.edu | ||
| Shanna E. | Baker | University of Memphis, Department of Psychology | United States | razevedo@memphis.edu | ||
| Title | Assessing metacognitive activities: Is using a questionnaire a valid way? |
|---|---|
| Abstract | Gonny L.M. Schellings, Bernadette H.A.M. Van Hout-Wolters, Graduate School of Teaching and Learning, University of Amsterdam, Wibautstraat 2-4, 1091 GM Amsterdam, The Netherlands. g.l.m.schellings@wanadoo.nl Marcel V.J. Veenman Graduate School of Teaching and Learning, University of Amsterdam. Dept. of Developmental and Educational Psychology, Leiden University. & Joost Meijer SCO-Kohnstamm Institution, University of Amsterdam. Educational research has yielded a diversity of assessment instruments for assessing learning strategies, each with their strengths and methodological flaws (Van Hout-Wolters, 2006). Administering questionnaires, for example, is least effortful. One may question, however, whether participants are able to ‘say what they will do, or recollect accurately what they have done’. Veenman (2005) concluded that research concerning the convergent-validity of questionnaires is lacking. Hence, there is a need for validity research with multi-method designs, which must be executed meticulously, meaning that different methods should assess the same metacognitive activities (van Hout-Wolters, 2006). In the present study, we examine whether a task-specific retrospective questionnaire assessing metacognitive activities yields similar results as assessing metacognitive activities concurrently with the think-aloud method. The questionnaire was straightforwardly constructed parallel to a hierarchical taxonomy developed in a preceding think-aloud study (Meijer, Veenman, & Van Hout-Wolters, 2006). Data from the questionnaire and the think-aloud method are compared in a within-subjects multi-method design. Sixteen students in secondary education (ninth grade) study a history text while thinking aloud. Immediately after studying the text, the students are presented with a metacognitive questionnaire, consisting of 58 items. Finally, a learning posttest is administered. Thinking-aloud protocols will be coded according to the Meijer et al. metacognitive taxonomy. Analyses will be performed on the level of specific metacognitive activities, as well as on super-ordinate categories of metacognitive skills (orientation, planning, execution, monitoring, evaluation, and reflection). At present, the data are being gathered and results will be available in due time. |
| Summary | Gonny L.M. Schellings Graduate School of Teaching and Learning, University of Amsterdam, Wibautstraat 2-4, 1091 GM Amsterdam, The Netherlands. g.l.m.schellings@wanadoo.nl Bernadette H.A.M. van Hout-Wolters Graduate School of Teaching and Learning, University of Amsterdam. Marcel V.J. Veenman Graduate School of Teaching and Learning, University of Amsterdam. Dept. of Developmental and Educational Psychology, Leiden University. & Joost Meijer SCO-Kohnstamm Institution, University of Amsterdam. Educational research has yielded a diversity of assessment instruments for assessing learning strategies, each with their strengths and methodological flaws (Van Hout-Wolters, 2006). Administering questionnaires, for example, is least effortful. One may question, however, whether participants are able to ‘say what they will do, or recollect accurately what they have done’. Veenman (2005) concluded that research concerning the convergent-validity of questionnaires is lacking. Hence, there is a need for validity research with multi-method designs, which must be executed meticulously, meaning that different methods should assess the same metacognitive activities (van Hout-Wolters, 2006). In the present study, we examine whether a task-specific retrospective questionnaire assessing metacognitive activities yields similar results as assessing metacognitive activities concurrently with the think-aloud method. The questionnaire was straightforwardly constructed parallel to a hierarchical taxonomy developed in a preceding think-aloud study (Meijer, Veenman, & Van Hout-Wolters, 2006). Data from the questionnaire and the think-aloud method are compared in a within-subjects multi-method design. Sixteen students in secondary education (ninth grade) study a history text while thinking aloud. Immediately after studying the text, the students are presented with a metacognitive questionnaire, consisting of 58 items. Finally, a learning posttest is administered. Thinking-aloud protocols will be coded according to the Meijer et al. metacognitive taxonomy. Analyses will be performed on the level of specific metacognitive activities, as well as on super-ordinate categories of metacognitive skills (orientation, planning, execution, monitoring, evaluation, and reflection). At present, the data are being gathered and results will be available in due time. Gonny L.M. Schellings Graduate School of Teaching and Learning, University of Amsterdam, Wibautstraat 2-4, 1091 GM Amsterdam, The Netherlands. g.l.m.schellings@wanadoo.nl Bernadette H.A.M. van Hout-Wolters Graduate School of Teaching and Learning, University of Amsterdam. Marcel V.J. Veenman Graduate School of Teaching and Learning, University of Amsterdam. Dept. of Developmental and Educational Psychology, Leiden University. & Joost Meijer SCO-Kohnstamm Institution, University of Amsterdam. Educational research has yielded a diversity of assessment instruments for assessing learning strategies, each with their strengths and methodological flaws (Van Hout-Wolters, 2006). Administering questionnaires, for example, is least effortful. One may question, however, whether participants are able to ‘say what they will do, or recollect accurately what they have done’. Veenman (2005) concluded that research concerning the convergent-validity of questionnaires is lacking. Hence, there is a need for validity research with multi-method designs, which must be executed meticulously, meaning that different methods should assess the same metacognitive activities (van Hout-Wolters, 2006). In the present study, we examine whether a task-specific retrospective questionnaire assessing metacognitive activities yields similar results as assessing metacognitive activities concurrently with the think-aloud method. The questionnaire was straightforwardly constructed parallel to a hierarchical taxonomy developed in a preceding think-aloud study (Meijer, Veenman, & Van Hout-Wolters, 2006). Data from the questionnaire and the think-aloud method are compared in a within-subjects multi-method design. Sixteen students in secondary education (ninth grade) study a history text while thinking aloud. Immediately after studying the text, the students are presented with a metacognitive questionnaire, consisting of 58 items. Finally, a learning posttest is administered. Thinking-aloud protocols will be coded according to the Meijer et al. metacognitive taxonomy. Analyses will be performed on the level of specific metacognitive activities, as well as on super-ordinate categories of metacognitive skills (orientation, planning, execution, monitoring, evaluation, and reflection). At present, the data are being gathered and results will be available in due time. Gonny L.M. Schellings Graduate School of Teaching and Learning, University of Amsterdam, Wibautstraat 2-4, 1091 GM Amsterdam, The Netherlands. g.l.m.schellings@wanadoo.nl Bernadette H.A.M. van Hout-Wolters Graduate School of Teaching and Learning, University of Amsterdam. Marcel V.J. Veenman Graduate School of Teaching and Learning, University of Amsterdam. Dept. of Developmental and Educational Psychology, Leiden University. & Joost Meijer SCO-Kohnstamm Institution, University of Amsterdam. Educational research has yielded a diversity of assessment instruments for assessing learning strategies, each with their strengths and methodological flaws (Van Hout-Wolters, 2006). Administering questionnaires, for example, is least effortful. One may question, however, whether participants are able to ‘say what they will do, or recollect accurately what they have done’. Veenman (2005) concluded that research concerning the convergent-validity of questionnaires is lacking. Hence, there is a need for validity research with multi-method designs, which must be executed meticulously, meaning that different methods should assess the same metacognitive activities (van Hout-Wolters, 2006). In the present study, we examine whether a task-specific retrospective questionnaire assessing metacognitive activities yields similar results as assessing metacognitive activities concurrently with the think-aloud method. The questionnaire was straightforwardly constructed parallel to a hierarchical taxonomy developed in a preceding think-aloud study (Meijer, Veenman, & Van Hout-Wolters, 2006). Data from the questionnaire and the think-aloud method are compared in a within-subjects multi-method design. Sixteen students in secondary education (ninth grade) study a history text while thinking aloud. Immediately after studying the text, the students are presented with a metacognitive questionnaire, consisting of 58 items. Finally, a learning posttest is administered. Thinking-aloud protocols will be coded according to the Meijer et al. metacognitive taxonomy. Analyses will be performed on the level of specific metacognitive activities, as well as on super-ordinate categories of metacognitive skills (orientation, planning, execution, monitoring, evaluation, and reflection). At present, the data are being gathered and results will be available in due time. |
| Keywords | Assessment methods Metacognition Self-regulation |
| Appendices | |
| Authors | ||||||
|---|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | Presenting | |
| Gonny L.M. | Schellings | Graduate School of Teaching and Learning, UvA | Netherlands | g.l.m.schellings@wanadoo.nl | * | |
| Bernadette H.A.M. | Van Hout-Wolters | University of Amsterdam | Netherlands | B.H.A.M.vanHout-Wolters@uva.nl | ||
| Marcel V.J. | Veenman | University of Amsterdam/ Leiden University | Netherlands | Veenman@fsw.leidenuniv.nl | ||
| Joost | Meijer | SCO Kohnstamm Institute, Univ. of Amsterdam | Netherlands | J.Meijer@uva.nl | ||
| Title | Metacognition and Strategy Use in the College Classroom |
|---|---|
| Abstract | Metacognition and Strategy Use in the College Classroom: A Validation Study Gregory Schraw University of Nevada, Las Vegas 4505 Maryland Parkway, Box 453003, Las Vegas, Nevada 89154, USA. gschraw@unlv.nevada.edu John Nietfeld; North Carolina State University Li Cao; West Georgia University Lori Olafson; University of Nevada, Las Vegas We investigated whether a self-report instrument of metacognitive knowledge was related to measures of mental ability, motivation, and strategy use during a college course on learning, and performance on exams. 60 undergraduates completed the 52-item Metacognitive Awareness Inventory (MAI) (Schraw & Dennison, 1994), the Ravens Progressive Matrices, the Hope Scale, a strategy use inventory, and completed a final exam. The MAI yielded two reliable factors (i.e., knowledge of cognition, regulation of cognition) that explained 75% of sample variance. The knowledge of regulation factor was correlated positively with strategy use and the Hope Scale, but was uncorrelated with the Ravens test and final exam. These findings suggested the MAI has good construct validity, but poor predictive validity when used to predict a single final exam among college students. Future studies should examine the relationship between the MAI and more complex measures of learning. |
| Summary | Metacognition and Strategy Use in the College Classroom: A Validation Study Gregory Schraw University of Nevada, Las Vegas 4505 Maryland Parkway, Box 453003, Las Vegas, Nevada 89154, USA. gschraw@unlv.nevada.edu John Nietfeld; North Carolina State University Li Cao; West Georgia University Lori Olafson; University of Nevada, Las Vegas We investigated whether a self-report instrument of metacognitive knowledge was related to measures of mental ability, motivation, and strategy use during a college course on learning, and performance on exams. 60 undergraduates completed the 52-item Metacognitive Awareness Inventory (MAI) (Schraw & Dennison, 1994), the Ravens Progressive Matrices, the Hope Scale, a strategy use inventory, and completed a final exam. The MAI yielded two reliable factors (i.e., knowledge of cognition, regulation of cognition) that explained 75% of sample variance. The knowledge of regulation factor was correlated positively with strategy use and the Hope Scale, but was uncorrelated with the Ravens test and final exam. These findings suggested the MAI has good construct validity, but poor predictive validity when used to predict a single final exam among college students. Future studies should examine the relationship between the MAI and more complex measures of learning. Metacognition and Strategy Use in the College Classroom: A Validation Study Gregory Schraw University of Nevada, Las Vegas 4505 Maryland Parkway, Box 453003, Las Vegas, Nevada 89154, USA. gschraw@unlv.nevada.edu John Nietfeld; North Carolina State University Li Cao; West Georgia University Lori Olafson; University of Nevada, Las Vegas We investigated whether a self-report instrument of metacognitive knowledge was related to measures of mental ability, motivation, and strategy use during a college course on learning, and performance on exams. 60 undergraduates completed the 52-item Metacognitive Awareness Inventory (MAI) (Schraw & Dennison, 1994), the Ravens Progressive Matrices, the Hope Scale, a strategy use inventory, and completed a final exam. The MAI yielded two reliable factors (i.e., knowledge of cognition, regulation of cognition) that explained 75% of sample variance. The knowledge of regulation factor was correlated positively with strategy use and the Hope Scale, but was uncorrelated with the Ravens test and final exam. These findings suggested the MAI has good construct validity, but poor predictive validity when used to predict a single final exam among college students. Future studies should examine the relationship between the MAI and more complex measures of learning. Metacognition and Strategy Use in the College Classroom: A Validation Study Gregory Schraw University of Nevada, Las Vegas 4505 Maryland Parkway, Box 453003, Las Vegas, Nevada 89154, USA. gschraw@unlv.nevada.edu John Nietfeld; North Carolina State University Li Cao; West Georgia University Lori Olafson; University of Nevada, Las Vegas Metacognition and Strategy Use in the College Classroom: A Validation Study Gregory Schraw University of Nevada, Las Vegas 4505 Maryland Parkway, Box 453003, Las Vegas, Nevada 89154, USA. gschraw@unlv.nevada.edu John Nietfeld; North Carolina State University Li Cao; West Georgia University Lori Olafson; University of Nevada, Las Vegas We investigated whether a self-report instrument of metacognitive knowledge was related to measures of mental ability, motivation, and strategy use during a college course on learning, and performance on exams. 60 undergraduates completed the 52-item Metacognitive Awareness Inventory (MAI) (Schraw & Dennison, 1994), the Ravens Progressive Matrices, the Hope Scale, a strategy use inventory, and completed a final exam. The MAI yielded two reliable factors (i.e., knowledge of cognition, regulation of cognition) that explained 75% of sample variance. The knowledge of regulation factor was correlated positively with strategy use and the Hope Scale, but was uncorrelated with the Ravens test and final exam. These findings suggested the MAI has good construct validity, but poor predictive validity when used to predict a single final exam among college students. Future studies should examine the relationship between the MAI and more complex measures of learning. |
| Keywords | Assessment methods Metacognition Self-regulation |
| Appendices | |
| Authors | ||||||
|---|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | Presenting | |
| Gregory | Schraw | University of Nevada, Las Vegas | United States | gschraw@unlv.nevada.edu | * | |
| John | Nietfeld | North Carolina State University | United States | gschraw@unlv.nevada.edu | ||
| Li | Cao | West Georgia University | United States | gschraw@unlv.nevada.edu | ||
| Lori | Olafson | University of Nevada, Las Vegas | United States | gschraw@unlv.nevada.edu | ||
