Computer-based technology opens new possibilities for researchers to deal with new issues and methods to investigate on-line processes when people understand and learn with text and graphics. This research will produce a better understanding of why some students reach good levels of understanding and learning, whereas some others do not. Magliano and colleagues will present a new test of reading comprehension called the Reading Strategy Assessment Tool (R-SAT) that elicits and analyzes automatically verbal protocols that readers generate as they read narrative, historical, and scientific texts. Ainsworth and colleagues explores how techniques developed in computational linguistics and machine learning could be used to help code verbalizations produced when students self-explain diagrams of the cardio-vascular system. Boucheix will present results based on eye tracking technologies that get precise behavioural indicators of students’ underlying processes when they understand animated and multiple representations of complex mechanical systems. Tabbers’s presentation will deal with research studies using the dual-task paradigm to investigate the role of working memory processes when people learn from text and pictures.

We are constructing a new test of reading comprehension called the Reading Strategy and Assessment Tool (R-SAT). R-SAT elicits and analyze verbal protocols that readers generate as they read fictive narrative, historical narrative and scientific texts. R-SAT is administered on the computer and test takers and R-SAT employs latent semantic analysis (LSA) and word matching algorithms to assess the quality of the protocols. R-SAT is consistent with a growing movement in test development that has students produce open-ended responses, rather than answer multiple-choice questions. The hope is that by asking questions on-line, we will have a more explicit account of comprehension processes than what could be detected by a multiple-choice test of comprehension. 

After reading pre-selected target sentences, R-SAT readers are asked to produce two types of open ended responses.   An indirect approach requires readers to report thoughts with think aloud instructions used by Trabasso and Magliano (1996). Using the “indirect” method, Magliano and Millis (2003) found that recall and comprehension scores were predicted by an LSA analysis of the protocols. Good comprehenders generated protocols that had high cosines with prior causally-related text, and to a lesser extent, the current sentence. Poor comprehenders tended only to use content from the target sentence. As predicted by theories of comprehension, the better readers appeared to be integrating the current sentence to the causal structure of the text. 

A second “direct” approach involves having readers produce answer to specific questions (e.g., why questions). Performance on questions chosen for R-SAT were shown to be related to measure of comprehension (e.g., performance on standardize tests and open ended short answer questions for texts read silently). A majority of these questions assess a readers ability to access causal antecedents for the current sentences that are in the prior discourse context. Furthermore, a majority of these causal antecedents would require the reader to reactivate information from the prior discourse representation as only one sentence is available to the reader at any given time.

The direct and indirect protocols are scored via word matching algorithms. These algorithms “count” the number of words in a protocol that are present in semantic benchmarks that reflect different aspects of the response. With respect to the indirect protocols, word counts are calculated for the current sentence, local sentence (one sentence back), all distal sentences (2 or more sentences back), and new words not mentioned in the prior discourse. With respect to the direct protocols, word counts are calculated between the protocols and ideal answers. We calculate average word counts for each of these variables for each participants.

Based on the results of a prior study, we have constructed three forms of R-SAT. Each form contains two texts for the fictive narrative, historical narrative, and science genre. The texts in each form were matched in terms of grade level. Furthermore, there was roughly an equal amount of sentences requiring indirect and direct responses. The primary goal of the presentation was to validate these form by assess the extent to which they are correlated with the comprehension of additional texts, as measure by short-answer questions. The performance of R-SAT was compared to that of a traditional multiple-choice test of comprehension, the Gates McGinitie (GM).

In the current study, 180 undergraduates participated for course credit. There were approximately sixty participants for each of the three forms. The study tool place in two sessions. In the first session, participants were administered the GM test of reading comprehension. They also read two short texts, one historical and one science text. They answered 10 short answer questions after reading each text. In the second session, participants were administered R-SAT on personal computers.

 

 

To understand learning with text and graphics, researchers typically take advantage of process measures such as verbal protocols. However, analysing an hour of protocols can take from ten to fifty hours. Therefore, we are interesting in exploring how techniques developed in computational linguistics and machine learning could be used to help code verbalisations. Our approach (CODELEARNER) suggests that a system should have three key functions: accuracy (system and human coder assign the same code), economy (number of examples a researcher has to code to train the system) and predictability (whether a system can estimate its own performance from a smaller subset of data).

To test the system, we compared its performance to human coding of self-explanations given by learners studying concrete or abstract diagrams of the heart. There were 23,330 words, sectioned into 1784 different segments and the human coder decided that 699 of these segments were self-explanations, 1022 were paraphrases and 63 were monitoring statements.

CODELEARNER’s accuracy was 75% when trained with 1600 example segments. However, the Cohen's Kappa (0.52) would not be deemed satisfactory for inter-rater reliability in standard experimental situations.

CODELEARNER’s was more successful at economy exhibiting a steep learning curves (see Figure 1), providing the system with only 300 coded segments allows it to achieve an accuracy rate of 70%.

CODELEARNER was successful at predicting its level of accuracy with a larger training set. It can accurately predict how well it will do with datasets twice the size as the one it was provided with.

To understand learning with text and graphics, researchers typically take advantage of process measures as well as outcome measures. Verbal protocols are the most common process measure but leave researchers with considerable time and effort costs in recording, transcribing, segmenting, and coding. Overall, analysing an hour of protocols can take from ten to fifty hours. But we really need is to study processes over longer periods of time, and in considerable detail, if we are to understand, for instance, how learning changes with time and in real rather than artificial contexts. Therefore, we are interesting in exploring how techniques developed in computational linguistics and machine learning could be used to help code the verbalisations given in learning situations.

 

Our approach (embedded in a system known as CODELEARNER) suggests that researcher will want three key functions from such a system: accuracy, economy and predictability. By accuracy, we mean that a system and a human coder would assign the same code, economy refers to the number of examples a researcher would need to code in order to train the system and predictability is the extent to which a system is able to estimate its own performance on the full data from a smaller subset.

 

 

Previous research has shown that students develop a deeper understanding of material they study if they generate explanations to themselves whilst learning (Chi, Bassok, Lewis, Reimann, & Glaser, 1989). Ainsworth & Loizou (2003) presented students with information about the circulatory system in either text or diagrams and prompted them to (verbally) self-explain. Diagrams students outperformed text students on almost every measure of learning and generated more self-explanations. Consequently, the current study sought to explore if the nature of the diagrams (abstract versus concrete) influenced the self-explanation effect.

 

 

Overall participants learnt a considerable amount from pre to post-test (F(42,3) = 111.1, p<0.001) and on the post-test only items the self-explainers scored more than the non self-explainers (F(42,2) = 4.59, p<0.02). The more self-explanations learners gave the better they tended to do at post-test (r = .59, p<0.001). There was no effect of the type of diagram on either learning outcomes or self-explaining, but there was an effect on monitoring statements (F(1,22) = 4.91, p<0.05).

 

However, the results we focus on here if how well our system did in coding the self-explanations (technical details can be found in Forsyth, Ainsworth, Clarke and O’Malley (submitted).

 

Overall, transcribing the 24 self-explanation participants resulted in 23,330 words which were sectioned into 1784 different segments. The human coder decided that 699 of these segments were self-explanations, 1022 were paraphrases of the material studied and 63 were monitoring statements (see Table 1 for an example of each).

Table 1. Example text categories.

 

 

 

 

 

 

 

 

Ainsworth, S. E., & Loizou, A. T. (2003). The effects of self-explaining when learning with text or diagrams. Cognitive Science, 27(4), 669-681.

 

Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 5, 145-182.

 

Forsyth, R.. Ainsworth, S. Clarke, D. and O’Malley, C (submitted) Semi-Automatic Categorical Coding of Verbal Data in Social Science: Progress & Prospects. International Journal of Human Computer Studies.

In the current research about the comprehension of animated and multiple representations, on line methods stay very few. But new eye tracking technologies allow to get more precise behavioural indicators: fixations number and duration, transitions between selected area of interest in the picture, precise eye trajectory, and scan paths. These measures can be newly combined with off-line comprehension investigations. This presentation aims to show the relevance and also the limits of such on-line methods to study multimedia comprehension. Two researches about different kind of animated multimedia presentation will be exposed. The first study concerns the comprehension of a complex mechanical system from an animated and controllable display. The second research investigates, also with the eye tracking technique, the topic of the collaborative comprehension in technical learning from multiple representations.

 

 

 

 

 

 

 

Experts spent less time exploring the system before beginning the description phase. In using the gaze tracker software, Areas of Interest (AOI) were defined that matched the relevant elements involved in the sub-systems of the piano device. The functional parts that move most were fixated more frequently than the functional parts that move least. These latter parts are fundamental to an understanding of the system’s core causal chains. The parts of the system involved in dynamic causal chains, that move most and also that move least were fixated more frequently in the description phase than in the exploration phase. In particular, experts fixated these functional areas more frequently than novices, with pianists showing an intermediary pattern. We found high correlation between the description scores and the number fixations in the most important functional parts of the piano system. It is notable that several of these components are those that actually move least. Participants with Low description scores made more perception-based errors in the causality of the movements than those with High scores.