A secondary qualitative analysis of the international PISA 2000 survey will be presented. Some elements of PISA 2003 have also been used. The entire research has been carried out by the universities of Luxembourg and Moncton (Canada). Several analyses have been done to explain which factors could explain the differences in results between the participating countries. The presentation will demonstrate that, despite efforts made by the OECD, the results obtained in answering items of the survey are not independent of cultural characteristics, which can be seen as potential biases of the measures.

Methodology

All of the data of the international PISA 2000 and 2003 surveys have been used, but the focus of interest was the results of Luxembourg and Canadian students (because of similar social backgrounds in terms of economy and multilinguism, and the huge differences between them in the results of the OECD surveys). Cognitive results, as well as the answers of the student questionnaire (socio-economic, social and personnel data provided by the students about themselves and their families) have been the basis of the analysis. Supplementary data have been produced through a questionnaire survey with a sample of 441 youngsters (226 Luxemburg and 215 Canadian students) aged 15. These questionnaires were built in a similar way to the theoretical structure of the PISA survey, with half of the items being similar and the other half being dissimilar to the format used in this international comparative study. Students were asked to rate each item to the degree they felt at ease, used to, and able to answer the questions. The evaluations provided by the students were used to gather information about their degree of familiarity to item formats.

Results

A first analysis has shown that familiarity with the ways of questioning can partly explain the performance on different item types. Comparison between the answers of Luxembourgish and Canadian students showed that those who succeeded in PISA were those who were more used to the sort of questioning and writing used in the survey. This result has been confirmed by using perceived familiarity of students from different school types using de facto different pedagogical methods. In Luxembourg, students have more knowledge based on the formal structure of language than on verbal information processing as it is assessed by PISA. The reverse is true for Canadian students, and this difference could explain part of the performance differences in the international survey.

A second research has demonstrated a difference between the a priori theoretical framework structure that has been used to generate the items of PISA 2000 and the structure obtained from analysis of the empirical data. The results in the different subscales can largely be explained by one common cognitive factor, being close to the fluid intelligence factor (Gf) in the scientific literature (Cattell, 1963; Carroll, 1993). This result is not in contradiction with the first result of our research, because it has been shown that the Gf factor is not independent from environmental influences. Research by Ackerman, Bowen, Beier, & Kanfer (2001) has shown that Gf has only a very small influence on school knowledge, except for physics and technologies. On the other hand, it is mediated by crystallized intelligence (Gc) in all the other scholarly disciplines. If school knowledge does not depend mainly on Gf but on Gc, fluid intelligence is nonetheless important in knowledge performance. This theory would explain why national performances of students are linked to teaching methods and the ways of questioning that are predominant in a country.

A third and last research has shown that social, familial, economic and personal characteristics may be related to the cognitive performance of students. We could show that a number of these characteristics are very typical for Luxembourg compared to other countries in general and to Canada in particular. Two factors are of particular relevance for explaining differences in the cognitive results in Luxembourg. The first being the language used at home. Speaking Luxemburgish or German at home has a more favourable impact on results. The second factor being the higher grades obtained by students in German language. German is the language of alphabetisation of pupils in primary schools, it is easily understandable that pupils speaking this language or a language close to it, namely Luxemburgish, are favoured at school. This could explain why these pupils then differentiate themselves from others in all teaching situations consisting of verbal information processing. The other factors provide less statistical explanation of the student results.