For large groups of students (300 to 600) it is not easy to implement examinations that conciliate qualities such as objectivity of correction, , practicability, rapidity, predictivity, reliability and, above all, theoretical construct validity in terms of mental processes. For each of the six levels of Bloom et al.’s (1956) taxonomy of the cognitive processes, we have refined the concepts, on the basis of cognitivists’ and epistemologists’ works (Reder, 1987 ; Bredart & Modollo, 1998 ; Park & Reder, 2004 ; Perry, 1970) . For instance, at Knowledge level, we have designed a new instrument called “Large MCQ”, mid-term way between recall and recognition. At comprehension level, we have developed Smedslund’s (1997) definition of comprehension, designing “interchanges in translations”. At the application and analysis levels, we stress the distinction between questions and problems, suggested definitions to classify questions and propose the concept of cognitive vigilance. At the synthesis level, we propose a multicriterial self evaluation procedure in open ended answers and essays. At the evaluation level, we propose the use of Degrees of Confidence (De FInetti, 1965) and the representation of quality of answers as a spectral distribution of quality (Leclercq, 2003). Given those concepts and instruments, we made steps towards assessment results that can be diagnostic and distinctive (Engelhart, 1994) in terms of Bloom’s taxonomy. We use a combination of new forms of MCQs, of Open Ended Questions and of Degrees of Certainty in the same testing session. Students have been trained by simulated exams and the effect of this has been measured through the evolution of 80 students in 3 parallel tests taken in 3 month time. Results are analysed instrument by instrument.

For large groups of students (300 to 600) it is not easy to implement examinations that conciliate qualities such as objectivity of correction, , practicability, rapidity, predictivity, reliability and, above all, theoretical construct validity in terms of mental processes. For each of the six levels of Bloom et al.’s (1956) taxonomy of the cognitive processes, we have refined the concepts, on the basis of cognitivists’ and epistemologists’ works (Reder, 1987 ; Bredart & Modollo, 1998 ; Park & Reder, 2004 ; Perry, 1970) .

At Knowledge level, we have designed a new instrument called “Large MCQ” or LMCQ, mid-term way between recall and recognition. This LMCQ system forces the student to recall first his answer, then to check whether it is in the list and finally copy on the answer sheet this number. All correct answers are not in the list. In this case, students are invited to indicate that NONE of the solutions is correct, usually with a code. The “None of them” option is what we call a “General Implicit Solution” (GIS), General meaning that it has the same meaning for all the questions in a test. Implicit meaning that they are not repeated within each question. They can also be explicit and, than, are called SGE.

At Comprehension level, we have developed Smedslund’s (1997) definition of comprehension, designing “interchanges in translations”. Since  there could be several good “translations” of the same expression. As a consequence of this, we introduced the General Implicit Solution (GIS) “Them all”, that make MCQs, with already the “None of them” option, a better candidate to assess comprehension.

At the application and analysis levels, we stress the distinction between questions and problems, suggested definitions to classify questions and propose the concept of cognitive vigilance. An “analysis question” is one where the highest mental load is in translation

(1) - either into a more general situation or in the discovery of the ways to solve a problem, for instance because it is an “ill formulated problem”.

(2) -or into the detection of a problem (in terms of logical incompatibility, limits, lack of data, etc.)

To handle this kind of question, we developed the use of two additional General implicit Solutions, namely “Missing Data” and Absurdity”.

At the synthesis level, we propose a multicriterial self evaluation procedure in open ended answers and essays. Examples of scales may be :

Scale 1 : the quality of the structure of the student’s oral answer to the question.

Scale 2 : the (spontaneous) use of appropriate the exact wording, such as  technical vocabulary

Scale 3 : The originality of examples (not simply replicating the examples given by the professor).

At the evaluation level, we propose the use of Degrees of Confidence and the representation of quality of answers as a spectral distribution of quality. the best way to ask the certainty is to have student select it on a 6 points scale, namely : 0%, 20%, 40%, 60%, 80%, 100% (Leclercq, 2003). On the basis of the concept of “partial knowledge”, De Finetti (1965, 111) has argued and demonstrated that “It is only subjective probability that can give an objective meaning to every response and scoring method”.

In terms of meaning, Leclercq (2003) suggested to plot each answer on an axis, he calls”quality spectrum” representing the answer’s quality, ranking from the worse (incorrect with the higher DC, i. e. 100%) to the best (correct with the higher DC, i. e. 100%). Hunt (1993) suggested to distinguish between three types of knowledge : “misinformed, uninformed, informed”. We would place them respectively on the left, middle and right side of the continuum.

These concepts have been applied with students in first year in Psychology. We used a combination of new forms of MCQs, of Open Ended Questions and of Degrees of Certainty in the same testing session. Inter test correlations have been computed. The evolution of 80 of them that have passed 3 successive parallel tests in November and December 2005 and in January 2006 (the exam) is displayed in several graphs, depending of the kind of questions (4 questions of each of the 5 types in each of the 3 tests). Here is one of the graphical representation of data :

Given those concepts and instruments, we made steps towards assessment results that can be diagnostic and distinctive. Experimental data show that the students are not prepared in secondary schools to face some of those concepts and instruments, but that a training (successive simulations) helps bettering in the majority of them. There are still a lot of questions to answer at three levels : the concepts, the instruments and the processing of data, including scoring methods. We hope that the present article has given room to further debates and developments.

Bredart, S. & Modolo, K. (1988). Moses strikes again : focalization effect on a semantic illusion. Acta Psychologica, 67, 135-144

De Finetti, B. (1965), Methods for discriminating levels of partial knowledge concerning a test item, British Journal of Mathematical and Statistical Psychology, 18, 87-123.

Hunt, D. (1993), Human Self-Assessment : Theory and Application to Learning and Testing, in: Leclercq D. & Bruno J. (1993), Item Banking : Interactive Testing and Self-Assessment, NATO ASI Series, F 112, Berlin : Springer Verlag, 177-189.

Leclercq, D. (1982). Confidence Marking, its use in testing, in Postlethwaite & Choppin (Eds), Evaluation in education: An international review series. (Vol.6-2, pp. 161-287). Oxford: Pergamon.

Park, H. & Reder, L. (2004). Moses Illusion. In R.F. Pohl (Rd). Cognitive Illusions. Hove : Psychology Press., 275-291.

Smedslund, J. (1997), The forgotten  variable of  understanding . Cahiers de Psychologie Cognitive – Current Psychology of Cognition, 16 (1-2), 217-221.