Education is undergoing fundamental changes in many European countries. As an important part of education, assessment is changing as well. Birenbaum et al. (2006) state that current assessment practices in European countries fail to address learners’ needs because they focus on assessment of learning instead of assessment for learning, they are limited in scope and they drive teaching for assessment instead of teaching for learning. It is clear that assessment methods need to be changed, which implies that our ways to determine the quality of these assessments also needs to be changed. Because one single assessment method is not enough to assess competencies and fulfil the purpose of assessment for learning, we argue for the use of Competency Assessment Programmes (CAPs), which are combinations of both classical tests and recently developed assessment methods and involve both formative and summative assessments (Baartman, Bastiaens, Kirschner, & Van der Vleuten, 2006).

As CAPs comprise both classical and recently developed assessment methods, CAP quality should be determined by an integrated set of both classical and new quality criteria. We argue that no different quality criteria should be used for formative and summative assessment, and that CAP quality, including both of these, should be evaluated as an integrated whole. Making a distinction between quality criteria for formative and summative assessment brings along the risk of using learning-related criteria like feedback quality and the effect on student learning for formative assessment, and technical test-related criteria for summative assessment.

This paper focuses on the evaluation of such CAPs. Many schools are currently developing competency-based education and concomitant assessment programmes. To assist schools in evaluating their newly developed CAPs, we developed a self-evaluation procedure based on 12 quality criteria for CAPs developed in earlier studies (Baartman et al., 2006). A self-evaluation method was chosen because it is becoming an increasingly important approach to both school improvement and accountability. In the Netherlands, self-evaluation has become a topic of debate since vocational schools have to demonstrate assessment quality to an Examination Quality Centre (EQC) to retain their accreditation. The EQC audit procedure is currently being redesigned into one in which self-evaluation serves as a starting point for external evaluation. Advantages of self-evaluation include lower fear of external evaluation and stimulation of self-reflection. A disadvantage is the lack of sufficient and appropriate data and evidence to support a school’s claims.

This research focuses on the process of carrying out a self-evaluation. Research questions were:

(1) How do schools proceed when they evaluate their own CAP using the self-evaluation procedure?

(2) How do schools substantiate their ratings by means of examples or evidence?

(3) Does the self-evaluation procedure stimulate reflection on CAP quality?

Method

Self-evaluation procedure

The self-evaluation procedure requires a group of personnel from the same school to evaluate their CAP together as probably few people have a complete overview of their school’s CAP. The self-evaluation consists of two phases. In the first phase, all participants evaluated their CAP individually by means of an electronic tool, available through the Internet. They evaluated their CAP on the 12 quality criteria developed in earlier studies: acceptability, authenticity, cognitive complexity, comparability, costs & efficiency, educational consequences, fairness, fitness for purpose, fitness for self-assessment, meaningfulness, reproducibility of decisions and transparency. They rated their CAP via an analog slide-bar that could be moved from “not at all” to “completely”, below which was an invisible 0-100 scale, and were asked to substantiate these rating by some evidence. The second phase consisted of a group reflection interview in which all individual ratings and substantiations were assembled and discussed. Participants were explicitly instructed to react to each others’ ratings and substantiations.

Context and participants

The study was carried out in Laboratory Technology education in 8 vocational schools in the Netherlands. At each school, the department manager, a teacher and a member of the examination board participated. An earlier carried out pilot study had revealed that these three functionaries are well-acquainted with the school’s CAP.

Data analyses

The percentage of low (0-35), medium (36-65) and high (66-100) ratings and the percentage of ratings substantiated with an example was calculated. Qualitatively, recurrent (group) processes and themes were extracted from the typed out interviews. Miles and Huberman’s (1984) phases for qualitative analysis (data reduction, data display and conclusion / verification) were used to categorise the substantiations given, which were rank ordered and labelled independently by two researchers.

Results

More ratings were substantiated after the reflection interview than in the individual self-evaluations (Z = -6.182, p = .000). More high and medium than low ratings were given (Z = -.5.501, p = .000). The qualitative results indicate the importance of the reflection interview. It tended to serve as a way of collaboratively defining the CAP. The three functionaries perceived their CAP from a different perspective and contributed to an overall picture of the CAP’s quality. A recurrent theme was the distinction between formative and summative assessment. Schools appear not to make a clear distinction between the two, whereas the EQC solely focuses on summative assessment. Schools gave appropriate substantiations to their ratings in 90% of the cases, but 30% of the substantiations contained a circular argument (“it is authentic because it is authentic”). Reflection on CAP quality was shown in the new insights gained and spontaneously mentioned improvements of the CAP (“we could specify per assessment project who the assessors are”).

Conclusions

It seems to be possible to evaluate CAPs by means of a self-evaluation, but schools need to be supported in this process. The use of multiple functionaries and a reflection interview seem necessary to get an overall picture of the CAP. Special attention should be paid to substantiating the claims made, as this is what is asked for in external evaluation.

Baartman, L.K.J., Bastiaens, T. J, Kirschner, P.A., & Van der Vleuten, C.P.M. (2006). The wheel of competency assessment. Presenting quality criteria for Competency Assessment Programmes. Studies in Educational Evaluation, 32, 153-177.

Birenbaum et al. (2006). Position paper. A learning integrated assessment system. Educational Research Review, 1, 61-67.