Proposal view
Proposal Type: Individual Paper 
Domain: Assessment and Evaluation 
SIG: Assessment and Evaluation 
Type Submitted Paper 
Equipment PC and projector
Paper Details
Title From theoretical notions of competence to adequate psychometric models
Abstract In educational assessment, the term competence is associated with the idea of assessing complex combinations of abilities and skills that are needed in specific real life situations. This complex, context specific definition of competencies differs from most traditional constructs in cognitive ability testing, which are often conceptualized to consistently affect performance behavior across a large range of different situations. Nevertheless, in empirical studies often the same traditional psychometric models that are usually used in ability testing are applied in the measurement of competencies. The aim of this paper is to analyze and illustrate the different demands on psychometric models that arise from the conceptualization of competencies as complex, context specific constructs compared to traditional ability testing. Adequate measurement models for competencies can do more than summarize to what extent an individual will master certain situations. If interactions between different abilities of persons and different situational demands are represented adequately, psychometric models for the measurement of competencies can support the understanding of why persons are successful in coping with certain situations. A brief overview of existing psychometric models that may meet these demands is given, and one possible model meeting the demands outlined in the paper is illustrated using data from a large scale assessment of foreign language competencies. Implications of the use of adequate psychometric models for the development of empirical methods to assess competencies are discussed.
Summary In educational research, several authors have claimed that the measurement of general, domain-independent skills or traits is of little use when performance in real life situations is to be understood or promoted. As Bandura (1990) put it, "there is a marked difference between possessing knowledge and skill and being able to use them well under diverse circumstances". Being able to cope with a certain range of situations is often summarized as a competence. Weinert (1999) suggested that the functional, context-related approach associated with the concept of competence be used in assessments of educational outcomes. Recently, the concept of competence has been used to describe outcomes in various educational studies.
When used in educational assessment, the term competence is accordingly associated with the idea of assessing complex combinations of abilities and skills that are needed in specific real life situations instead of testing general, rather abstract cognitive abilities. This complex, context specific definition of competencies differs from most traditional constructs in cognitive ability testing, which are often conceptualized to consistently affect performance across a large range of different situations. Nevertheless, in empirical studies often the same traditional psychometric models that are usually used in ability testing are applied in the measurement of competencies. The aim of this paper is to analyze and illustrate the different demands on psychometric models that arise from the conceptualization of competencies as complex, context specific constructs compared to traditional ability testing; a brief overview of existing psychometric models that may meet these demands is given. One possible model meeting the demands outlined in the paper is illustrated using data from a large scale assessment of foreign language competencies.
If competencies are understood as complex, context specific constructs, psychometric models used in their measurement should adequately reflect these characteristics. Adequate measurement models for competencies can do more than summarize to what extent an individual will master certain situations. If interactions between different abilities of persons and different situational demands are represented adequately, psychometric models for the measurement of competencies can support the understanding of why persons are successful in coping with certain situations. Results based on adequate models of competence can provide insights in the processes underlying success or failure, and knowledge about the relevant combinations of skills and abilities that form specific competencies. Following this rationale, the German Research Foundation has recently commissioned a priority programme that focuses on the theoretical models of competence in the assessment of educational outcomes.
The first demand on measurement models that can be derived from the concept of competencies is that they should be able to reflect interactions between situational demands and different skills and abilities on the person’s side. Within the most common item response models (e.g. the Rasch model or the 2PL), each task used in the assessment process usually is assumed to reflect one single underlying ability dimension. Consequently, these relatively simple models yield only one rather descriptive continuous score if they are used as measurement models. They cannot take into account that successfully coping with certain situations or tasks may require combinations of different abilities. To incorporate more complex interactions between different abilities and situational demands, measurement models need to be multidimensional, i.e. they need to include more than one dimension simultaneously to represent different abilities forming a competence. Specifically, measurement models for competencies should allow for within item-multidimensionality, since the solution of one task may require combinations of different abilities.
Competencies are – by definition – related to a specified range of situations. Consequently, a second obvious demand to be made on measurement models for competencies is to take into account characteristics of these relevant situations as well as those of persons. In an assessment context, this means tasks used to assess competencies need to be characterized in terms that reflect situational demands in real life. These task characteristics should then be explicitly incorporated into psychometric models to measure competencies. One type of task characteristics are specifications of which abilities are needed for the solution of certain tasks. This kind of information is needed to define loading patterns in multidimensional measurement models. Another complementary way to describe tasks is the level of demands that they pose for a person’s abilities. Translated into a psychometric model, these task characteristics would be used to model difficulties of different tasks. Again, most traditional psychometric models used in ability testing do not allow to systematically model task difficulties; instead each task is assigned one individual difficulty. A well known psychometric model designed to incorporate task characteristics is the linear logistic test model (LLTM). However, the original LLTM is a one-dimensional model, thus not meeting the first requirement to allow for multidimensionality when measuring competencies.
The paper will give an overview of different models based on item response theory (IRT) that fulfill the requirements for an adequate measurement of competencies. One class of models combines multiple latent dimensions with a systematic incorporation of task characteristics; e.g. in a multidimensional LLTM. Within another class of models, called multiple classification models, different abilities are represented by latent classes.
Finally, the paper illustrates the idea of an adequate measurement model for competencies using data from a large scale assessment of foreign language competence. Data from various tasks assessing listening, reading and grammar skills are simultaneously analyzed in a three-dimensional LLTM that incorporates various task characteristics to model task difficulties. The results are illustrated by individual ability profiles for different groups of students. The levels of these profiles can be described with reference to the difficulties of specific combinations of task characteristics.
Implications of the use of adequate psychometric models for the development of empirical methods to assess competencies are discussed. It is argued that to develop and validate empirical models of competence, tasks should primarily be constructed with respect to a valid representation of the range of relevant situations. Task characteristics that describe the necessary combinations of abilities to master these situations as well as the extent of situational demands should be derived from domain-specific theories.
Keywords Assessment of competence
Item response theory (IRT)
Psychometrics
Appendices
Authors
Name Surname Institution Country e-mail EARLI Number Presenting
Johannes Hartig DIPF Germany hartig@dipf.de   *  
Eckhard Klieme DIPF Germany klieme@dipf.de    
Visit NQcontent
© European Association for Research on Learning and Instruction, 2012 All rights reserved.