aims

One goal of undergraduate assessment in medical education is to predict medical students’ (future) performance. In the area of skills testing, the objective structured clinical examination (OSCE) has been of great value as a tool to test a number of skills in a limited time, with bias reduction and improved reliability. It is therefore used in several final and licensing examinations to test junior doctor’s skills competences.

But can OSCE’s measure competences acquired through medical clerkships in the field of basic clinical skills?

method

Undergraduate students (N=32) were given a questionnaire with 184 basic clinical skills. We asked them to score the number of times they performed each skill during their clerkships (12 month period in year 6). We assessed the students before starting their clerkships in year 6 and again at the start of year 7 (undergraduate training takes 7 years in Belgium) using a 14 station OSCE assessing basic clinical skills. Both a checklist and global ratings were used to score performance. Clerkship experience and both OSCE scores (checklist and global rating) were used in a linear regression, with OSCE-scores from year 6 as a moderator.

findings

No significant relations were found between frequency of performing skills during clerkships and OSCE scores. Students scored significantly higher OSCE scores in year 5 then in year 7; this means that after having practiced the skills during clerkships, they score lower on the same test.

Take-home messages

OSCE’s seem not to reflect clinical experience. Other more integrated assessment methods may prove to be more valid to test final undergraduate skills levels.

Aim

The assessment of medical competence of doctors has been the subject of many studies during the last decades, resulting in innovative assessment methods. Traditionally, clinical evaluation consisted primarily of faculty observations, oral examinations and multiple-choice tests. As new evaluation tools have been developed, researchers have excitedly studied the properties of these tools, including aspects of reliability and validity.

In the Objective Structured Clinical Examination (OSCE) students rotate through a number of stations, with standardized patients and/or trained observers. In each station, a student performs a procedure and/or answers a questionnaire. It is an approach to the assessment of clinical competence which pays attention to the objectivity of the examination.

As evidence on the reliability and validity of the OSCE grew the assessment instrument was introduced into many medical curricula throughout the world. It is now also part of the United States Medical Licensing Examination, because of the justification that the proficiencies that are to be assessed by this examination are presumed to be relevant to the safe practice of medicine.

Increasing experience with OSCEs however has demonstrated some limitations of their application. Most assessment methods measure clinical factual knowledge rather than the organization of knowledge that allows clinicians to recognize and handle situations effectively.

We wanted to know if the (our) OSCE captured levels of expertise acquired through clerkships. If students practice their basic medical skills more during clerkships, we expect them to increase their level of competence of basic medical skills. However, do students who practice skills more during clerkships also actually score higher on OSCE scores?

Method

Medical education in Belgium has a duration of 7 years. During the first 5 years of the curriculum, focus is on the gathering of factual knowledge, on acquiring basic medical and communication skills in small groups in a skills lab, and on learning the basics of scientific research. Year 6 consists of a 12 month fulltime clerkship period covering all clinical disciplines. After these clerkships, in their final year, students choose in which medical area they want to specialize  and take classes and clerkships according to their future career choice. At the end of the 7th year, after having successfully passed the final exams and having presented their master’s thesis, students receive their undergraduate (Master’s degree in medicine, equivalent to MD) degree.

Subjects

In 2003, before starting their clerkship period, 32 students did an OSCE assessment on basic medical and communication skills as part of their examination program (OSCE5). They volunteered to participate in the same assessment after having finished their clerkship period, 14 months later (OSCE7). The subjects were a representative sample from the entire year group on gender, class results and age.

Instruments

As assessment tool, a 14 stations OSCE was used, in which each station took 10 minutes and in 8 stations, simulated patients were used. All 14 stations tested  procedures of basic clinical skills, as taught during the first 5 years of the medical curriculum.

Next to taking the OSCEs, students also filled in a questionnaire. The questionnaire contained all of the 340 basic clinical skills students had to be able to perform at the end of their 7 year training. From this questionnaire, we used 182 basic skills that could be linked to the stations used in the OSCE, and the corresponding data were included in the analysis.

Data analysis

A linear regression analysis was performed to investigate to what extent the OSCE7 score could be predicted by the associated Experience score. To avoid confounding effects from students’ variations in skills ability, the OSCE5 score was introduced in the regression.

findings

When we compared OSCE results from year 5 and 7 for our sample students, students total test score was statistically significantly lower on OSCE7 compared to OSCE5 on an overall of all stations. Mean OSCE5 score was 7.6 out of 10 (Stand.dev.0.65), while mean OSCE7 score was 7.3 (Std.dev. 0.55) with a p<0.01.

The questionnaire filled out by the 32 students showed that the percentage of skills that are practiced more than 10 times, varied across stations.

When analyzing OSCE7 scores we see that the R square is .56, which implies that 56% of the variance in OSCE7 scores is explained by the variables Experience and OSCE5. The OSCE5 scores significantly explains part of the variation in OSCE7, with a standardized regression coefficient of .64. Summarizing, the OSCE7 score was partly predicted by the OSCE5 score, but was not affected by the amount of times students practiced basic medical skills during their clerkship period.

Station level R squares were rather low: only 2 to maximum 18% of the OSCE7 station score was explained by the amount of times skills were practiced and OSCE5. None of the regression was statistically significant, indicating that also at station level the number of times students practiced skills relating to that station during clerkships did not affect OSCE7 scores of that station.

Theoretical and educational significance

OSCEs can be of great value for testing students skills competences prior to the fulltime clerkship experience. Clinical competence learnt through clerkships however is an extremely complex construct and one that requires multiple, mixed and higher order methods of assessment to support valid interpretations in high-level skill areas.

For assessing graduating junior doctor’s competences we suggest evaluating the student with a real patient in a natural hospital setting, because we then focus on assessing high-level skill areas to see whether a student has reached the “fit-for-practice” level of competence, rather than on the proficiency of low-level medical skills. It presents students with a complete and realistic clinical case.

There is as yet little research into possible approaches to this in undergraduate medical schools; there are few validated strategies to assess actual clinical practice and this should be the focus of further research.