Epistemological beliefs have received a lot of attention in educational research. It is assumed that studens’ epistemological beliefs are related to their problem solving skills (Hofer, 2001). A widely used instrument to measure epistemological beliefs is Schommer’s questionnaire. Despite the broad use of the instrument, conceptual and methodological issues about the instrument have been raised (see for instance, Clarebout, Elen, Luyten, & Bamps, 2001). From various perspectives, therefore, researchers have been looking for alternative approaches to assess epistemological beliefs. Four contributions each present a different instrument for measuring epistemological beliefs. In this symposium the advantages and disadvantages of these alternative approaches are discussed.

We present an overview of nine studies on different applications of CAEB, a semantic differential to measure epistemological beliefs. The aim of the overview is to examine the quality of the instrument in different empirical contexts. CAEB (connotative aspects of epistemological beliefs) is a semantic differential to examine students’ associative-evaluative beliefs about the nature of knowledge in different domains. In CAEB students read an initial sentence like “knowledge in botanic can be described as:” and are asked to judge their beliefs about knowledge (thus in this case about botanic) on 24 pairs of adjectives (like: dynamic – static; structured – unstructured). CAEB allows to establish judgement-profiles on item level as well as to calculate factor scores of the participants. Criteria to judge the quality of CAEB were: a) replication of the factor structure, b) ability to measure differences in students’ beliefs about different academic domains, c) relations to other variables in the learning process. We were able to replicate a stable two factor solution with the dimensions texture (beliefs about structure and accuracy of knowledge) and variability (beliefs about stability and dynamic of knowledge) in all studies. CAEB proved to be able to measure differences of students’ epistemological beliefs for different academic domains in three studies. Further on, we found interactions between students’ epistemological beliefs measured with CAEB and different variables of learning processes within six studies. Up to now we can conclude that our studies with CAEB are promising. These results will be discussed in relation to the possibilities and limits of CAEB to measure epistemological beliefs.

We present an overview of nine studies on different applications of CAEB, a semantic differential to measure epistemological beliefs. The aim of the overview is to examine the quality of the instrument in different empirical contexts. CAEB (connotative aspects of epistemological beliefs) is a semantic differential to examine students’ associative-evaluative beliefs about the nature of knowledge in different domains. In CAEB students read an initial sentence like “knowledge in botanic can be described as:” and are asked to judge their beliefs about knowledge (thus in this case about botanic) on 24 pairs of adjectives (like: dynamic – static; structured – unstructured). CAEB allows to establish judgement-profiles on item level as well as to calculate factor scores of the participants. Criteria to judge the quality of CAEB were: a) replication of the factor structure, b) ability to measure differences in students’ beliefs about different academic domains, c) relations to other variables in the learning process.

a) We were able to replicate a stable two factor solution with the dimensions texture (beliefs about structure and accuracy of knowledge) and variability (beliefs about stability and dynamic of knowledge) in all studies.

b) CAEB proved to be able to measure differences of students’ epistemological beliefs for different academic domains in three studies:

-         In a first study 634 students of biology and ecology were asked to complete the CAEB for the subject areas: plant identification, genetics, and physics.  We found significant differences between students’ epistemological beliefs in all three domains. To give an example we found that knowledge in plant identification was seen as significantly less structured and accurate than knowledge in genetics which was seen as significantly less structured than knowledge in physics (factor: texture).

-         In a second study (n = 653 students of biology, geography and pharmacy) differences between students judgments for the domains plant identification and organic chemistry could be found. Knowledge in plant identification was seen as significantly less structured than knowledge in organic chemistry (factor: texture).

-         In a third study 119 students of chemistry and 138 students of theology were asked to judge the domains chemistry and theology. We found significant differences for the domains as well as significant effects for the field of study. To give an example: All students judged knowledge in chemistry to be more structured than knowledge in theology. Nevertheless we found that theology students judged theology as more structured than students of chemistry did.

c) We found interactions between students’ epistemological beliefs measured with CAEB and different variables of learning processes within six studies:

-         We found significant correlations between students’ (n = 77, trainee teachers) beliefs about the nature of knowledge in mathematics and their mathematic anxiety measured with the MARS (Mathematics Anxiety Rating Scale), their judgements on how much they liked mathematics in their own schooltime, and their judgement on how much they like to teach mathematics. To give an example, we found a correlation that trainee teachers who judged knowledge in mathematics to be more unstructured also disliked mathematics more in their own schooltime than those who believed in structured knowledge in mathematics.  

-         We measured the effects of different learner related variables on help seeking processes within an interactive learning environment for plant identification (n = 74 students of biology and pharmacy; working in dyads). Dyads of students who believed that knowledge in plant identification is uncertain und unstructured accessed the help functions more often than dyads who believed it to be more certain and structured.

-         We found a significant correlation between students’ (n = 24 psychology students) revision processes during processes of hypertext writing and their beliefs in the variability of knowledge: students with more sophisticated beliefs revised their contents more often.

-         In two studies on metacognitive calibration students (first study: n = 77, students of biology; second study: n = 102, students of biology and humanities) were asked to judge six learning tasks of different complexity (according to the revised taxonomy of Bloom) with a questionnaire to examine students assessments on self-regulated learning (COPES-questionnaire, developed from Winne’s COPES-model of self-regulated learning). Results in both studies confirmed significant relationships between students’ epistemological beliefs on the one hand and their metacognitive calibrations (judgments in dependency of the task complexity) on the other hand.

-         We asked biology students (n = 25) and humanities students (n = 26) to learn within a hierarchical hypertext about the topic “genetic fingerprinting”. We found that students’ epistemological beliefs were related to their navigation in the hypertext and to their judgements about the comprehensibility of the contents. To give an example: Students who believed that knowledge is more unstructured judged more complex contents as less comprehensible.

We proposed a distinction between a procedural and a declarative level of personal epistemology, designated as epistemic reasoning respectively epistemic beliefs. We hypothesized that epistemic beliefs are developed later than epistemic reasoning and tested this hypothesis in two studies with the assumption that internal consistency within and relations between measures of epistemic reasoning and measures of epistemic beliefs are higher depending of the developmental level of the underlying constructs. Epistemic reasoning was measured using bipolar scales containing explicit contrasting labels. Respondents judged two domains: knowledge on nature and knowledge on man and society. In each domain one scale was established: uncertainty of knowledge within each domain. Epistemic reasoning was measured using knowledge dilemmas and asking the respondent to indicate level of agreement with statements that represent different ways of epistemic reasoning. Three scales were derived: absolutism, multiplism, and evaluativism. In the first study 209 respondents participated. Internal consistency of uncertainty of knowledge scales appeared to increase with five consecutive grade levels, starting with secondary third grade. Internal consistencies of absolutism were more equal, however an increase was detectable in the consistency of multiplism and evaluativism. Patterns of correlations between and among measures of epistemic reasoning and epistemic beliefs were variable and difficult to interpret. In the second study the number of grade levels was decreased to three (secondary fourth, sixth, and bachelor students) in favor of the number of respondents. In total 330 respondents participated. Analysis is still in progress, but internal consistencies both of uncertainty measures and of epistemic reasoning measures were more or less equal.

Research on personal epistemology is still severely hampered by methodical and methodological problems. Several studies on the development of personal epistemology we carried out using theory mind type tasks (Carpendale & Chandler, 1996) and ‘epistemological’ reasoning tasks (Kuhn, Cheney, & Weinstock, 2000), suggested that the measurement of personal epistemology is disturbed by factors related to the knowledge content that is used to evoke epistemologically relevant responses (Rozendaal, de Brabander, & Dahl, 2003; De Brabander & Rozendaal, 2005). These experiences inspired us, to speculate on a conceptual difference between two levels of personal epistemology: an operational level and a reflective level (De Brabander, & Rozendaal, 2005). Gradually this distinction surfaced as a difference between procedural and declarative knowledge.

We designated the procedural level of personal epistemology “epistemic reasoning”. Epistemic reasoning is the application of epistemological ‘rules’ in a concrete and content-specific context. For the time being we hypothesize that important types of epistemic reasoning might be characterized as absolutistic, multiplistic, or evaluativistic (Kuhn & Weinstock, 2002), however, not as developmental stages of the knower, but as stances taken within a specific problem situation. Such a conceptualization narrows the gap between developmental views of epistemic reasoning and the resources view of Hammer & Elby (2002). In addition to the procedural level of epistemic reasoning we postulated a declarative level of “epistemic beliefs”. This declarative level involves an explicit articulation of the relationship between the objective and the subjective within a body of knowledge in general. We speculate that the development of epistemic beliefs also is characterized by  absolutistic, multiplistic, and evaluativistic positions and that this development is driven by reflection on intensive experiences with knowledge (re)production. Therefore, we believe that the development of epistemic beliefs in contrast to epistemic reasoning starts rather late and only for those who are intensively engaged in one or more knowledge domains. The relationship between the two levels is recursive, but is never a one-to-one relationship because of the differences between how the two types of knowledge operate.

We tested the distinction between epistemic reasoning and epistemic beliefs in two studies. In both we used the Connotative Aspects of Epistemological Beliefs (CAEB; Kienhues, Stahl, & Bromme, 2005) instrument and a questionnaire for measurement of developmental levels of epistemological beliefs (FREE; Krettenauer, 2005). The CAEB contains bipolar contrasts, like “simple vs. complex”, “stable vs unstable”, “certain vs. uncertain”, and the respondent is asked to judge a body or domain of knowledge on a Likert scale. Because the respondent is required to connect an explicit label to the concept of a body of knowledge in general, this instrument appeared adequate to measure epistemic beliefs. We used two broad bodies of knowledge to be judged: “knowledge on nature” and “knowledge on man and society”. We used the FREE to measure epistemic reasoning. This instrument presents the respondent a series of knowledge dilemmas. In each the respondent is offered three statements that imply an absolutistic, a multiplistic, and a evaluativistic way of handling validity claims of the two opposing statements. The respondent records his agreement with each statement.

Because we expected epistemic beliefs to develop later than epistemic reasoning, we investigated in both studies whether with level of knowledge experience a) internal consistency of only the CAEB increased, and not of the FREE; b) the relationship between CAEB and FREE increased. In both studies subjects were drawn from university and pre-university education.

Study I

In the first study both instruments were translated as to represent as faithfully as possible the intention of the original instruments. The sample included 209 respondents with over 24 respondents per grade level.

Analysis of the internal consistency using Cronbach’s α showed that the CAEB in both knowledge areas actually contained one scale, uncertainty of knowledge, with sufficient reliability. From analyses on consecutive grade levels starting at secondary third grade we found that with one exception consistency at lower grades was lower than at higher grades. Thus, with respect to epistemic beliefs, the results appear to support the late onset hypothesis of epistemic beliefs. Judgements of the two knowledge domains, however, were not related.

Reliability analyses of three epistemic reasoning scales, absolutisme, multiplisme, and evaluativism, showed that only absolutism conformed to our consistency hypothesis. The consistency of multiplism and absolutism is highest at the bachelor level, although the end value is not much higher than the value in grade 3.

Patterns of association between measures of uncertainty and absolutisme, multiplism, and evaluativism were difficult to interpret, the hypothesized increment was not apparent.

Study 2

In the second study we tried to improve apparent deficiencies of the Dutch version of the FREE. Furthermore, we improved the level of control on the measurement context. In an attempt to improve stability of results we decreased the number of grade levels (secondary fourth and sixth, and bachelor students), but increased the number of respondents per level (100 or more). The data are not completely analyzed yet, but the first results indicate that a pattern of increase of internal consistency of the measure of uncertainty (CAEB) is not apparent. Internal consistency of the absolutism, multiplism, and evaluativism scales is in all grade levels more or less equal. Also a clear pattern of increasing associations between uncertainty and the three epistemic reasoning scales did not emerge. However splitting the respondents according to their knowledge domain in students of natural sciences and students of social or cultural sciences revealed more clear and counteracting patterns in these relationships at the level of bachelor students that to a certain extent comply with the hypothesized distinction between epistemic beliefs and epistemic reasoning.

From the results of the studies taken together, we concluded that the conceptual distinction between epistemic beliefs and epistemic reasoning is promising, but that the relationship between these contructs and their role in knowledge “work” is far more complex than hitherto conceptualized. These complexities and several implications for the measurement of personal epistemology are discussed.

In order to assess people's underlying approach to the nature of knowledge claims, epistemological instruments have generally been designed to be more or less content-free so that responses will not be influenced by prior opinions or knowledge. Although this approach is justified if one wants to assess how people make sense of how people approach unfamiliar knowledge domains or understand how discrepant knowledge claims arise, it may test a small subset of everyday knowing. It would seem that most knowledge claims that people would truly consider in a normal day concern areas that they would care about or have at least some prior knowledge. This presentation will examine the use of a content-free instrument, explore findings that indicate that prior subject or cultural knowledge might influence responses, and present variations on the instrument that might better demonstrate how content might influence responses to assessment items. The implications are twofold: (1) People's epistemological "competence," as measured by content-free measures may not accurately indicate their performance when having to consider knowledge claims relevant to their own experience, and (2) more attention should be paid to designing assessments that can tap epistemological understandings while not obscuring the influence of people's everyday, domain specific, and cultural knowledge.

Most epistemological assessments consist of content-free items. They are so constructed in order to uncover underlying beliefs about knowledge. Such general beliefs might be obscured by participants’ concrete, prior knowledge. In addition, by using content-free items, prior knowledge differences between participants can be controlled for.

Students’ beliefs about knowledge in learning environments have typically been assessed with Likert-scale belief surveys with items, such as “Things are simpler than most professors would have you believe” or “Truth is unchanging” (Schommer, 1998), that purposely do not specify any specific claims about knowledge or what is true. Even belief surveys that imply a content by suggesting a domain, such as “In this subject, most work has only one right answer” or “If you read something in a textbook for this subject, you can be sure it is true,” (Hofer, 2000), deliberately avoid actual content.

Despite good reasons for controlling prior knowledge in making group comparisons, one might wonder whether such assessments are valid. Learning environments do involve prior knowledge much of the time. Thus, responses to questions about beliefs about non-specified knowledge may not reflect what students believe about the specific knowledge claims presented to them. Nevertheless, it is alternatively possible that students do invoke specific knowledge when responding to such assessments, particularly in the sorts of assessments that prompt respondents to focus on a specific subject matter.

Another type of assessment employs discrepant knowledge claims apart from any explicit school context. This assessment type has been used in investigating the relationship between epistemological understandings and informal reasoning (Kuhn & Weinstock, 2002). A content-free assessment would seem reasonable because there are meaningful situations when people confront competing knowledge claims about which they have no or little prior knowledge, but about which they must apply their understandings of how knowledge is constructed and the requirements of knowledge justification. For instance, jurors are presented a lot of new, often discrepant information. To make sense and reason about this information and then claim knowledge about what happened may depend on the type of epistemological understandings jurors bring with them to the task (Weinstock & Cronin, 2003). Perhaps, then, there is an epistemology of informal reasoning that concerns how people understanding uncertain, discrepant, unfamiliar knowledge claims.

However, what is important in more typical everyday reasoning concerns how people reason about issues that they care about or have some prior knowledge about. This raises the question of whether a content-free assessment can adequately capture people’s epistemic reasoning about issues filled with relevant knowledge. The proposed presentation will focus on the implications of and possible problems with the use of a particular, paper-and-pencil, discrepant claims content-free task (Kuhn, Cheney, & Weinstock, 2000).

The Task

The paper-and-pencil instrument (Kuhn et al., 2000) has been used as an indicator of the epistemological level of people engaging in argument tasks (Mason & Scirica, 2006; Weinstock, Neuman, & Glassner, 2006). It involves the presentation of two content-free claims. The items clearly indicate that there is a discrepancy, but do not say what the discrepancy is. An example of such an item is:

Robin thinks one book's explanation of why WWI began is right.

Chris thinks another book's explanation of why WWI began is right.

Is only one of them right, or is it possible that both are right to a certain degree? (Circle one answer):

A.     Only one of them is right.

B.     It is possible for both to be right to a certain degree

If it is possible that both of them are right, is it possible that one of them could be more right than the other? (Circle one answer):

A.     One of them could be more right than the other.

B.     One could not be more right than the other.

There are similar items in five domains: taste, aesthetics, values, physical truth, and, like the example, social truth. The items are intended to give little information that would allow a judgment of one being more right than the other.

The use of the task has led to some results inconsistent with theoretical expectations. It was predicted that people would become relativist in the values domain later than they became relativist in the truth domains. But several studies have found that more people than expected remain absolutist in the values domain (Kuhn et al., 2000; Weinstock, 2005; Weinstock et al., 2006). This may be because the items for this domain differ from the others (see Krettenauer, 200x) in that they give information that would allow one to have a position in the debate (e.g., Robin thinks lying is wrong; Chris thinks lying is permissible in certain situations). If these items were changed to be content-free (i.e., Robin has one belief about whether lying is wrong, and Chris has a different belief), perhaps less people would appear to be absolutist. Conversely, if items in other domains were changed to have more relevant content, perhaps people would give more absolutist responses. The instrument might produce different results not only if the discrepant knowledge claims were made explicit, but if people had strong implicit knowledge that skewed their responses.

The proposed presentation will discuss studies using the instrument that have found differences between groups. These differences might be attributed to differences in epistemological development due to experience, or they might reflect implicit domain (Tabak & Weinstock, 2005) or cultural (Weinstock, 2005) knowledge that adolescents bring to bear when responding to the assessment. The presentation will also present results from data being collected in two cultures using the assessment along with two modified versions. In one version, the two debaters are not peers, but one will be an adolescent and the other will represent authority (parent, teacher, textbook). It is assumed that some cultures will consider this, and not just the specific discrepant claims in dispute. The other modification will include items with reference to phenomena that are more relevant to one culture or another, assuming that this will create different responses than would have predicted by the use of the content-free instrument.

Epistemological beliefs, or ideas about knowledge and knowing, have gained significant attention in educational research, as it is assumed that these beliefs influence learning (Hofer, 2001). Schraw, Dunkle, and Bendixen (1995), for instance, found that students’ epistemological beliefs influenced students’ performance on solving ill-defined problems. Similar results were also found by Spiro, Feltovich, and Coulson (1996).

Until now, mainly survey instruments have been used to assess these beliefs. A well-known, and widely used, example of such a survey instrument is the Schommer (1990) questionnaire. While a number of methodological critiques have been raised (Clarebout, Elen, Luyten & Bamps, 2001) on these survey instruments, a good alternative has not yet been identified.

            This contribution explores the possibility of using drawings as an instrument to gain insight into students’ epistemological beliefs. This idea is inspired by research done already in the mid 50s, were scholars used drawings to unravel students’ beliefs about scientists (Mead & Métraux, 1957). In 1992, Vosniadou and Brewer used drawings – combined with interviews – to gain insight into students’ views of the world. Drawings have also been used in research on teacher development (Swenne, Jörg & Korthagen, 2001; Hancock & Gallard, 2004). In these studies, drawings were used as a technique to reveal how student teachers perceive themselves as teachers. Similarly, Picker and Berry (2000), and Rock and Shaw (2000), used drawings in order to investigate and compare students’ images of mathematicians (they asked 12- to 13-year olds to draw a mathematician).

Methodology

Participants

In this study, 216 first year educational science students participated. Participation in the research resulted in two credit points for a course on ‘learning and instruction’. On average students were 19-years old.

Procedure

Students were asked to draw 'knowledge'. They were instructed to make their drawings in such a way as to clarify what knowledge is for someone who does not understand words. The drawings were anonymous. Students were told explicitly to work individually and reminded that drawing skills would not be evaluated. They received five minutes to draw knowledge, after which all drawings were collected by the researcher.

To process the collected data, the drawings were randomly divided among three researchers. Each researcher individually ordered the drawings according to similarities. Afterwards, all groups were compared and a categorisation system was devised. Once devised, the categorisation system was employed by two other researchers, who had not participated in the elaboration of the system, to identify prototypical examples of each category. This step served as a check for the categorisation system itself.

Results

Four groups can be distinguished. The first group sees knowledge as personal property. Students in this group typically draw a person with a ‘textballoon’ or enlarged head that contains the ‘knowledge’. This knowledge is expressed by symbols, drawings or words. The second group sees knowledge as something external. Drawings of students in this group depict books, computers, or authorities. In a third group, the drawings respresent knowledge as both personal property and something embedded in external artifacts or other persons. An example is the representation of a learner and a library. A textballoon is connected to the pupil in which symbols and words are presented. This represents the internal part, while the library represents the external part. A final group of drawings represents knowledge as a process.  Drawings in this category often contain arrows indicating that something enters the head of a person and something leaves the head. These drawings frequently stress the brains overtly by, for instance, enlarging them.

Results show that 82 students (37.96%) see knowledge as personal property. Twenty-four students (11.11%) represent knowledge as something external that can be found in books, computers or/and authorities. Seventy-two students (33.33%) represent knowledge as a combination of something external and internal. Twenty-seven students (12.50%) represent knowledge as a process. From the 216 drawings, only 11 drawings (5.09%) could not be categorized in the system.

The results of the two independent researchers who employed the categorisation system to order the drawings indicate that it is useful. Using the categorisation system, these researchers could easily identify prototypical images from the total pool for each group.

Discussion and Conclusion

            This study shows that it is indeed possible to use drawings to reveal students’ epistemological beliefs. However, it may be that this categorisation system is too specifically related to the sample. Although the ‘back-categorisation’ of the drawings into the system gave insight into the usability. Further research is needed to see whether this categorisation system can also be applied to other samples, for instance, in other cultures, or for other age groups. In any case, drawings have the advantage over questionnaires in that students do not simply agree or disagree with a statement, but may express their beliefs in less confined, and perhaps more creative, ways. In other words, the technique does not prompt students to choose a specific direction, as is often criticized when statements are given.  Moreover, it also facilitates participants who do not possess the necessary linguistic skills to articulately speak about their beliefs (as suggested by Elen & Lowyck (1999), which is a problem in the case of interview techniques. Unfortunately, the method also has some disadvantages as it is a fairly time consuming process to categorize the drawings.  Additionally, the drawings may force students with an abstract and complex concept of knowledge to fit it into something drawable.

            Although, it was explicitly mentioned that students’ drawing skills would not be judged, one can still wonder, whether students drew knowledge according to what they were able to draw rather than how they actually view knowledge. This probably becomes even more of an issue once the drawings are no longer anonymous. It is advised that before epistemological beliefs measured by drawings can be studied as a predictor of learning, further research should address these methodological issues.