The ability to evaluate arguments is important academically and for real-life decision-making. A student that can not distinguish a claim that is supported by a related reason from either an unsupported claim or an unwarranted argument will be at a disadvantage when reasoning through the many choices at hand. Although argument evaluation is an ability that the U.S. expects its educational system to impart to students (NSS, 1996), many American students leave high school unable to understand, evaluate and write arguments (NAEP, 1996, 1998). Given the importance of argument-based skills, it is essential that all students become proficient text-based reasoners and this will require explicit instruction and practice (Perkins, 1991, Kuhn, 1991). The current series of experiments examines U.S. undergraduates’ ability to distinguish well-structured arguments from poorly-structured arguments and to test the effectiveness of a tutorial to improve their application of this skill.

At minimum, an argument is a claim (controversial statement) supported by at least one reason. Claims and reasons are connected by warrants -- general principle that permit the conclusion given the reason (Toulmin, 1958). Accessing the presence of an acceptable reason that supports the claim with an acceptable warrant is critical when reading arguments. For example, in Argument 1 the absence of a reason makes it an unsupported claim not an argument. Argument 2 is also a flawed argument because it is unwarranted. The implied general principle that “it is unfair to ban something that is banned by other laws” is not coherent. The same reason can be used to support another claim as in Argument 3 but it does not warrant the claim in Argument 2. Thus, one must analyze the relationship between the claim and reason.

#1. Banning cell phone use while driving is unfair.

#2. Banning cell phone use while driving is unfair because laws against inattentive driving already exist.

3. Banning cell phone use while driving is unnecessary because laws against inattentive driving already exist.

Experiment 1

Experiment 1 provided a measure of undergraduates’ ability to evaluate argument quality and tested the effectiveness of a tutorial to improve this skill. Sixty undergraduates were randomly assigned to Training condition (Tutorial vs No-tutorial control). The paper-and-pencil tutorial described argument components, provided examples that contrasted “flawed” and “acceptable” arguments, and provided practice items with feedback available for self-scoring.

A flawed argument judgment test was constructed to detect whether participants were sensitive to argument quality. Twenty-four stem items were manipulated in terms of their Quality: Acceptable (4a), Unwarranted (4b), Unsupported (4c).

4a. Banks shouldn't charge ATM fees because the fees make many customers unhappy.

#4b. Banks shouldn't charge ATM fees because banks are financial institutions.

#4c. Banks shouldn't charge ATM fees.

Participants were asked to mark whether the argument was structurally “flawed” or “acceptable”. They were instructed not to consider how convincing the argument was or whether they agreed with it. Participants received an equal number of items of each Quality and received only one version of each item.

To ensure item classification, 10 graduate students were given the test. They correctly rejected the Unsupported claims (M=97%) and the Unwarranted arguments (M=94%). Thus, the flawed arguments were judged as flawed by skilled argument evaluators. The results for the undergraduates found a significant Quality X Training condition interaction F(2, 114)=4.313, p < .05. There was no difference for Acceptable (Control M=69%, Tutorial M=72%) or Unwarranted arguments (Control M=70%, Tutorial M=66%) but there was a significant difference for Unsupported claims (Control M=61%, Tutorial M=81%). Thus, undergraduates have difficulty detecting flawed arguments, especially when it requires detecting a problem connecting the reason to the claim. This is evidenced by both the difference in accuracy between grad students and control undergrads and by the lack of a training effect on these unwarranted items.

Experiments 2a and 2b

In Experiment 2a, the tutorial was modified to help students notice each component of the claim: Theme (ATM fees) and predicate (shouldn't charge). Recently we have found that undergraduates poorly recall the claim predicate (M=76%) compared to the theme (M=95%). This finding is critical because the predicate is necessary to determine whether the reason actually supports the claim. Therefore, the revised tutorial included instruction and practice to help students become aware of the importance of the predicate.

Participants were randomly assigned to condition: No-tutorial Control, Tutorial, vs Tutorial with immediate feedback. Then participants were given the flawed judgment test. Performance on the flawed judgment test showed there was a significant Quality X Training interaction F(6, 240)=2.667, p < .05. Simple effects of Training for each Quality type found that there was no difference on the flawed judgment test for the three conditions on the Acceptable arguments, F < 1.There was, however, a difference for Unsupported claims F(2, 80)=9.375, p < .05 and Unwarranted arguments F(2, 80)=4.005, p < .05. Post-hoc tests revealed that both tutorials improved participant’s accuracy in judging unsupported claims as flawed but that only the Feedback tutorial significantly improved accuracy of Unwarranted arguments. These results replicate the previous finding that undergraduates do not consistently notice when there is a disconnect between the claim and reason. It also shows that practice with immediate feedback was required for them to begin to develop this skill. It was very encouraging that a short tutorial helped improve students' analysis of argument quality.

In Experiment 2b we gave undergraduates a JAVA version of the tutorial so the training could be presented over the web. The pattern of results replicated Experiment 2a in that the tutorial condition increased accuracy for the Unwarranted arguments and Unsupported claims compared to a no-tutorial control condition. This web-based tutorial is important because it will allow students anywhere to use it to improve their argument evaluation skills.

Discussion

This series of experiments show that college students have difficulty evaluating the quality of simple arguments. They also show the effectiveness of a tutorial training students to attend to claim predicates and providing immediate feedback. Future research will address the impact of learning to evaluate arguments on students’ comprehension and production of arguments.