Proposal view
| Proposal Type: | Individual Paper |
|---|---|
| Domain: | Assessment and Evaluation |
| SIG: | Assessment and Evaluation |
| Type | Submitted Paper |
| Equipment |
Overhead projector PC and projector |
| Paper Details |
|---|
| Title | Assessing Computer Literacy Using Simulated and Real World Tasks |
|---|---|
| Abstract | The importance of ICT proficiency is steadily increasing in modern society and there is ever increasing interest in its assessment. This paper discusses the development and properties of a computer-based assessment of ICT literacy that seamlessly combined questions to assess knowledge and understanding, performance of specific functions within software simulations and creation of products using live applications within a rotated set of thematic modules. Typically, students collected and appraised information and then synthesised and reframed it. The assessment was administered to a nationally representative sample of nearly 7,400 Australian students in Year 6 and Year 10 from 520 schools. The paper discusses issues concerned with the design and delivery of computer-based assessments that incorporate standard test questions, simulated skill assessments and authentic tasks using actual applications in controlled conditions. The paper discusses issues associated with the development of a reliable scale across a set of rotated modules that incorporate simulated and authentic tasks and the interpretation of that scale in terms of what students can do. In addition it reports on an analysis of variations associated with differences among students in terms of year level, background characteristics and patterns of computer use. |
| Summary | 1: Objectives The purpose of this paper is to discuss the development and properties of a computer-based assessment of ICT literacy. ICT literacy is defined as “the ability of individuals to use ICT appropriately to access, manage, integrate and evaluate information, develop new understandings, and communicate with others in order to participate effectively in society” (International ICT Literacy Panel, 2002; Kelly & Haber, 2006). The assessment was administered to a large nationally representative sample of Australian students. The paper discusses the design and delivery of computer-based assessments that incorporate standard test questions, simulated skill assessments and authentic tasks using actual applications in controlled conditions. 2: Educational and scientific importance Proficiency in information and communication technologies (ICT) has become important for life in modern society and the assessment of ICT literacy has become an important component of monitoring the extent to which students develop “skills and knowledge for tomorrow’s world” (OECD, 2004). A challenge for any assessment of ICT literacy is that of scalability versus authenticity. Assessments that can be administered consistently to large number of students as paper, on-line surveys and automated skills assessments, provide useful information but are limited in the analysis of complex behaviours such the ability to evaluate and integrate information. Assessment techniques that assess higher-level abilities (such as rubric-scored portfolios or observations) have proven to be difficult to scale above the classroom level, due to the resources required to ensure graders score work products consistently. Completing authentic tasks in real contexts was seen as fundamental to the design of this ICT literacy assessment and this symposium discusses the issues involved in developing an assessment that combines different types of task in the one assessment. 3. Assessment Instrument The assessment model defined a single variable, ICT literacy, which integrated three related strands: identifying, finding, organising and working with information; creating adapting, transforming and sharing information; and using ICT in a social context. The assessment instrument consisted of seven discrete thematic modules in a rotated design where all students attempted a common module called the General Skills Test (GST) and two of six Hybrid Assessment Modules (HAMs). The GST included only simulation and multiple-choice assessment items. The HAMs integrated conventional simulation, multiple-choice and constructed response items with live application software. Conducting the assessment with a number of HAMs ensured that the assessment instrument assessed what is common to the ICT construct across a breadth of contexts. 4: Data Data were gathered from students in a nationally representative sample of schools selected with a probability proportional to size and 15 students randomly selected from each school. A total of 3,746 Year 6 and 3,647 Year 10 students completed the assessment in 263 elementary and 257 secondary schools across Australia. The assessment was administered using 21 networks (or mini-labs) taken into schools by administrators. Student response to simulation items were scored automatically and products created through live applications were saved for later assessment by trained raters. 5: Analysis The assessment was analysed using Rasch modelling techniques (Andrich, Sheridan & Lou, 2004: Wu, Adams & Wilson, 1998). It investigated the dimensionality of the scale and equated the modules in the rotated design (on the basis of common person responses) because they were not intended to be of equal difficulty. In fact they ranged in average difficulty of -1.54 to +1.73 logits. A common scale across Years 6 and 10 was established using 32 items (of the total of 162 items) as links between the two levels. The process for defining which items were used as links in calibrating the scale was exhaustive in reviewing differential item functioning (DIF) and dependencies between items within modules. An investigation of dependencies resulted in some items based on common stimulus material and with similar content being combined into polytomous items. The scale was calibrated using a concurrent analysis with 7,363 cases. The calibration provided a high person separation index of 0.93 and a difference in the mean Year 6 ability compared to the mean Year 10 ability being of the order of 1.69 logits.. 6: Results. For reporting and analysis student scores on the ICT scale were transformed so that for Year 6 students the mean score was 400 points and the standard deviation was 100. In addition descriptions of the scale were developed around six proficiency levels with each spanning a bandwidth of 1.25 logits. These were characterised on the basis of the substantive ICT literacy content contained within the boundaries of each level. Cut-points to define marginal ICT literacy of Year 6 and Year 10 students were established in consultation with a pool of national ICT in education experts. The national survey gathered information about student characteristics and their access to ICT resources. There was a substantial difference between Year 6 and Year 10 students but the differences between females and males were not significant. There was a significant influence of family socioeconomic status but no significant influence of language background. High scores on the ICT literacy scale were associated with greater use of computers at home and greater use for purposes such as accessing the internet for information, entertainment or downloading material as well as word processing. Results are reported on the basis of a multi-level analysis of student and school factors associated with ICT literacy. References Andrich, D., Lyne, A., Sheridan, B. and Lou, G.(2004) Rasch Unidimensional Measurement Model 2020 (RUMM 2020) a Windows interactive program for analysing item response data according to the Rasch Unidimensional Measurement Model. Perth, Western Australia; RUMM Laboratory Pty Ltd. International ICT Literacy Panel (Educational Testing Service) (2002). Digital Transformation: A Framework for ICT Literacy. Princeton, NJ: Educational Testing Service Kelly, M. & Haber, J. (2006). National Educational Technology Standards for Students (NETS*S): Resources for Assessment. The International Society for Technology and Education : Eugene, OR. Wu. M.L., Adams, R.J and Wilson, M.R. (1998) ACER Conquest Generalised Item Response Modelling Software. Melbourne: ACER |
| Keywords | Assessment Computerized testing Large-scale national assessment projects |
| Appendices | |
| Authors | ||||||
|---|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | Presenting | |
| John | Ainley | Australian Council for Educational Research | Australia | ainley@acer.edu.au | * | |
| Julian | Fraillon | Australian Council for Educational Research | Australia | fraillon@acer.edu.au | ||
| Chris | Freeman | Australian Council fro Educational Research | Australia | freeman@acer.edu.au | ||
| Juliette | Mendelovits | Australian Council for Educational Research | Australia | mendelovits@acer.edu.au | ||
