Gyöngyvér Molnár: Assessment of knowledge application with IRT

This paper reports the results of an examination of the relationship between two ways of assessing students’ knowledge. More than 5000 Hungarian students (9 to 17-year-olds) were assessed in 2002 regarding their performance on reading, mathematics literacy and science tests as well as their application of the same knowledge in complex problem solving tasks. The test included multiple-choice, short answer, and extended response items. Two IRT programs were used to analyse the results, OPLM (Verhelst, Glas and Verstralen, 1995) and ConQuest (Wu, Adams and Wilson, 1997). This paper compares the results and the benefits of these alternatives. The model applied to the survey is a generalised form of the Rasch model. This is a mixed coefficients model where items are described by a fixed set of unknown parameters, while student outcome levels (the latent variable) are random effects. For each item parameter, the ConQuest fit mean square statistic index provided an indication of the compatibility between the model and the data. For each student, the model describes the probability of obtaining different item scores. Figures are included to show the distribution of Rasch-estimated item difficulties. The student achievement distribution is located parallel to the item difficulty distribution. This implies that, on average, the students in the study had an ability level that was adequate for a 50 percent chance of solving an average item correctly. The accumulation of comparisons across cases yields an item-fit statistic. Each of the domains was scaled separately to examine the targeting of the tests. Trend indicators show how results change over time. The outcomes draw a profile of useful knowledge and skills among 9- to 17-year-olds.

MAGYAR PEDAGÓGIA 103. Number 4. 423-446. (2003)

Levelezési cím / Address for correspondence: Molnár Gyöngyvér, Szegedi Tudomány¬egye¬tem, Pedagógiai Tanszék, MTA Képességkutató Csoport, H–6722 Szeged, Petőfi S. sgt. 30–34.

 
PDF