Tibor Vígh : Methodological questions of building an IRT-based item bank for foreign language skills Results of analyzing of the Hungarian Matura exam tasks in German language

Launched in Hungary in 2005, the new foreign language Matura exam aims to assess and evaluate communicative language competence, conforming to the Common European Framework of Reference (CEFR). Though the ordinary and the advanced levels of the exam are designed to correspond to levels A2-B1 and B2 CEFR levels, respectively, there has been no empirical evidence to confirm this link. This paper proposes (1) to examine the extent to which the empirical difficulty levels of the Matura tasks cover the ability levels of exam-takers; (2) to investigate the possibilities of creating an IRT based language item bank representing the CEFR levels; and (3) to investigate the methodological issues and problems emerging in this process. The first part gives an overview of the bases and models of item response theory (IRT), as well as its application for assessing language skills. The second part discusses the necessity and problems of the empirical foundation of Matura language levels. The third part presents the results of the analysis of a data base. The 2006 ordinary level German as a foreign language Matura items in reading, listening comprehension and use of German were examined using the Rasch model. The fourth part presents the independent analysis of the person-item maps of the 2005 and 2006 ordinary and advanced level German Matura booklets. Arguments are presented for the necessity of examining the equivalence of booklets, for the analysis of the two Matura levels and the CEFR levels in a new assessment with the use of anchor and reference items, and for the creation of an IRT based item bank. Findings from European projects (e.g. Dialang, Dutch grid and EBAFLS) may be useful in selecting reference tasks and in characterising texts and items for the Matura tasks. The proposed item bank could function as a reference point for the development of exam tasks and could contribute to the empirical verification of the correspondence between the Matura and the CEFR language levels.

MAGYAR PEDAGÓGIA 108. Number 1. 29-51. (2008)

Levelezési cím / Address for correspondence: Vígh Tibor Szegedi Tudományegyetem, Nevelés¬tudo¬má¬nyi Doktori Iskola, H–6722 Szeged, Petőfi S. sgt. 30–34.

 
PDF