Andrea Kárpáti, András Zempléni, Norman D. Verhelst, Niels H. Velduijzen and Diederik W. Schönau: Expert Agreement in Judging art Projects

Between 1993–1995, a research project for the modernisation of the Hungarian system of final examination was carried out by the Hungarian Institute for Public Education in cooperation with Cito, the Dutch National Institute for Educational Measurement and the Hungarian Academy of Crafts and Design. The subjects involved were mathematics, English language, mother tongue, biology and visual arts. Beyond the adaptation of the Dutch examination methods, the aim of the authors of this paper was to improve and further develop the core element of the Dutch art examination, the jurying procedure of art and design projects prepared in the course of 3 months of the examination period. Jurying is a widely applied method of assessment in the visual arts, but we could not find any published research results on the validity and reliability of this procedure. As the new model of the final examinations in the arts in Hungary envisages the replacement of the current exam based on a set of academic drawing exercises with arts projects, it seems necessary to describe how many jurors, what sort of criteria and what form of assessing the works is necessary for reliable results. After comparative studies of educational documents and on-site observations of art teaching and examination practices by experts in both countries, we executed two experimental final examinations in art and design in Hungary. In the first trial, in 1994, 201 students from 21 schools representing 18 cities and towns in all geographical regions of the country did both the practical and the theoretical part of the Dutch final examination of that year. The 12 project tasks to choose from were unaltered – they included 6 two-dimensional tasks in graphic arts, painting, design and photography and 6 three-dimensional, plastic and design tasks. Portfolios of two projects (an expressive and a design task) were judged by a group of 37 trained jurors in the first jurying experiment and by 31 newly selected and trained jurors in the second. The analysis of the results shows that project-based tasks, especially design projects, can be reliably assessed if at least two, optimally three jurors are employed who base their judgements on a set of assessment criteria and not on their global impressions only. „Vertical assessment” - judging a work according to all criterion at the same time, the traditional method used by all fine arts competitions and exhibition jurying procedures - proved to be significantly less reliable than judging all the works according to one criterion at a time. This research proves that results of art, education can be reliably assessed through sophisticated tasks representing all genres of the visual arts not just drawing. But it also proves that traditional jurying practices need to be changed for the examination procedure. The employment of trained jurors, preferably one external and one or two internal (school-based) evaluators who compare all project portfolios (evidence of background studies, sketches, plans, variations and a final version of the project task) according to one given criterion at a time is necessary for reliable assessment.

MAGYAR PEDAGÓGIA 97. Number 3-4. 203-234. (1997)

Levelezési cím / Address for correspondence: Kárpáti Andrea, Eötvös Loránd Tudományegyetem Neveléstudományi Tanszék, H–1146 Budapest, Ajtósi-Dűrer sor 19–21.

 
PDF