Benő Csapó: Comparative measure for the development of skills and competencies in cross-sectional surveys: Introducing the gamma coefficient

Procedural components of knowledge (skills, abilities, competencies, expertise etc.) have been receiving growing attention both in the development of curricula and in the design of evaluation instruments. The development of these components of knowledge span several years, and in order to monitor their development practically the same (or parallel) tests may be administered to students of different ages. This way, the occasions when the achievements of students of different ages have to be compared become more frequent. This paper introduces a simple and standard way of expressing changes occurring in the cousse of one academic year, and illustrates how the proposed method has been put into practice by reanalyzing and comparing the results of some recent large-scale assessment projects. To express developmental data obtained from cross-sectional assessment in a standard form, a coefficient called gamma is proposed. The gamma coefficient is computed so that the difference of the means (measured in the same period of the academic year but in different grades) is divided by the mean of the two standard deviations, and then this ratio is divided by the number of years between the two grades. gamma = {(Mean2-Mean1) / [(StdDev1+StdDev2)/2]}/(Grade2-Grade1) Thus gamma is the standard measure of development of the assessed sample (population); it is the change in a skill that takes place during one academic year, expressed in standard deviation units. Gamma, defined in this way, is analogous with the measure of effect size (denoted by `d') that is used for presenting the results of training experiments and as the basis of meta-analyses. Therefore, the natural development of a skill and the development as a result of specific training can be compared directly. The use of gamma is illustrated by data from previous surveys on combinative, inductive, logical and proportional reasoning, classification skills, word problem solving skills and the Raven Intelligence Test. Gamma values were computed from the means and standard deviations of the achievements in two neighboring measurement points. The gamma values presented in this paper typically range from ca. 0.1 to 0.4 with some interesting exceptions and anomalies. The results presented in this paper show that the use of gamma allows for several fruitful comparisons and poses questions for further investigations. (E.g. are low gamma values due to the low sensibility/modifiability of a given skill in a period, or to the weakness of stimuli that effect the development during that period?) Standard methods of processing data and presenting results make it easier to synthesize the research findings.

MAGYAR PEDAGÓGIA 102. Number 3. 391-410. (2002)

Levelezési cím / Address for correspondence: Department of Education, University of Szeged, H­6722 Szeged, Petőfi sgt. 30-­34.

 
PDF