Answer to "How do you aggregate data?"
Editor: Neil Rice
One of the important features of longitudinal programmatic assessments, as opposed to single examinations in a program of assessment, is a requirement to accurately record students’ progress to inform decisions on progression. In order to be able to measure progress, an appropriate method for aggregating previous and current performance data is required.
Normalised scoring: Students scores are standardised or scaled in some way so as to be directly comparable within and across cohorts and time. Normalisation of scores from different assessments using standardised scores enables comparison of students’ performance within a cohort and across assessments of different type and difficulty. With knowledge of test difficulty and cohort ability, it is a fairly simple psychometric analysis to calculate progress in absolute terms between one test point and subsequent test points using scaled (%) or standardised scores. In assessment systems where psychometric data is rich and much is known about the assessment constructs employed, the use normalised scores to aggregate assessment data is a powerful tool.
Aggregation of grades: Students are allocated an ordinal categorical grade for an individual assessment according to agreed criteria. Cumulative running grades are calculated on the basis of a student’s grades over a series of assessments. For example student scores in an assessment are divided into 4 categories – ‘excellent’, ‘satisfactory’, ‘doubtful’ and ‘unsatisfactory’, and a cumulative aggregate grade can be calculated on the basis of the last 2 (or more) tests. A student moving from a satisfactory grade to an unsatisfactory grade may be of more concern that a student moving from a doubtful grade to a satisfactory grade for example. Allocation and aggregation of grades is often an attractive method for measuring progression in programmatic assessments. Using small numbers of ordinal grades somewhat mediates the inherent inaccuracy of using continuous score scales in assessments of naturally varying difficulty across and within cohorts. Aggregating grades also has the advantage of that student performance is often non-linear throughout a course of studies and that performance in more recent assessments is often of more importance than historical performance.