Proposal view
| Proposal Type: | Individual Paper |
|---|---|
| Domain: | Assessment and Evaluation |
| SIG: | Assessment and Evaluation |
| Type | Submitted Paper |
| Equipment |
PC and projector |
| Paper Details |
|---|
| Title | Meta-analysis for repeated measures designs in educational research – options and challenges |
|---|---|
| Abstract | In empirical educational research, meta-analysis is widely considered the most adequate statistical tool in establishing efficacy of interventions. Meta-analysis uses the summary statistics (effect sizes) from individual studies as raw data and analyzes the heterogeneity of effect sizes across studies. In educational research, key concepts like learning are time-dependent processes by definition. Hence, the evaluation of intervention effects or long-term trajectories inevitably requires longitudinal or repeated- measurement designs. However, until recently, the statistical models to deal with within-subject designs within a meta-analytical framework were based on unrealistic model assumptions. We review the core statistical problems and discuss current approaches to solve them. All analytical steps and recommendations are illustrated with meta-analytical data on the effect of school type on student achievement gains. Recommendations for research practice will be derived. |
| Summary | Meta-analysis for repeated measures designs in educational research – options and challenges Aims: The search for empirically validated interventions and the subsequent promotion of evidence-based interventions (EBI) are key issues in the public debate about science-to-practice-transfer in educational research and school psychology. Meta-analysis is widely considered an adequate statistical tool in establishing efficacy of interventions and an important statistical tool to evaluate their effectiveness under "real-world" conditions. In educational research and school psychology, key concepts like learning, competence gain or psychosocial development are time-dependent processes by definition. Hence, the evaluation of intervention effects or long-term trajectories inevitably requires longitudinal or repeated-measurement designs. However, until recently, the statistical models to deal with within-subject designs within a meta-analytical framework were based on unrealistic model assumptions. Therefore, the goal of this paper is to review some of the problems that arise when integrating results from longitudinal and repeated measures designs using meta-analysis and to point out the current approaches to solve the statistical shortcomings. We will also derive practical recommendations for applied researchers in the field of educational research. Throughout this paper we will illustrate all analytical steps and recommendations with our meta-analytical data on the effect of school type on student achievement gains. Review of available statistical approaches: More so than it is true for other complex statistical methods, the success of a meta-analysis depends on accurate decisions about the intended analysis. Important decisions in this process deal with (1) the choice of the appropriate effect size measure, (2) the availability of a known sampling error variance estimate for the chosen effect size, and (3) the specification of a flexible meta-analytical model to be used with longitudinal data. The remainder of this paper will deal with these decisional steps by looking at basic repeated measures designs that are typical in intervention evaluation research. As a number of authors have pointed out (cf. Morris & DeShon, 2002), many published meta-analyses and even textbooks on meta-analysis fail to address the issue of alternative effect size metrics. Take, for example, a simple single-group pretest-posttest design evaluating a specific intervention effect: Some studies in the meta-analytic collection of primary studies will yield an effect size in raw score metric, i.e. dividing the mean difference in posttest and pretest scores by the standard deviation of the pretest using the idea proposed by Cohen (SDpre). Other studies, however, will only provide sufficient statistics in change score metric, i.e. the mean posttest-pretest difference divided by the standard deviation of the pre-post difference scores(SDdiff) . Effect size estimates can only be compared in a meaningful way across studies if all effect sizes are scaled in the same metric. Although the mean difference is identical in both metrics the effect sizes can differ substantially because the mean difference is related to populations with different standard deviations. The difference between SDpre and SDdiff is a function of the correlation between pre- and posttest scores (within-subject stability). When r is greater than .5, SDdiff will be smaller than SDpre, and consequently, the change score effect size will be larger than the raw score effect size. The pretest-posttest correlation obviously cannot change the size of the effect itself, but it makes the effect more "visible" by reducing the standard error. It has been demonstrated that the difference metric for a (reasonable) pre-post correlation of r = .75 will double the effect size compared to the raw score effect size metric. To avoid such non-ignorable inconsistencies, the meta-analyst has to make an informed decision to transform all effect sizes into either metric. To accomplish this, an estimate of the population correlation between pre- and posttest scores is required. Unfortunately, however, the correlation between measures is almost never reported in publications. Some meta-analysts try to cope with this adversity by restricting the included study sample to those studies which report results in raw score metric. However, as Morris and DeShon (2002) point out, the pretest–posttest correlation is also used in the estimate of the sampling error variance and, therefore, will have to be estimated regardless of which approach is used. Sampling error variance refers to the extent to which a statistic is expected to vary from study to study, simply as a function of sampling error. Estimates of sampling error are used in a meta-analysis for both, computing the overall mean across studies and testing the homogeneity of effect sizes. Sampling error variance is, simply put, a function of the sample size but is also influenced by the study design. For example, when r is large, the repeated measures design will provide more precise estimates of population parameters, and the resulting effect size will have a smaller sampling error variance. The only viable way to deal with this lack of data is to estimate the size of the correlation between measures from previous findings, e.g. from published reliability estimates, or to ask researchers of primary studies to calculate these estimates and provide them to the meta-analyst. Theoretical and educational significance: Meta-analyses can be seen as the empirical building blocks for educational policy decisions and are increasingly considered as such. If, however, inconsistencies in meta-analytical procedures and incorrect estimates of global effects lead to false conclusions about efficacy and effectiveness of educational interventions or large-scale programs these building blocks are prone to cause distrust, especially among decision makers and practitioners. Only if care is taken to examine the correlational nature of repeated measures designs and to compute effect size correctly can the scientific and practical progress promised by meta-analysis be realized. Thus, in the long run editorial policies have to change to allow for sound longitudinal meta-analyses. Authors must be motivated to report pre-post correlations of their dependent measures, and meta-analysts should search for ways of presenting their results in a more "user-friendly" manner, e.g. by better visualizing meta-analytical results and providing easy- to-grasp effect size displays. References: Morris, S. B. & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105-125. |
| Keywords | Data analysis Generalizability theory Meta-analysis |
| Appendices | |
| Authors | ||||||
|---|---|---|---|---|---|---|
| Name | Surname | Institution | Country | EARLI Number | Presenting | |
| Hans A. | Pant | Humboldt-University Berlin | Germany | panter@isq-bb.de | * | |
| Kai S. | Cortina | University of Michigan | United States | kai.cortina@umich.edu | ||
