Üdvözlettel,

Kovács Kristóf

Inflated effects in empirical research are ubiquitous but become smaller over time: Meta-meta-analytical evidence for the decline effect

Principles of a-priori hypothesizing, care- and thoughtful study design, and effect corroboration by direct replication are standards that ensure the meaningfulness of obtained results in empirical research and that are embraced by virtually all empirical researchers. Recently, however, the trust in empirical research in general and Psychological Science in particular has been undermined by unreliable effect estimates, biased results, and lacking reproducibility. Strategic researcher behaviors and publication process-related mechanisms that promote the publication of striking (but wrong) or inflated results were frequently cited as potential drivers for effect invalidity and misrepresentations. Although effect declines have been documented in a number of specific cases in the literature, no systematic account of effect changes over time is to date available. Consequently, in this presentation, I provide evidence for cross-temporal effect changes (regardless of the respective research question) based on more than 400 meta-analyses (N = 270,000,000+; k = 28,000+) that have been published in five flagship journals in Psychology. On the whole, analyses of effect trajectories indicate average effect changes of about a small effect size (i.e., r = .10) for every 15 years that elapsed following their initial publication (i.e., the first account that has been published for a given research question). Importantly, the aggregated evidence shows that effect declines outnumber increases at a ratio of about 3:2 and are on average twice as strong. Moreover, larger exploratory and summary effects appear to be associated with more substantial effect declines. Remedies for such systematic ubiquitous effect misrepresentations warrant changes in scientific quality control (i.e., the review process) and the evaluation of academics (e.g., in terms of tenure or promotion committees). Promoting study preregistration, the publication of primary data, discovery- and replication-sampling approaches, as well as the use of safeguard power in replications may be first steps in the right direction.