Not a panacea: the role of presenting descriptive statistics and effect sizes in abstracts on medical students' misconceptions about p-values.
Eisenhut Frieder S FS, Wiemann Leon L, Binder Karin K, Fischer Martin R MR et al.
Most scientific studies report inferential statistical results using p-values. For clinical decision-making, correctly interpreting the results of significance tests is particularly important, for example, to differentiate between statistical significance and clinical relevance. However, p-values are frequently misinterpreted, and reporting effect sizes alongside p-values has therefore been recommended to support the interpretation of statistical results. We examined 121 medical students' misconceptions about p-values when presented with three fictitious but realistic medical abstracts containing comparisons of group means and t-tests. Medical context, sample size, p-value, and effect size were systematically varied across the abstracts (abstract 1: N1 = N2 = 7,500; p < 0.001; Cohen's d = 0.12; abstract 2: N1 = N2 = 10; p = 0.04; Cohen's d = 1.00; abstract 3: N1 = N2 = 40; p = 0.09; Cohen's d = 0.38). We varied the amount of statistical information provided in each abstract (information set 1: sample size and p-value; information set 2: sample size, p-value, means, and standard deviations; information set 3: the same information and effect size Cohen's d). We examined whether additional statistical information was associated with fewer misconceptions and how the provided information influenced students' perceived effect size. In all versions, a high proportion of medical students across all semesters of study demonstrated multiple misconceptions about p-values. The number of endorsed misconceptions substantially decreased with the provision of more statistical information only in the context with a large sample size, a statistically significant result, and a very small effect size. In this context, providing additional statistical information reduced the tendency to infer clinical relevance and effect magnitude directly from statistical significance. Although reporting standardized effect sizes has repeatedly been recommended, their inclusion alone appears insufficient to prevent mis- and overinterpretations of inferential statistical results. The interpretation of statistical evidence should be addressed more intensively and longitudinally throughout medical education, including by explicitly addressing typical misconceptions. Alternative teaching approaches to statistical inference, including simulation-based approaches, should also be considered.