Lowering the p value threshold in recently published Respiratory Medicine RCTs

Abstract

Introduction: The use and misuse of p values remains controversial, with increasing support for lowering the threshold of statistical significance from 0.05 to 0.005, to reduce false positive results. A recent evaluation of randomised controlled trials (RCTs) determined that 70% would maintain significance if the p value threshold was <0.005¹. We sought to evaluate the impact of lowering the p value threshold in RCTs published in major respiratory journals.

Methods: RCTs published in the American Journal of Respiratory and Critical Care Medicine (AJRCCM), European Respiratory Journal (ERJ), Chest, Lancet Respiratory Medicine and Thorax (2017-2018) were identified. Pooled analyses and RCTs using Bayesian or noninferiority analyses were excluded. Data extracted included whether each RCT was multicentre, multinational, a drug trial and sample size.

Results: 125 studies were included. 62/125 (49.6%) reported a significant primary end point with p value<0.05. Of these, 31/62 (50.0%) were less than 0.005. Of the 125 p values 31% in AJRCCM and 27% in ERJ were <0.005. Multicenter trials were more likely to result in p values<0.005 (OR 9.4; p<0.0001).

Discussion: Of statistically significant primary end points published in respiratory medicine RCTs, only 50% would maintain statistical significance with a threshold of 0.005. This may be because treatment effects are not large enough to be assessed at this level of significance. Lowering the p value threshold would have an important impact on the clinical applicability of RCT findings and alternative methods such as Bayesian analysis should be explored as potential solutions to the controversy of p values.

Reference:

[1] JAMA. 2016;315(11):1141-1148.