What is a Second-Generation p-value?
The second-generation p-value (SGPV) is an extension of the p-value that formally accounts for scientific relevance. It turns out that verifying that a statistically significant result is scientifically meaningful is not only good scientific practice, but it is also a natural way to control the Type I error rate. As a result, the second-generation p-value has nice frequency properties that are automatically controlled through the sample size.
The second-generation p-value is essentially the proportion of data-supported hypotheses that are null, or practically null, hypotheses. This is an advance for data-rich environments, where traditional p-value adjustments are needlessly punitive.
About the SGPV
Interpretation and Properties
The second-generation p-value, denoted by Pδ, depends on a interval null hypothesis. The subscript δ denotes this dependence and distinguishes it from the classical p-value.
Interval null hypotheses are constructed by incorporating information about the scientific context – such as inherent limits on measurement precision, clinical significance, or scientific significance – into statistical hypotheses that are stated a priori. The interval null should contain, in addition to the precise point null hypothesis, all other point hypotheses that are practically null and would maintain the scientific null premise. While the point null may be numerically distinct, all the hypotheses in the interval null are considered scientifically equivalent to the null premise.
Given a interval null hypothesis, the second-generation p-value, Pδ, has the following properties:
The second-generation p-value, Pδ, is a number between 0 and 1.
Pδ is essentially the fraction of data-supported hypotheses that are null hypotheses.
When Pδ = 0, the data only support hypotheses that are scientifically or clinically meaningful, i.e., those that are meaningful alternative hypotheses.
When Pδ = 1, the data only support null, or practically null, hypotheses, i.e., those that are not scientifically or clinically meaningful.
When Pδ ≈ 1/2, the data are strictly inconclusive. The degree of inconclusiveness is represented by Pδ.
Pδ has improved error rate control.
Pδ, because it is not a probability, is not adjusted for multiple comparisons.
A distinguishing feature of second-generation p-values is that they are intended as summary statistics that indicate when a study has met its a priori defined endpoint.