Probability of success

The probability of success (POS) is a statistics concept commonly used in the pharmaceutical industry including by health authorities to support decision making.

The probability of success is a concept closely related to conditional power and predictive power. Conditional power is the probability of observing statistical significance given the observed data assuming the treatment effect parameter equals a specific value. Conditional power is often criticized for this assumption. If we know the exact value of the treatment effect, there is no need to do the experiment. To address this issue, we can consider conditional power in a Bayesian setting by considering the treatment effect parameter to be a random variable. Taking the expected value of the conditional power with respect to the posterior distribution of the parameter gives the predictive power. Predictive power can also be calculated in a frequentist setting. No matter how it is calculated, predictive power is a random variable since it is a conditional probability conditioned on randomly observed data. Both conditional power and predictive power use statistical significance as the success criterion. However, statistical significance is often not sufficient to define success. For example, a health authority often requires the magnitude of the treatment effect to be bigger than an effect which is merely statistically significant in order to support successful registration. In order to address this issue, we can extend conditional power and predictive power to the concept of probability of success. For probability of success, the success criterion is not restricted to statistical significance. It can be something else such as a clinical meaningful result.

Types of POS

• Conditional probability of success (CPOS): It is the probability of observing success (in terms of the observed result) in the future given the observed data and the treatment effect equaling a specific value. CPOS is an extension of conditional power. Its success criteria are not restricted to statistical significance. However when the success is defined as statistical significance, it becomes conditional power.
• Predictive probability of success (PPOS): It is the probability of observing success in the future given the observed data. PPOS is an extension of predictive power. Its success criteria are not restricted to statistical significance. However when the success is defined as statistical significance, it becomes predictive power. Note that PPOS is a conditional probability conditioned on randomly observed data. Hence it is a random variable.
• Posterior probability of success (OPOS): It is the probability of success (in terms of the treatment effect parameter) calculated using posterior probability. Note that OPOS is a conditional probability conditioned on randomly observed data. Hence it is a random variable.

Application in clinical trials design

Pilot trial design using PPOS

Traditional pilot trial design is typically done by controlling type I error rate and power for detecting a specific parameter value. The goal of a pilot trial such as a phase II trial is usually not to support registration. Therefore it doesn't make sense to control type I error rate, especially a big type I error, as typically done in a phase II trial. A pilot trial usually provides evidence to support a Go/No Go decision for a confirmatory trial. Therefore it makes more sense to design a trial based on PPOS. To support a No/Go decision, traditional methods require the PPOS to be small. However the PPOS can be small just due to chance. To solve this issue, we can require the PPOS credible interval to be tight such that the PPOS calculation is supported by sufficient information and hence PPOS is not small just due to chance. Finding an optimal design is equivalent to find the solution to the following 2 equations.

1. PPOS=PPOS1
2. upper bound of PPOS credible interval=PPOS2

where PPOS1 and PPOS2 are some user-defined cutoff values. The first equation ensures that the PPOS is small such that not too many trials will be prevented entering next stage, to guard against false negatives. The first equation also ensures that the PPOS is not too small such that not too many trials will enter the next stage, to guard against false positives. The second equation ensures that the PPOS credible interval is tight such that the PPOS calculation is supported by sufficient information. The second equation also ensures that the PPOS credible interval is not too tight such that it won't demand too many resources.

Futility interim design using PPOS

Traditional futility interim is designed based on beta spending. However beta spending doesn't have an intuitive interpretation. Therefore it is difficult to communicate to non-statistician colleagues. Since PPOS has an intuitive interpretation, it makes more sense to design futility interim using PPOS. To declare futility, we mandate the PPOS to be small and PPOS calculation to be supported by sufficient information. According to Tang, 2015 finding the optimal design is equivalent to solving the following 2 equations.

1. PPOS=PPOS1
2. upper bound of PPOS credible interval=PPOS2

Defensive efficacy interim design using CPOS

Traditional efficacy interim is designed based on spending functions. Since spending functions don't have an intuitive interpretation, it is difficult to communicate to non-statistician colleagues. In contrast probability of success has an intuitive interpretation and hence can facilitate communication with non-statistician colleagues. Tang (2016) proposes the use of the following criteria to support efficacy interim decision making: mCPOS>c1 lCPOS>c2 where mCPOS is the median of CPOS with respect to the distribution of the parameter and lCPOS is the lower bound of the credible interval of CPOS. The first criterion ensures that the probability of success is large. The second criterion ensures that the credible interval of CPOS is tight; the CPOS calculation is supported by enough information; hence the probability of success is not large by chance. Finding the optimal design is equivalent to finding the solution to the following equations:

1. mCPOS=c1
2. lCPOS=c2

See also

• Credible interval
• Posterior probability
• Interim analysis