Issues in the use of adaptive clinical trial designs

Sequential sampling plans are often used in the monitoring of clinical trials in order to address the ethical and efficiency issues inherent in human testing of a new treatment or preventive agent for disease. Group sequential stopping rules are perhaps the most commonly used approaches, but in recent years, a number of authors have proposed adaptive methods of choosing a stopping rule. In general, such adaptive approaches come at a price of inefficiency (almost always) and clouding of the scientific question (sometimes). In this paper, I review the degree of adaptation possible within the largely prespecified group sequential stopping rules, and discuss the operating characteristics that can be characterized fully prior to collection of the data. I then discuss the greater flexibility possible when using several of the adaptive approaches receiving the greatest attention in the statistical literature and conclude with a discussion of the scientific and statistical issues raised by their use.     

Frequentist evaluation of group sequential clinical trial designs

Group sequential stopping rules are often used as guidelines in the monitoring of clinical trials in order to address the ethical and efficiency issues inherent in human testing of a new treatment or preventive agent for disease. Such stopping rules have been proposed based on a variety of different criteria, both scientific (e.g. estimates of treatment effect) and statistical (e.g. frequentist type I error, Bayesian posterior probabilities, stochastic curtailment). It is easily shown, however, that a stopping rule based on one of these criteria induces a stopping rule on all other criteria. Thus, the basis used to initially define a stopping rule is relatively unimportant so long as the operating characteristics of the stopping rule are fully investigated. In this paper we describe how the frequentist operating characteristics of a particular stopping rule might be evaluated to ensure that the selected clinical trial design satisfies the constraints imposed by the many different disciplines represented by the clinical trial collaborators.     

Application of Bayesian statistics to decision making during a clinical trial.

We describe the application of Bayesian methods to the monitoring and analysis of a trial of treatment for patients with advanced colorectal carcinoma. We discuss the choice of prior distribution and justify the use of a truncated normal distribution with a probability mass at zero difference. The stopping rule, based on the trials of the posterior distribution and a chosen range of equivalence, yields an upper boundary very close to the Pocock group sequential boundary. The Bayes stopping rule is quite sensitive to the amount of probability mass at zero in the prior distribution.     

A Bayesian approach to sequential meta‐analysis

As evidence accumulates within a meta‐analysis, it is desirable to determine when the results could be considered conclusive to guide systematic review updates and future trial designs. Adapting sequential testing methodology from clinical trials for application to pooled meta‐analytic effect size estimates appears well suited for this objective. In this paper, we describe a Bayesian sequential meta‐analysis method, in which an informative heterogeneity prior is employed and stopping rule criteria are applied directly to the posterior distribution for the treatment effect parameter. Using simulation studies, we examine how well this approach performs under different parameter combinations by monitoring the proportion of sequential meta‐analyses that reach incorrect conclusions (to yield error rates), the number of studies required to reach conclusion, and the resulting parameter estimates. By adjusting the stopping rule thresholds, the overall error rates can be controlled within the target levels and are no higher than those of alternative frequentist and semi‐Bayes methods for the majority of the simulation scenarios. To illustrate the potential application of this method, we consider two contrasting meta‐analyses using data from the Cochrane Library and compare the results of employing different sequential methods while examining the effect of the heterogeneity prior in the proposed Bayesian approach. Copyright © 2016 John Wiley & Sons, Ltd.     

Multi‐arm group sequential designs with a simultaneous stopping rule

Multi‐arm group sequential clinical trials are efficient designs to compare multiple treatments to a control. They allow one to test for treatment effects already in interim analyses and can have a lower average sample number than fixed sample designs. Their operating characteristics depend on the stopping rule: We consider simultaneous stopping, where the whole trial is stopped as soon as for any of the arms the null hypothesis of no treatment effect can be rejected, and separate stopping, where only recruitment to arms for which a significant treatment effect could be demonstrated is stopped, but the other arms are continued. For both stopping rules, the family‐wise error rate can be controlled by the closed testing procedure applied to group sequential tests of intersection and elementary hypotheses. The group sequential boundaries for the separate stopping rule also control the family‐wise error rate if the simultaneous stopping rule is applied. However, we show that for the simultaneous stopping rule, one can apply improved, less conservative stopping boundaries for local tests of elementary hypotheses. We derive corresponding improved Pocock and O'Brien type boundaries as well as optimized boundaries to maximize the power or average sample number and investigate the operating characteristics and small sample properties of the resulting designs. To control the power to reject at least one null hypothesis, the simultaneous stopping rule requires a lower average sample number than the separate stopping rule. This comes at the cost of a lower power to reject all null hypotheses. Some of this loss in power can be regained by applying the improved stopping boundaries for the simultaneous stopping rule. The procedures are illustrated with clinical trials in systemic sclerosis and narcolepsy. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.     

Sequential urn designs with elimination for comparing K > or =3 treatments

A fully sequential procedure is proposed for comparing K > or =3 treatments with immediate binary responses. The procedure uses an adaptive urn design to randomize patients to the treatments and stopping rules are incorporated for eliminating less promising treatments. Simulation is used to assess the performance of the procedure for several adaptive urn designs, in terms of expected numbers of treatment failures and allocation proportions, and the effect on estimation at the end of the trial is also addressed. It is concluded that the drop-the-loser rule is more effective than equal allocation and all of the other designs considered. The practical benefits of the procedure are illustrated using the results of a three-treatment lung cancer study. It is then shown how the sequential elimination procedure may be used in dose-finding studies and its performance is compared with a recently proposed method. Several possible extensions to the work are briefly indicated.     

Bayesian decision sequential analysis with survival endpoint in phase II clinical trials

Chen and Chaloner (Statist. Med. 2006; 25 :2956–2966. DOI: 10.1002/sim.2429 ) present a Bayesian stopping rule for a single‐arm clinical trial with a binary endpoint. In some cases, earlier stopping may be possible by basing the stopping rule on the time to a binary event. We investigate the feasibility of computing exact, Bayesian, decision‐theoretic time‐to‐event stopping rules for a single‐arm group sequential non‐inferiority trial relative to an objective performance criterion. For a conjugate prior distribution, exponential failure time distribution, and linear and threshold loss structures, we obtain the optimal Bayes stopping rule by backward induction. We compute frequentist operating characteristics of including Type I error, statistical power, and expected run length. We also briefly address design issues. Copyright © 2009 John Wiley & Sons, Ltd.     

Randomised trials with provision for early stopping for benefit (or harm): The impact on the estimated treatment effect

Stopping rules for clinical trials are primarily intended to control Type I error rates if interim analyses are planned, but less is known about the impact that potential stopping has on estimating treatment benefit. In this paper, we derive analytic expressions for (1) the over-estimation of benefit in studies that stop early, (2) the under-estimation of benefit in completed studies, and (3) the overall bias in studies with a stopping rule. We also examine the probability of stopping early and the situation in meta-analyses. Numerical evaluations show that the greatest concern is with over-estimation of benefit in stopped studies, especially if the probability of stopping early is small. The overall bias is usually less than 10% of the true benefit, and under-estimation in completed studies is also typically small. The probability of stopping depends on the true treatment effect and sample size. The magnitude of these effects depends on the particular rule adopted, but we show that the maximum overall bias is the same for all stopping rules. We also show that an essentially unbiased meta-analysis estimate of benefit can be recovered, even if some component studies have stopping rules. We illustrate these methods using data from three clinical trials. The results confirm our earlier empirical work on clinical trials. Investigators may consult our numerical results for guidance on potential mis-estimation and bias in the treatment effect if a stopping rule is adopted. Particular concern is warranted in studies that actually stop early, where interim results may be quite misleading.     

Bayesian predictive approach to interim monitoring in clinical trials

This paper reviews Bayesian strategies for monitoring clinical trial data. It focuses on a Bayesian stochastic curtailment method based on the predictive probability of observing a clinically significant outcome at the scheduled end of the study given the observed data. The proposed method is applied to derive efficacy and futility stopping rules in clinical trials with continuous, normally distributed and binary endpoints. The sensitivity of the resulting stopping rules to the choice of prior distributions is examined and guidelines for choosing a prior distribution of the treatment effect are discussed. The Bayesian predictive approach is compared to the frequentist (conditional power) and mixed Bayesian-frequentist (predictive power) approaches. The interim monitoring strategies discussed in the paper are illustrated using examples from a small proof-of-concept study and a large mortality trial.     

The use of the triangular test with response-adaptive treatment allocation

A clinical trial is considered in which two treatments with binary responses are to be compared. A popular sequential stopping rule, the triangular test, is studied when various response-adaptive treatment allocation rules are applied, such as the recently proposed drop-the-loser rule, an urn randomization scheme. The paper extends previous work by Coad and Rosenberger, who combined the triangular test with the randomized play-the-winner rule. The purpose of the paper is to investigate to what extent the variability of an adaptive design affects the overall performance of the triangular test. The adaptive rules under consideration are described and some of their asymptotic properties are summarized. Simulation is then used to assess the performance of the triangular test when combined with the various adaptive rules. The main finding is that the drop-the-loser rule is the most promising of the adaptive rules considered in terms of a less variable allocation proportion and a smaller number of treatment failures. The use of this rule with the triangular test is beneficial compared with the triangular test with equal allocation, since it yields fewer treatment failures on average while providing comparable power with similar expected sample size. The results of an AIDS trial are used to illustrate the performance of the triangular test when combined with the drop-the-loser rule.     

The continual reassessment method and its applications: a Bayesian methodology for phase I cancer clinical trials

We discuss the continual reassessment method (CRM) and its extension with practical applications in phase I and I/II cancer clinical trials. The CRM has been proposed as an alternative design of a traditional cohort design and its essential features are the sequential (continual) selection of a dose level for the next patients based on the dose-toxicity relationship and the updating of the relationship based on patients' response data using Bayesian calculation. The original CRM has been criticized because it often tends to allocate too toxic doses to many patients and our proposal for overcoming this practical problem is to monitor a posterior density function of the occurrence of the dose limiting toxicity (DLT) at each dose level. A simulation study shows that strategies based on our proposal allocate a smaller number of patients to doses higher than the maximum tolerated dose (MTD) compared with the original method while the mean squared error of the probability of the DLT occurrence at the MTD is not inflated. We present a couple of extensions of the CRM with real prospective applications: (i) monitoring efficacy and toxicity simultaneously in a combination phase I/II trial; (ii) combining the idea of pharmacokinetically guided dose escalation (PKGDE) and utilization of animal toxicity data in determining the prior distribution. A stopping rule based on the idea of separation among the DLT density functions is discussed in the first example and a strategy for determining the model parameter of the dose-toxicity relationship is suggested in the second example.     

Early termination of a two‐stage study to develop and validate a panel of biomarkers

Two‐stage designs to develop and validate a panel of biomarkers present a natural setting for the inclusion of stopping rules for futility in the event of poor preliminary estimates of performance. We consider the design of a two‐stage study to develop and validate a panel of biomarkers where a predictive model is developed using a subset of the samples in stage 1 and the model is validated using the remainder of the samples in stage 2. First, we illustrate how we can implement a stopping rule for futility in a standard, two‐stage study for developing and validating a predictive model where samples are separated into a training sample and a validation sample. Simulation results indicate that our design has type I error rate and power similar to the fixed‐sample design but with a substantially reduced sample size under the null hypothesis. We then illustrate how we can include additional interim analyses in stage 2 by applying existing group sequential methodology, which results in even greater savings in the number of samples required under both the null and the alternative hypotheses. Our simulation results also illustrate that the operating characteristics of our design are robust to changes in the underlying marker distribution. Copyright © 2012 John Wiley & Sons, Ltd.     

Group sequential designs for cure rate models with early stopping in favour of the null hypothesis

Ewell and Ibrahim derived the large sample distribution of the logrank statistic under general local alternatives. Their asymptotic results enable us to extend several group sequential designs which allow for early stopping in favour of the null hypothesis to the setting in which the cure rate model is appropriate. In particular, we derive stopping rules for the cure rate model using conditional power, predictive power and repeated confidence intervals methods. We illustrate the methods proposed using a hypothetical phase III clinical trial which is typical for melanoma studies.     

Optimal continuous sequential boundaries for monitoring toxicity in clinical trials: a restricted search algorithm

Continuous monitoring of severe adverse experiences can ensure the timely termination of a clinical trial if the therapy is shown to be harmful. In this paper we present methods for choosing a stopping rule for continuous monitoring of toxicity in small trials. They are especially useful for small phase II trials of about 30 patients for monitoring a binary toxicity event that is observed relatively quickly compared to the efficacy outcome. In 1987 Goldman described an algorithm for computing the exact type I error rate (alpha) and power (1-beta) of a specified discrete stopping boundary for sequential monitoring of a study with a fixed maximum number of patients (N) to be enrolled on the experimental therapy. Only an upper boundary was used since trials are only terminated for an excess frequency of toxicity and not for a low rate. By repeated use of this algorithm a stopping rule can be identified which has nearly the chosen level of (alpha) and a reasonable power depending on the design parameters of the study. The work reported here embeds this earlier algorithm as a subroutine in a larger FORTRAN program which searches all boundaries that fulfil constraints on size and power, as specified by the user. The search is restricted so that only those boundaries with size in a small neighbourhood of the chosen alpha are examined and displayed if the power is above a set minimum. These restrictions reduce the number of boundaries examined to only 0.4 per cent of all possible boundaries, thus reducing running time to a practical few seconds. Many such boundaries exist, the one with the largest power can then be chosen for monitoring the trial. The average sample number (ASN) and the expected relative loss (ERL) are also computed. The criterion for choosing may also be based on small ASN or low ERL in addition to power and appropriate alpha.     

Application of a stopping rule based on total treatment failures: the postoperative Crohn's disease trial.

The Postoperative Crohn's Disease Trial (PCDT), a placebo-controlled randomized trial of Rowasa I in the prevention of postoperative recurrence of Crohn's disease, is used as an example of how a stopping rule based on total endpoint occurrences can provide considerable advantage over standard fixed sample size methods. It can be used when the primary outcome is occurrence or time to occurrence and does not raise the troublesome issues regarding the unblinding of group differences that other sequential methods create. The main advantage of the total endpoint stopping rule is that it provides set power. Standard fixed sample size designs provide a given power only on average. The power actually achieved in a particular fixed sample size trial is largely determined by the overall observed rate of endpoint occurrences. This claim about the total endpoint stopping rule is well established in the statistical literature and, as well as outlining the mathematical details in an Appendix, we use computer simulation of the PCDT to demonstrate that use of the stopping rule will allow termination of the trial while maintaining power and type I error at a predetermined level.     

Group sequential design for comparative diagnostic accuracy studies: implications and guidelines for practitioners.

PURPOSE: Comparative diagnostic accuracy (CDA) studies are typically small retrospective studies supporting a higher accuracy for one modality over another for either staging a particular disease or assessing response to therapy, and they are used to generate hypotheses for larger prospective trials. The purpose of this article is to introduce the group sequential design (GSD) approach in planning these larger trials. METHODS: Methodology needed for using GSD in the CDA studies is recently developed. In this article, GSD with the O'Brien and Fleming (OBF) stopping rule is described and guidelines for sample size calculation are provided. Simulated data is used to demonstrate the application of GSD in the design/analysis of a clinical trial in the CDA study setting. RESULTS: The expected sample size needed for planning a trial with GSD (under the OBF stopping rule) is slightly inflated but may ultimately result in greater savings of patient resources. CONCLUSION: GSD is a specialized statistical method that is helpful in balancing the ethical and financial advantages of stopping a study early against the risk of an incorrect conclusion and should be adopted for planning CDA studies.     

Supplementary analysis of probabilities at the termination of a group sequential phase II trial

We consider estimation of various probabilities after termination of a group sequential phase II trial. A motivating example is that the stopping rule of a phase II oncologic trial is determined solely based on response to a drug treatment, and at the end of the trial estimating the rate of toxicity and response is desirable. The conventional maximum likelihood estimator (sample proportion) of a probability is shown to be biased, and two alternative estimators are proposed to correct for bias, a bias-reduced estimator obtained by using Whitehead's bias-adjusted approach, and an unbiased estimator from the Rao-Blackwell method of conditioning. All three estimation procedures are shown to have certain invariance property in bias. Moreover, estimators of a probability and their bias and precision can be evaluated through the observed response rate and the stage at which the trial stops, thus avoiding extensive computation.     

Exact group-sequential designs for clinical trials with randomized play-the-winner allocation

The use of both sequential designs and adaptive treatment allocation are effective in reducing the number of patients receiving an inferior treatment in a clinical trial. In large samples, when the asymptotic normality of test statistics can be utilized, a standard sequential design can be combined with adaptive allocation. In small samples the planned error rate constraints may not be satisfied if normality is assumed. We address this problem by constructing sequential stopping rules with specified properties by consideration of the exact distribution of test statistics under a particular adaptive allocation scheme, the randomized play-the-winner rule. Using this approach, compared to traditional equal allocation trials, trials with adaptive allocation are shown to require a larger total sample size to achieve a given power. More interestingly, the expected number patients allocated to the inferior treatment may also be larger for the adaptive allocation designs depending on the true success rates.     

Stopping rules for surveys with multiple waves of nonrespondent follow-up

In surveys with multiple waves of follow-up, nonrespondents to the first wave are sometimes followed intensively but this does not guarantee an increase in the response rate or an appreciable change in the estimate of interest. Most prior research has focused on stopping rules for Phase I clinical trials. To our knowledge there are no standard methods to stop follow-up in observational studies. Previous research suggests optimal stopping strategies where decisions are based on achieving a given precision for minimum cost or reducing cost for a given precision. In this paper, we propose three stopping rules that are based on assessing whether successive waves of sampling provide evidence that the parameter of interest is changing. Two of the rules rely on examining patterns of observed responses while the third rule uses missing data methods to multiply impute missing responses. We also present results from a simulation study to evaluate our proposed methods. Our simulations suggest that rules that adjust for nonresponse are preferred for decisions to discontinue follow-up since they reduce bias in the estimate of interest. The rules are not complicated and may be applied in a straightforward manner. Discontinuing follow-up would save time and possibly resources, and adjusting for the nonresponse in the analysis would reduce the impact of nonresponse bias.     

Bayesian evaluation of group sequential clinical trial designs

Clinical trial designs often incorporate a sequential stopping rule to serve as a guide in the early termination of a study. When choosing a particular stopping rule, it is most common to examine frequentist operating characteristics such as type I error, statistical power, and precision of confidence intervals (Statist. Med. 2005, in revision). Increasingly, however, clinical trials are designed and analysed in the Bayesian paradigm. In this paper, we describe how the Bayesian operating characteristics of a particular stopping rule might be evaluated and communicated to the scientific community. In particular, we consider a choice of probability models and a family of prior distributions that allows concise presentation of Bayesian properties for a specified sampling plan.