Is statistical significance always significant?

One way in which we learn new information is to read the medical literature. Whether or not we do primary research, it is important to be able to read literature in a critical fashion. A seemingly simple concept in reading is to interpret p values. For most of us, if we find a p value that is <.05, we take the conclusion to heart and quote it at every opportunity. If the p value is >.05, we discard the paper and look elsewhere for useful information. Unfortunately, this is too simplistic an approach. The real utility of p values is to consider them within the context of the experiment being performed. Defects in study design can make an interpretation of a p value useless. One has to be wary of type I (seeing a "statistically significant" difference just because of chance) and type II (failing to see a difference that really exists) errors. Examples of the former are publication bias and the performance of multiple analyses; the latter refers to a trial that is too small to demonstrate the difference. Finding significant differences in surrogate or intermediate endpoints may not help us. We need to know if those endpoints reflect the behavior of clinical endpoints. Selectively citing significant differences and disregarding studies that do not find them is inappropriate. Small differences, even if they are statistically significant, may require too much resource expenditure to be clinically useful. This article explores these problems in depth and attempts to put p values in the context of studies.     

Tree-structured gatekeeping tests in clinical trials with hierarchically ordered multiple objectives

This paper discusses a new class of multiple testing procedures, tree-structured gatekeeping procedures, with clinical trial applications. These procedures arise in clinical trials with hierarchically ordered multiple objectives, for example, in the context of multiple dose-control tests with logical restrictions or analysis of multiple endpoints. The proposed approach is based on the principle of closed testing and generalizes the serial and parallel gatekeeping approaches developed by Westfall and Krishen (J. Statist. Planning Infer. 2001; 99:25-41) and Dmitrienko et al. (Statist. Med. 2003; 22:2387-2400). The proposed testing methodology is illustrated using a clinical trial with multiple endpoints (primary, secondary and tertiary) and multiple objectives (superiority and non-inferiority testing) as well as a dose-finding trial with multiple endpoints.     

A consistency‐adjusted alpha‐adaptive strategy for sequential testing

In a clinical trial with two clinically important endpoints, each of which can fully characterize a treatment benefit to support an efficacy claim by itself, a minimum degree of consistency in the findings is expected; otherwise interpretation of study findings can be problematic. Clinical trial literature contains examples where lack of consistency in the findings of clinically relevant endpoints led to difficulties in interpreting study results. The aim of this paper is to introduce this consistency concept at the study design stage and investigate the consequences of its implementation in the statistical analysis plan. The proposed methodology allows testing of hierarchically ordered endpoints to proceed as long as a pre‐specified consistency criterion is met. In addition, while an initial allocation of the alpha level is specified for the ordered endpoints at the design stage, the methodology allows the alpha level allocated to the second endpoint to be adaptive to the findings of the first endpoint. In addition, the methodology takes into account the correlation between the endpoints in calculating the significance level and the power of the test for the next endpoint. The proposed Consistency‐Adjusted Alpha‐Adaptive Strategy (CAAAS) is very general. Several of the well‐known multiplicity adjustment approaches arise as special cases of this strategy by appropriate selection of the consistency level and the form of alpha‐adaptation function. We discuss control of the Type I error rate as well as power of the proposed methodology and consider its application to clinical trial data. Published in 2010 by John Wiley & Sons, Ltd.     

Exploring the benefits of adaptive sequential designs in time-to-event endpoint settings

Sequential analysis is frequently employed to address ethical and financial issues in clinical trials. Sequential analysis may be performed using standard group sequential designs, or, more recently, with adaptive designs that use estimates of treatment effect to modify the maximal statistical information to be collected. In the general setting in which statistical information and clinical trial costs are functions of the number of subjects used, it has yet to be established whether there is any major efficiency advantage to adaptive designs over traditional group sequential designs. In survival analysis, however, statistical information (and hence efficiency) is most closely related to the observed number of events, while trial costs still depend on the number of patients accrued. As the number of subjects may dominate the cost of a trial, an adaptive design that specifies a reduced maximal possible sample size when an extreme treatment effect has been observed may allow early termination of accrual and therefore a more cost-efficient trial. We investigate and compare the tradeoffs between efficiency (as measured by average number of observed events required), power, and cost (a function of the number of subjects accrued and length of observation) for standard group sequential methods and an adaptive design that allows for early termination of accrual. We find that when certain trial design parameters are constrained, an adaptive approach to terminating subject accrual may improve upon the cost efficiency of a group sequential clinical trial investigating time-to-event endpoints. However, when the spectrum of group sequential designs considered is broadened, the advantage of the adaptive designs is less clear.     

Late-starter sites in randomized controlled trials

In a cohort of 14 randomized controlled trials conducted by the Adult AIDS Clinical Trials Group between 1986 and 1999 with a target sample size of >400 (total enrollment 15,531 patients), we evaluated whether "late-starter" sites can make a meaningful contribution to eventual trial accrual. The sites that started recruiting within 5 months from the time the first patient entered the trial were eventually responsible for over 90% of the total enrollment in 11 of the 14 trials. Across the 14 trials, some sites were consistently among the first to start enrollment, whereas others were routinely among the last. The late-starter sites are unlikely to make important contributions to eventual trial enrollment in large clinical trials conducted by groups with a fixed number of sites. Protracting administrative efforts to add more sites many months after a multicenter trial has started may not be useful to trial accrual.     

Predictive probability of success using surrogate endpoints

The predictive probability of success of a future clinical trial is a key quantitative tool for decision-making in drug development. It is derived from prior knowledge and available evidence, and the latter typically comes from the accumulated data on the clinical endpoint of interest in previous clinical trials. However, a surrogate endpoint could be used as primary endpoint in early development and, usually, no or limited data are collected on the clinical endpoint of interest. We propose a general, reliable, and broadly applicable methodology to predict the success of a future trial from surrogate endpoints, in a way that makes the best use of all the available evidence. The predictions are based on an informative prior, called surrogate prior, derived from the results of past trials on one or several surrogate endpoints. If available, in a Bayesian framework, this prior could be combined with data from past trials on the clinical endpoint of interest. Two methods are proposed to address a potential discordance between the surrogate prior and the data on the clinical endpoint. We investigate the patterns of behavior of the predictions in a comprehensive simulation study, and we present an application to the development of a drug in Multiple Sclerosis. The proposed methodology is expected to support decision-making in many different situations, since the use of predictive markers is important to accelerate drug developments and to select promising drug candidates, better and earlier.     

Changes are still needed on multiple co‐primary endpoints

The Food and Drug Administration in the United States issued a much‐awaited draft guidance on ‘Multiple Endpoints in Clinical Trials’ in January 2017. The draft guidance is well written and contains consistent message on the technical implementation of the principles laid out in the guidance. In this commentary, we raise a question on applying the principles to studies designed from a safety perspective. We then direct our attention to issues related to multiple co‐primary endpoints. In a paper published in the Drug Information Journal in 2007, Offen et al. give examples of disorders where multiple co‐primary endpoints are required by regulators. The standard test for multiple co‐primary endpoints is the min test which tests each endpoint individually, at the one‐sided 2.5% level, for a confirmatory trial. This approach leads to a substantial loss of power when the number of co‐primary endpoints exceeds 2, a fact acknowledged in the draft guidance. We review approaches that have been proposed to tackle the problem of power loss and propose a new one. Using recommendations by Chen et al. for the assessment of drugs for vulvar and vaginal atrophy published in the Drug Information Journal in 2010, we argue the need for more changes and urge a path forward that uses different levels of claims to reflect the effectiveness of a product on multiple endpoints that are equally important and mostly unrelated. Copyright © 2017 John Wiley & Sons, Ltd.     

Statistical modeling and prediction of clinical trial recruitment

Real-time prediction of clinical trial accrual can support logistical planning, ensuring that studies meet but do not exceed sample size targets. We describe a novel, simulation-based prediction method that is founded on a realistic model for the underlying processes of recruitment. The model reflects key features of enrollment such as the staggered initiation of new centers, heterogeneity in enrollment capacity, and declining accrual within centers. The model's first stage assumes that centers join the trial (ie, initiate accrual) according to an inhomogeneous Poisson process in discrete time. The second part assumes that each center's enrollment pattern reflects an early plateau followed by a slow decline, with a burst at the end of the trial following the announcement of an imminent closing date. By summing up achieved and projected enrollment, one can predict accrual as a function of time and, thereby, the time when the trial will achieve a planned enrollment target. We applied our method retrospectively to two real-world trials: NSABP B-38 and REMATCH (Randomized Evaluation of Mechanical Assistance for the Treatment of Congestive Heart Failure). In both studies, the proposed method produced prediction intervals for time to completion that were more accurate than those from conventional predictions that assume a constant rate of enrollment, estimated either from the entire trial to date or over a recent time window. The advantage is substantial in the early stages of NSABP B-38. We conclude that a method based on a realistic accrual model offers improved accuracy in the prediction of enrollment landmarks, especially at the early stages of large trials that involve many centers.     

Methods and challenges in measuring the impact of national pneumococcal and rotavirus vaccine introduction on morbidity and mortality in Malawi

BackgroundPneumonia and gastroenteritis are leading causes of vaccine-preventable childhood morbidity and mortality. Malawi introduced pneumococcal conjugate and rotavirus vaccines to the immunisation programme in 2011 and 2012, respectively. Evaluating their effectiveness is vital to ensure optimal implementation and justify sustained investment.Methods/DesignA national evaluation platform was established to determine vaccine effectiveness and impact in Malawi. Impact and effectiveness against vaccine-type invasive pneumococcal disease, radiological pneumonia and rotavirus gastroenteritis are investigated using before-after incidence comparisons and case-control designs, respectively. Mortality is assessed using a prospective population cohort. Cost-effectiveness evaluation is nested within the case-control studies. We describe platform characteristics including strengths and weaknesses for conducting vaccine evaluations.DiscussionIntegrating data from individual level and ecological methods across multiple sites provides comprehensive information for policymakers on programme impact and vaccine effectiveness including changes in serotype/genotype distribution over time. Challenges to robust vaccine evaluation in real-world conditions include: vaccination ascertainment; pre-existing rapid decline in mortality and pneumococcal disease in the context of non-vaccine interventions; and the maintenance of completeness and quality of reporting at scale and over time. In observational non-randomised designs ascertainment of vaccine status may be biased particularly in infants with fatal outcomes. In the context of multiple population level interventions targeting study endpoints attribution of reduced incidence to vaccine impact may be flawed. Providing evidence from several independent but complementary studies will provide the greatest confidence in assigning impact. Welcome declines in disease incidence and in child mortality make accrual of required sample sizes difficult, necessitating large studies to detect the relatively small but potentially significant contribution of vaccines to mortality prevention. Careful evaluation of vaccine effectiveness and impact in such settings is critical to sustaining support for vaccine programmes. Our evaluation platform covers a large population with a high prevalence of HIV and malnutrition and its findings will be relevant to other settings in sub-Saharan Africa.     

Sample size calculation for the proportional hazards cure model

In clinical trials with time‐to‐event endpoints, it is not uncommon to see a significant proportion of patients being cured (or long‐term survivors), such as trials for the non‐Hodgkins lymphoma disease. The popularly used sample size formula derived under the proportional hazards (PH) model may not be proper to design a survival trial with a cure fraction, because the PH model assumption may be violated. To account for a cure fraction, the PH cure model is widely used in practice, where a PH model is used for survival times of uncured patients and a logistic distribution is used for the probability of patients being cured. In this paper, we develop a sample size formula on the basis of the PH cure model by investigating the asymptotic distributions of the standard weighted log‐rank statistics under the null and local alternative hypotheses. The derived sample size formula under the PH cure model is more flexible because it can be used to test the differences in the short‐term survival and/or cure fraction. Furthermore, we also investigate as numerical examples the impacts of accrual methods and durations of accrual and follow‐up periods on sample size calculation. The results show that ignoring the cure rate in sample size calculation can lead to either underpowered or overpowered studies. We evaluate the performance of the proposed formula by simulation studies and provide an example to illustrate its application with the use of data from a melanoma trial. Copyright © 2012 John Wiley & Sons, Ltd.     

Predicting accrual in clinical trials with Bayesian posterior predictive distributions

Investigators need good statistical tools for the initial planning and for the ongoing monitoring of clinical trials. In particular, they need to carefully consider the accrual rate-how rapidly patients are being recruited into the clinical trial. A slow accrual decreases the likelihood that the research will provide results at the end of the trial with sufficient precision (or power) to make meaningful scientific inferences. In this paper, we present a method for predicting accrual. Using a Bayesian framework we combine prior information with the information known up to a monitoring point to obtain a prediction. We provide posterior predictive distributions of the accrual. The approach is attractive since it accounts for both parameter and sampling distribution uncertainties. We illustrate the approach using actual accrual data and discuss practical points surrounding the accrual problem.     

Ensemble forecast modeling for the design of COVID-19 vaccine efficacy trials

To rapidly evaluate the safety and efficacy of COVID-19 vaccine candidates, prioritizing vaccine trial sites in areas with high expected disease incidence can speed endpoint accrual and shorten trial duration. Mathematical and statistical forecast models can inform the process of site selection, integrating available data sources and facilitating comparisons across locations. We recommend the use of ensemble forecast modeling – combining projections from independent modeling groups – to guide investigators identifying suitable sites for COVID-19 vaccine efficacy trials. We describe an appropriate structure for this process, including minimum requirements, suggested output, and a user-friendly tool for displaying results. Importantly, we advise that this process be repeated regularly throughout the trial, to inform decisions about enrolling new participants at existing sites with waning incidence versus adding entirely new sites. These types of data-driven models can support the implementation of flexible efficacy trials tailored to the outbreak setting.     

Assessment of equivalence on multiple endpoints.

Some clinical trials aim to demonstrate therapeutic equivalence on multiple primary endpoints. For example, therapeutic equivalence studies of agents for the treatment of osteoarthritis use several primary endpoints including investigator's global assessment of disease activity, patient's global assessment of response to therapy, and pain. In this paper, thoughts on simultaneous equivalence assessment on three endpoints are presented. As pointed out by Berger and Hsu (1996), the conventional intersection-union test can be conservative. Simulation and computation are conducted to provide an insight on the conservativeness. We also provide a method to lower the confidence level and at the same time maintain the type I error when endpoints have normal distributions and are independent. If, in a particular analysis, the goal is to demonstrate equivalence on as many endpoints as possible, a step-up procedure can be used for selecting those endpoints for which equivalence may be demonstrated. This step-up procedure at the same time controls experimentwise error rate. The techniques are illustrated by a data example.     

Patterns of patient enrollment in randomized controlled trials

We aimed to describe enrollment patterns in a large cohort of randomized controlled trials (RCTs) and evaluate whether early recruitment predicts the ability of RCTs to reach their target enrollment. We considered all 77 efficacy RCTs initiated by the AIDS Clinical Trials Group between 1986 and 1996 (28,992 patients enrolled until November 1999). Thirteen RCTs (17%) failed to reach half their target recruitment. Enrollment trajectories showed that the initial rate of accrual determined the subsequent rates of enrollment. The target sample size was attained by 7/8, 11/14, 15/35 and 4/20 of trials with very rapid, rapid, moderate and slow enrollment during the first 3 months, respectively (P < 0.001). Enrollment during the first month or two strongly correlated with subsequent accrual (P < 0.001). The patient pool, the eligibility criteria, the attractiveness of a trial and adequacy of the network of clinical sites may influence RCT enrollment. Early enrollment offers strong evidence on the feasibility of a trial and is indicative of its future pace of recruitment.     

Use of composite endpoints in clinical trials

The success of a confirmatory clinical trial designed to demonstrate the efficacy of a new treatment is highly dependent on the choice of valid primary efficacy endpoint(s). The optimal clinical and statistical situation for the design of such a trial is one that starts with the selection of a single primary efficacy endpoint that completely characterizes the disease under study, admits the most efficient clinical and statistical evaluation of treatment effect, and provides clear and broad interpretation of drug effect. For diseases with multidimensional presentations, however, the selection of such an endpoint may not be possible, and so drug effectiveness is often characterized by the use of composite or multiple efficacy endpoint(s). The use of a composite endpoint with components that are only slightly correlated but not quite dissimilar in their recognized clinical relevance could lead to a more sensitive statistical test and thus, adequately powered trials with smaller sample size. This note discusses the utility of composite endpoints in clinical trials and some of the common approaches for dealing with multiplicity arising from their use. Copyright © 2014 John Wiley & Sons, Ltd.     

Combining multiple outcome measures in a meta-analysis: an application

In meta-analysis of clinical trials published in the medical literature it is customary to restrict oneself to standard univariate fixed or random effects models. If multiple endpoints are present, each endpoint is analysed separately. A few articles have been written in the statistical literature on multivariate methods for multiple outcome measures. However, these methods were not easy to apply in practice, because self-written programs had to be used, and the examples were only two-dimensional. In this paper we consider a meta-analysis on the effect on stroke-free survival of surgery compared to conservative treatment in patients with increased risk of stroke. Three summary measures per trial are available: short-term post-operative morbidity/mortality in the surgical group; long-term event rate in the surgical group, and the event rate in the conservative group. We analyse the three outcomes jointly with a general linear MIXED model, compare the results with the standard univariate approaches and discuss the many advantages of multivariate modelling. It turns out that the general linear MIXED model is a very convenient framework for multivariate meta-analysis. All analyses could be carried out in standard general linear MIXED model software.     

The randomized placebo-phase design for clinical trials

Randomized controlled trials are the criterion standard method for evaluating the effectiveness of medical treatments. There are situations, however, where the possibility of being in the control group in a randomized controlled trial is unacceptable to potential subjects or their physicians. This lack of acceptance is a reason for poor accrual. We developed and validated a new clinical trial design for survival data that may allay concerns about not receiving an investigational product and should be more acceptable. Called the randomized placebo-phase design, this new design asks whether, on average, those subjects who begin active treatment sooner respond sooner than those who begin it later. Using Monte Carlo computer simulations, we demonstrated that the design is valid and may offer advantages over traditional randomized controlled trials in some situations. The randomized placebo-phase design may be especially useful when highly potent therapies for rare diseases are tested or when accrual may be otherwise difficult.     

Gatekeeping strategies for clinical trials that do not require all primary effects to be significant

In this paper we describe methods for addressing multiplicity issues arising in the analysis of clinical trials with multiple endpoints and/or multiple dose levels. Efficient 'gatekeeping strategies' for multiplicity problems of this kind are developed. One family of hypotheses (comprising the primary objectives) is treated as a 'gatekeeper', and the other family or families (comprising secondary and tertiary objectives) are tested only if one or more gatekeeper hypotheses have been rejected. We discuss methods for constructing gatekeeping testing procedures using weighted Bonferroni tests, weighted Simes tests, and weighted resampling-based tests, all within the closed testing framework. The new strategies are illustrated using an example from a clinical trial with co-primary endpoints, and using an example from a dose-finding study with multiple endpoints. Power comparisons with competing methods show the gatekeeping methods are more powerful when the primary objective of the trial must be met.     

Using prior information to allocate significance levels for multiple endpoints

We maximize power in a replicated clinical trial involving multiple endpoints by adjusting the individual significance levels for each hypothesis, using preliminary data to obtain the optimal adjustments. The levels are constrained to control the familywise error rate. Power is defined as the expected number of significances, where expectations are taken with respect to the posterior distributions of the non-centrality parameters under non-informative priors. Sample size requirements for the replicate study are given. Intuitive principles such as downweighting insignificant variables from a preliminary study and giving primary endpoints more emphasis are justifiable within the conceptual framework. © 1998 John Wiley & Sons, Ltd.     

Impact of health savings accounts on precautionary savings, demand for health insurance and prevention effort

The paper examines the influence of health savings accounts (HSAs) on optimal savings, insurance demand and prevention effort over the course of a lifetime. This paper adds to the literature by investigating HSAs as both a form of insurance and as saving vehicle in an expected utility framework. Assuming no regulatory constraints on the deductible, we show that individuals voluntarily choose a positive deductible and increase their savings with HSAs. If the government-imposed minimum deductible becomes too great, however, individuals may instead choose to remain in traditional insurance. We determine the effect of HSAs on prevention effort. We find that an increased tax subsidy may worsen moral hazard issues. Assuming partial risk aversion to be less than or equal to one, individuals will either invest less in the health preservation effort and more money in the HSA or vice versa. However, they will never increase both effort and savings simultaneously as was intended when HSAs were introduced.