DÄ internationalArchive51-52/2021Estimands—A Basic Element for Clinical Trials

Review article

Estimands—A Basic Element for Clinical Trials

Part 29 of a Series on Evaluation of Scientific Publications

Dtsch Arztebl Int 2021; 118: 883-8. DOI: 10.3238/arztebl.m2021.0373

Pohl, M; Baumann, L; Behnisch, R; Kirchner, M; Krisam, J; Sander, A

Background: Clinical trials are of central importance for the evaluation and comparison of treatments. The transparency and intelligibility of the treatment effect under investigation is an essential matter for physicians, patients, and health-care authorities. The estimand framework has been introduced because many trials are deficient in this respect.

Methods: Introduction, definition, and application of the estimand framework on the basis of an example and a selective review of the literature.

Results: The estimand framework provides a systematic approach to the definition of the treatment effect under investigation in a clinical trial. An estimand consists of five attributes: treatment, population, variable, population-level summary, and handling of intercurrent events. Each of these attributes is defined in an interdisciplinary discussion during the trial planning phase, based on the clinical question being asked. Special attention is given to the handling of intercurrent events (ICEs): these are events—e.g., discontinuation or modification of treatment or the use of emergency medication—that can occur once the treatment has begun and might affect the possibility of observing the endpoints or their interpretability. There are various strategies for the handling of ICEs; these can, for example, also reflect the existing intention-to-treat (ITT) principle. Per-protocol analyses, in contrast, are prone to bias and cannot be represented in a sensible manner by an estimand, although they may be performed as a supplementary analysis. The discussion of potential intercurrent events and how they should appropriately be handled in view of the aim of the trial must already take place in the planning phase.

Conclusion: Use of the estimand framework should make it easier for both physicians and patients to understand what trials reveal about the efficacy of treatment, and to compare the results of different trials.

LNSLNS

Clinical trials represent one of the cornerstones of evidence-based medicine and are intended to investigate a medical question of interest, the answer to which is formulated as a trial objective. For a trial in which a new drug to treat type-2 diabetes is under investigation, one could formulate the following trial objective: “The objective of the trial is to investigate whether drug x reduces the mean change in HbA1c from treatment initiation to 24 weeks compared to placebo.” The formulation of the trial objective makes it clear that the treatment effect of the experimental treatment on the change in HbA1c is to be determined. However, many aspects remain implicit, such as dealing with patients that do not take the drug as intended. In reality, deviations from the intended course of treatment will be observed. These may include events such as treatment discontinuation/modification or use of emergency medication, thereby affecting the observed values. If one is interested in the treatment effect under ideal conditions without events of this kind, then the assessment based on such observed patient data is no longer straightforward. Specifying how such events should be dealt with in the actual trial is important. In the trial used here as an example, one needs to consider, among other things, how the changes in HbA1c in patients that received emergency medication during the course of the trial will be taken into account. One possible option is to consider all observations irrespective of the use of emergency medication. However, if one wishes to investigate the isolated effect of the trial treatment without the emergency medication, observed values following the use of the emergency medication would not be used as such. Although both options are feasible, each addresses a different question of interest.

The estimand framework offers a systematic approach to avoiding the abovementioned areas of uncertainty and resulting problems by means of clear and pre-specified definitions. The following article will define the estimand framework and will illustrate its application using the example trial. It charts the shift from current approaches to the estimand framework and outlines the latter’s practical application as well as its scientific relevance.

Glossary
Box
Glossary

General definition of the estimand framework

The estimand is a systematic description of the treatment effect to be quantified in order to answer the trial’s research objective. The estimand consists of the following five attributes: Treatment, Population, Variable, Population-Level Summary, and Handling of Intercurrent Events (ICEs). These terms are deliberately left in English since this is the language of the primary literature on the estimand framework and enables uniform and unambiguous communication across language borders. For the definition of the estimand, the individual attributes are specified based on the clinical question being asked. ‘Treatment’ describes the treatments to be investigated, while ‘Population’ defines which group of patients is addressed. The ‘Variable’ describes the measure based on which the treatment will be assessed (endpoint), and the ‘Population-Level Summary’ specifies how the patient-specific endpoints for the group comparison are summarized, for example, as the difference between the two group means. Examples of this can be found in the Table, where the attributes for the example trial are given. It is important to note here that there is an interplay between the attributes, hence they cannot be considered separately.

Example formulation of the attributes of the described estimand strategies for the example trial on diabetes in relation to the intercurrent event (ICE) ‘use of emergency medication.’
Table
Example formulation of the attributes of the described estimand strategies for the example trial on diabetes in relation to the intercurrent event (ICE) ‘use of emergency medication.’

ICEs are events that can occur once the treatment has started, affect the presence and/or interpretability of the observed variable, and are not bound to the trial setting (for example, treatment discontinuation due to toxicities, use of emergency medication). ICEs are not the same as missing values. Missing values are values that theoretically exist and are useful but which have not been observed (for example, due to loss to follow up, lost laboratory sample). An ICE, on the other hand, can result in a value being observed but not usable in any helpful way in the analysis, or in a value not being observed at all, as for example in the case of the ICE “death.” The Figure shows a schematic representation of possible patient-specific courses over the trial, including ICEs. There is no comprehensive list of relevant ICEs that is valid for all trials, since the specific trial situation (indication, treatments, follow-up period, among others) determines which ICEs may occur. Different ICEs may be seen in trials of long-term medication (for example, discontinuation of the medication) compared to surgical trials (for example, re-operation).

Schematic representation of ICEs over the patient-specific course in an example trial.
Figure
Schematic representation of ICEs over the patient-specific course in an example trial.

A central aspect of the estimand framework is the targeted focus on possible ICEs and how they are dealt with. The framework offers various strategies to do this. The four most important of these are described and illustrated below using the ICE ‘Use of emergency medication’ in the example trial. Further strategies are proposed in the literature (1, 2, 3, 4). It is important to note that the selected strategy impacts the other attributes, as shown and marked in bold in the Table.

The ‘treatment policy’ strategy ignores the ICE. This means that the observed values of the endpoint (variable) are included in the analysis, irrespective of whether or not the ICE previously occurred. In the example trial, this means that the change in HbA1c at 24 weeks is used, even if the patient previously received emergency medication.

The ‘hypothetical’ strategy evaluates the treatment effect in a hypothetical scenario in which the observed ICE would not have occurred. This means that values observed after the occurrence of an ICE are not included as such in the analysis. Instead, suitable statistical methods are used in order to estimate the treatment effect that would have been seen had the ICE not occurred. Thus, in the example trial, the actual, observed HbA1c value at 24 weeks would not be included in the evaluation in the case of patients that received emergency medication.

The ‘composite variable’ strategy views the ICE as a source of information and integrates it in the definition of the endpoint. This results in a composite endpoint. If one has, as in the example trial, a continuous endpoint (change in HbA1c) and an ICE, one can, for instance, convert the continuous endpoint into a binary endpoint. To this end, a threshold value that reflects a clinically relevant change is set. The combined endpoint then describes for each patient whether there has been success or failure, with success defined as exceeding the threshold value while at the same time not having used any emergency medication.

In the ‘while on treatment’ strategy, the focus shifts in terms of the temporal classification of the treatment effect—this is now only considered up until immediately before the time of onset of the ICE. In the example trial, this means that the change in HbA1c is investigated from the time of treatment initiation to 24 weeks or the time at which emergency medication was used.

The ‘treatment policy’ and ‘hypothetical’ strategies were already touched upon in the Introduction in the context of the example trial. The example relates to a trial comparing dapagliflozin to placebo (5) in which the sponsor chose a hypothetical strategy to take emergency medication into account in the analysis, whereas in talks on approval, the regulatory authorities requested a ‘treatment policy’ strategy (6). The trial in question was conducted before the estimand framework was introduced, and the application of the strategies was implicit. However, this shows that such differing expectations can be avoided with the estimands framework by specifying early on the strategies to be used for handling of ICEs.

Much of the content of the described attributes has already been specified in clinical trials, for example, which treatment will be investigated in which patient population, which endpoint is to be recorded, and how this is to be summarized. What is new is that, by using the attributes, this is now presented more accurately and more systematically. This should also be reflected in the relevant trial documents (including protocol and publication of results). The handling of events following treatment initiation, which now come under the uniform name intercurrent events, was also mostly thought of in terms of an intention-to-treat (ITT) or a per-protocol (PP) analysis. In the latter, only those patients that completed the trial according to protocol were taken into consideration. However, if the protocol violations that could also result from an ICE are related to treatment, this can lead to unequal groups and thus to biased results (7, 8). In the example trial, it could be that the trial medication causes more side effects in older patients, thereby leading to treatment discontinuation. If this is deemed to be a protocol violation, these patients would be excluded from a PP analysis, and unequal groups would be compared with each other. PP analyses cannot be represented in a sensible manner by an estimand and will play a minor role in future as supplementary analyses (1). In an ITT analysis, all randomized patients are considered in the group to which they were randomized, irrespective of what happens thereafter (randomization means random assignment to a group). Although this approach often most closely reflects clinical routine, an ITT analysis does not necessarily evaluate in all situations the effect that is of interest to the regulators, patients, and/or clinicians (9, 10, 11). In the estimand framework, the ‘treatment policy’ strategy equates to the ITT principle since, here too, what happens following randomization is ignored. Having said that, ‘hypothetical,’ ‘while on treatment,’ and ‘composite variable’ strategies also include all randomized patients in the analysis (12). Here, however, the handling of each individual ICE is explicitly specified, whereas in an ITT analysis, there is a tendency to give no explicit thought to ICEs, although the act of ignoring them results in answering a different research question. Therefore, the novel content-related features of the estimand framework come into play particularly as a result of the precisely defined handling of ICEs, which is then also reflected in the other attributes.

Practical application and relevance

In practical application, the first step involves a medical question, on the basis of which the estimand is defined. It is important here that the clinician and statistician jointly address possible ICEs and their handling, whereby the expertise and clinical experience of physicians is elementary. In a joint exchange, these experts should discuss possible ICEs and specify how they are to be handled, in order that the design, implementation, and analysis strategies of the trial can be set out in alignment with these specifications (13, 14). In this way, the treatment effect that is of interest can be evaluated at the end of the trial. The treatment effect of primary interest can differ depending on the individual perspective. A physician and their patient would probably like to know which treatment effect they can expect if the treatment is carried out as intended. For regulatory authorities, the relevant effect may be the one that can be expected in clinical routine. This generally equates to an average effect for the patient population that incorporates influences from events such as premature treatment discontinuation (9).

A variety of ICEs can occur in a trial, and a strategy for handling each of these ICEs needs to be defined. Also, different strategies can be deployed for the various ICEs. For example, emergency medication use could be handled with a ‘composite’ strategy and treatment discontinuation with a ‘treatment policy’ strategy. Both approaches influence the attributes and, in an interplay, yield the estimand that reflects the trial objective. A new estimand emerges as soon as an attribute and/or the strategy for an ICE is altered. In a trial, it is important to define in the planning phase the primary estimand that belongs to the investigation of the primary trial objective. Additional estimands that differ from the primary estimand, and thus address different questions, can also be defined. It should be pointed out that the primary estimand is not the same as the primary endpoint. The endpoint is part of the estimand and is represented by the ‘Variable’ attribute. The endpoint alone is not sufficiently informative in relation to the question at hand. This also means that separate estimands are defined for secondary objectives. These include the secondary endpoint as Variable and, together with the other attributes, the estimand produces a more precise specification compared to using the endpoint alone.

Systematic reviews will also benefit from the more precise definition of the trial objective and the transparent description of the quantified treatment effect. These reviews aim to identify and summarize trials that have investigated the same question of interest and often include meta-analyses in which individual trial results are combined into one total estimator across all trials. Trials that use the estimand framework simplify the selection of suitable and, above all, comparable trials for systematic reviews.

The estimand framework came into effect in 2020 as an addendum to the ICH E9 guideline “Statistical principles for clinical trials” (1) and has received great attention among statisticians since the publication of the draft version in 2017. For clinicians, the topic is of considerable relevance despite the fact that it can be found in a statistically oriented guideline, since it relates to the planning and interpretation of clinical trials in general. By embedding the estimand framework in the ICH guidelines, it will become a basic element of clinical trials in the near future. In the short term, the framework will come into play mainly in the planning of new trials and, in the mid-term, become manifest in the scientific publications of trial results. In their author guidelines, scientific journals often refer to guidelines such as the CONSORT statement, which already addresses the estimand framework in its “Adaptive Designs Extension Statement” (15). One can assume that this will also be adopted in other guidelines and statements (for example, SPIRIT), thereby representing a necessary prerequisite for the publication of trial results in line with good scientific practice.

The ICH offers extensive online training material on the topic of estimands (16). The material also includes suggestions as to where and/or how estimands can be described in the trial protocol.

In addition to numerous publications in statistics journals, the first articles addressing the topic of estimands in general have appeared in clinical journals (for example, [17–21]). The estimand framework has also already found its way into indication-specific guidelines (for example, [22, 23]). Furthermore, there are publications on trial protocols and the corresponding trial results that have already implemented the estimand framework (24, 25).

Conclusion and outlook

The estimand framework offers a systematic approach to defining the treatment effect investigated in a trial in such a way that the initial clinical question being asked can be answered. This puts greater focus on the questions: “What could/will happen during the trial?” and “How will this impact the treatment effect?” Addressing these questions should already form a basic element of the planning phase. With the estimand framework, a clearer and, above all, uniform framework has now been created to this end. Before the estimand framework can become a basic element of clinical trials, efforts are needed to raise awareness of the framework. This could be achieved, for example, through targeted further training programs for trial teams.

The implementation of the framework in the planning phase can initially lead to somewhat greater time requirements. However, this should not be regarded as a disadvantage, since the greater clarity and transparency represent a sign of quality, as well as preventing misunderstandings and problems in the further course and in the final analysis. This transparency increases our understanding of the efficacy of treatments. The framework has the potential to improve the reproducibility of trial results—an important aspect in view of the prevailing reproducibility crisis in medical research (26, 27, 28). Furthermore, the framework is intended to simplify the comparison of trial results in the future and support practicing physicians as well as affected patients in an informed decision-making process according to the principles of evidence-based medicine.

Conflict of interest statement
The authors declare that no conflict of interests exists.

Manuscript received on 18 June 2021, revised version accepted on 3 November 2021.

Translated from the original German by Christine Rye.

Corresponding author
Moritz Pohl, M.Sc.
Universitätsklinikum Heidelberg
Institut für Medizinische Biometrie
Im Neuenheimer Feld 130.3

69120 Heidelberg, Germany
pohl@imbi.uni-heidelberg.de

Cite this as:
Pohl M, Baumann L, Behnisch R, Kirchner M, Krisam J, Sander A: Estimands—a basic element for clinical trials. Part 29 of a series on evaluation of scientific publications. Dtsch Arztebl Int 2021; 118: 883–8. DOI: 10.3238/arztebl.m2021.0373

1.
ICH E9 (R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials – Step 5, EMA/CHMP/ICH/436221/2017, 2020. www.ema.europa.eu/en/ich-e9-statistical-principles-clinical-trials (last accessed on 25 October 2021).
2.
Darken P, Nyberg J, Ballal S, Wright D: The attributable estimand: a new approach to account for intercurrent events. Pharm Stat 2020; 19: 626–35 CrossRef MEDLINE
3.
Michiels H, Sotto C, Vandebosch A, Vansteelandt S: A novel estimand to adjust for rescue treatment in randomized clinical trials. Stat Med 2021; 40: 2257–71 CrossRef MEDLINE
4.
Bornkamp B, Rufibach K, Lin J, et al.: Principal stratum strategy: Potential role in drug development. Pharm Stat 2021; 20: 737–51 CrossRef MEDLINE
5.
Ferrannini E, Ramos SJ, Salsali A, Tang W, List JF: Dapagliflozin monotherapy in type 2 diabetic patients with inadequate glycemic control by diet and exercise: a randomized, double-blind, placebo-controlled, phase 3 trial. Diabetes care 2010; 33: 2217–24 CrossRef MEDLINE PubMed Central
6.
Holzhauer B, Akacha M, Bermann G: Choice of estimand and analysis methods in diabetes trials with rescue medication. Pharm Stat 2015; 14: 433–47 CrossRef MEDLINE
7.
Ranganathan P, Pramesh CS, Aggarwal R: Common pitfalls in statistical analysis: intention-to-treat versus per-protocol analysis. Perspect Clin Res 2016; 7: 144–6 CrossRef MEDLINE PubMed Central
8.
Kabisch M, Ruckes C, Seibert-Grafe M, Blettner M: Randomized controlled trials: part 17 of a series on evaluation of scientific publications. Dtsch Arztebl Int 2011; 108: 663–8 VOLLTEXT
9.
Keene ON, Ruberg S, Schacht A, et al.: What matters most? Different stakeholder perspectives on estimands for an invented case study in COPD. Pharm Stat 2020; 19: 370–87 CrossRef MEDLINE
10.
Keene ON, Wright D, Phillips A, Wright M: Why ITT analysis is not always the answer for estimating treatment effects in clinical trials. Contemp Clin Trials 2021; 108: 106494 CrossRef MEDLINE PubMed Central
11.
Akacha M, Bretz F, Ruberg S: Estimands in clinical trials—broadening the perspective. Stat Med 2017; 36: 5–19 CrossRef MEDLINE
12.
Leuchs AK, Brandt A, Zinserling J, Benda N: Disentangling estimands and the intention-to-treat principle. Pharm Stat 2017; 16: 12–9 CrossRef MEDLINE
13.
Mallinckrodt C, Molenberghs G, Lipkovich I, Ratitch B: Estimands, estimators and sensitivity analysis in clinical trials. 1st ed. Boca Raton: Chapman and Hall/CRC; 2020 CrossRef
14.
Fang Y, Jin M: Sample size calculation when planning clinical trials with intercurrent events. Ther Innov Regul Sci 2021; 55: 779–85 CrossRef MEDLINE
15.
Dimairo M, Pallmann P, Wason J, et al.: The adaptive designs CONSORT Extension (ACE) statement: a checklist with explanation and elaboration guideline for reporting randomised trials that use an adaptive design. BMJ 2020; 369: m115 CrossRef MEDLINE PubMed Central
16.
ICH E9(R1) Expert Working Group: Draft (Step 2) guideline ICH E9(R1) – estimands and sensitivity analysis in clinical trials – training module 1–3, www.database.ich.org/sites/default/files/E9-R1_EWG_Step2_TrainingMaterial.pdf (last accessed on 25 October 2021).
17.
Fiero MH, Pe M, Weinstock C, et al.: Demystifying the estimand framework: a case study using patient-reported outcomes in oncology. Lancet Oncol 2020; 21: e488–e94 CrossRef
18.
Wharton S, Astrup A, Endahl L, et al.: Estimating and reporting treatment effects in clinical trials for weight management: using estimands to interpret effects of intercurrent events and missing data. Int J Obes 2021; 45: 923–33 CrossRef MEDLINE PubMed Central
19.
Donohue MC, Model F, Delmar P, et al.: Initiation of symptomatic medication in Alzheimer‘s disease clinical trials: hypothetical versus treatment policy approach. Alzheimers Dement 2020; 16: 797–803 CrossRef MEDLINE PubMed Central
20.
Kahan BC, Morris TP, White IR, Carpenter J, Cro S: Estimands in published protocols of randomised trials: urgent improvement needed. Trials 2021; 22: 686 CrossRef MEDLINE PubMed Central
21.
Aroda VR, Saugstrup T, Buse JB, Donsmark M, Zacho J, Davies MJ: Incorporating and interpreting regulatory guidance on estimands in diabetes clinical trials: the PIONEER 1 randomized clinical trial as an example. Diabetes Obes Metab 2019; 21: 2203–10 CrossRef MEDLINE PubMed Central
22.
European Medicines Agency: Guideline on the clinical investigation of medicines for the treatment of Alzheimer’s disease, CPMP/EWP/553/95 Rev .2, 2020.
23.
European Medicines Agency: Guideline on clinical investigation of medicinal products in the treatment or prevention of diabetes mellitus – Draft, CPMP/EWP/1080/00 Rev. 2, 2018.
24.
Wilding JPH, Batterham RL, Calanna S, et al.: Once-weekly semaglutide in adults with overweight or obesity. N Engl J Med 2021; 384: 989–1002 CrossRef MEDLINE
25.
Aroda VR, Rosenstock J, Terauchi Y, et al.: PIONEER 1: Randomized clinical trial of the efficacy and safety of oral semaglutide monotherapy in comparison with placebo in patients with type 2 diabetes. Diabetes care 2019; 42: 1724–32 CrossRef MEDLINE
26.
Ioannidis JPA: Why most published research findings are false. PLoS Med 2005; 2: e124 CrossRef MEDLINE PubMed Central
27.
Niven DJ, McCormick TJ, Straus SE, et al.: Reproducibility of clinical research in critical care: a scoping review. BMC Med 2018; 16: 26 CrossRef MEDLINE PubMed Central
28.
Stupple A, Singerman D, Celi LA: The reproducibility crisis in the age of digital medicine. npj Digital Medicine 2019; 2: 2 CrossRef MEDLINE PubMed Central
Institute of Medical Biometry, University Hospital Heidelberg, Heidelberg, Germany
Moritz Pohl, M.Sc., Lukas Baumann, Mag., Rouven Behnisch, M.Sc., Dr. phil. Marietta Kirchner, Dr. sc. hum. Johannes Krisam, Dr. sc. hum. Anja Sander
Glossary
Box
Glossary
Schematic representation of ICEs over the patient-specific course in an example trial.
Figure
Schematic representation of ICEs over the patient-specific course in an example trial.
Example formulation of the attributes of the described estimand strategies for the example trial on diabetes in relation to the intercurrent event (ICE) ‘use of emergency medication.’
Table
Example formulation of the attributes of the described estimand strategies for the example trial on diabetes in relation to the intercurrent event (ICE) ‘use of emergency medication.’
1.ICH E9 (R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials – Step 5, EMA/CHMP/ICH/436221/2017, 2020. www.ema.europa.eu/en/ich-e9-statistical-principles-clinical-trials (last accessed on 25 October 2021).
2.Darken P, Nyberg J, Ballal S, Wright D: The attributable estimand: a new approach to account for intercurrent events. Pharm Stat 2020; 19: 626–35 CrossRef MEDLINE
3.Michiels H, Sotto C, Vandebosch A, Vansteelandt S: A novel estimand to adjust for rescue treatment in randomized clinical trials. Stat Med 2021; 40: 2257–71 CrossRef MEDLINE
4.Bornkamp B, Rufibach K, Lin J, et al.: Principal stratum strategy: Potential role in drug development. Pharm Stat 2021; 20: 737–51 CrossRef MEDLINE
5.Ferrannini E, Ramos SJ, Salsali A, Tang W, List JF: Dapagliflozin monotherapy in type 2 diabetic patients with inadequate glycemic control by diet and exercise: a randomized, double-blind, placebo-controlled, phase 3 trial. Diabetes care 2010; 33: 2217–24 CrossRef MEDLINE PubMed Central
6.Holzhauer B, Akacha M, Bermann G: Choice of estimand and analysis methods in diabetes trials with rescue medication. Pharm Stat 2015; 14: 433–47 CrossRef MEDLINE
7.Ranganathan P, Pramesh CS, Aggarwal R: Common pitfalls in statistical analysis: intention-to-treat versus per-protocol analysis. Perspect Clin Res 2016; 7: 144–6 CrossRef MEDLINE PubMed Central
8.Kabisch M, Ruckes C, Seibert-Grafe M, Blettner M: Randomized controlled trials: part 17 of a series on evaluation of scientific publications. Dtsch Arztebl Int 2011; 108: 663–8 VOLLTEXT
9.Keene ON, Ruberg S, Schacht A, et al.: What matters most? Different stakeholder perspectives on estimands for an invented case study in COPD. Pharm Stat 2020; 19: 370–87 CrossRef MEDLINE
10.Keene ON, Wright D, Phillips A, Wright M: Why ITT analysis is not always the answer for estimating treatment effects in clinical trials. Contemp Clin Trials 2021; 108: 106494 CrossRef MEDLINE PubMed Central
11.Akacha M, Bretz F, Ruberg S: Estimands in clinical trials—broadening the perspective. Stat Med 2017; 36: 5–19 CrossRef MEDLINE
12.Leuchs AK, Brandt A, Zinserling J, Benda N: Disentangling estimands and the intention-to-treat principle. Pharm Stat 2017; 16: 12–9 CrossRef MEDLINE
13.Mallinckrodt C, Molenberghs G, Lipkovich I, Ratitch B: Estimands, estimators and sensitivity analysis in clinical trials. 1st ed. Boca Raton: Chapman and Hall/CRC; 2020 CrossRef
14.Fang Y, Jin M: Sample size calculation when planning clinical trials with intercurrent events. Ther Innov Regul Sci 2021; 55: 779–85 CrossRef MEDLINE
15.Dimairo M, Pallmann P, Wason J, et al.: The adaptive designs CONSORT Extension (ACE) statement: a checklist with explanation and elaboration guideline for reporting randomised trials that use an adaptive design. BMJ 2020; 369: m115 CrossRef MEDLINE PubMed Central
16.ICH E9(R1) Expert Working Group: Draft (Step 2) guideline ICH E9(R1) – estimands and sensitivity analysis in clinical trials – training module 1–3, www.database.ich.org/sites/default/files/E9-R1_EWG_Step2_TrainingMaterial.pdf (last accessed on 25 October 2021).
17.Fiero MH, Pe M, Weinstock C, et al.: Demystifying the estimand framework: a case study using patient-reported outcomes in oncology. Lancet Oncol 2020; 21: e488–e94 CrossRef
18.Wharton S, Astrup A, Endahl L, et al.: Estimating and reporting treatment effects in clinical trials for weight management: using estimands to interpret effects of intercurrent events and missing data. Int J Obes 2021; 45: 923–33 CrossRef MEDLINE PubMed Central
19.Donohue MC, Model F, Delmar P, et al.: Initiation of symptomatic medication in Alzheimer‘s disease clinical trials: hypothetical versus treatment policy approach. Alzheimers Dement 2020; 16: 797–803 CrossRef MEDLINE PubMed Central
20.Kahan BC, Morris TP, White IR, Carpenter J, Cro S: Estimands in published protocols of randomised trials: urgent improvement needed. Trials 2021; 22: 686 CrossRef MEDLINE PubMed Central
21.Aroda VR, Saugstrup T, Buse JB, Donsmark M, Zacho J, Davies MJ: Incorporating and interpreting regulatory guidance on estimands in diabetes clinical trials: the PIONEER 1 randomized clinical trial as an example. Diabetes Obes Metab 2019; 21: 2203–10 CrossRef MEDLINE PubMed Central
22.European Medicines Agency: Guideline on the clinical investigation of medicines for the treatment of Alzheimer’s disease, CPMP/EWP/553/95 Rev .2, 2020.
23.European Medicines Agency: Guideline on clinical investigation of medicinal products in the treatment or prevention of diabetes mellitus – Draft, CPMP/EWP/1080/00 Rev. 2, 2018.
24.Wilding JPH, Batterham RL, Calanna S, et al.: Once-weekly semaglutide in adults with overweight or obesity. N Engl J Med 2021; 384: 989–1002 CrossRef MEDLINE
25.Aroda VR, Rosenstock J, Terauchi Y, et al.: PIONEER 1: Randomized clinical trial of the efficacy and safety of oral semaglutide monotherapy in comparison with placebo in patients with type 2 diabetes. Diabetes care 2019; 42: 1724–32 CrossRef MEDLINE
26.Ioannidis JPA: Why most published research findings are false. PLoS Med 2005; 2: e124 CrossRef MEDLINE PubMed Central
27.Niven DJ, McCormick TJ, Straus SE, et al.: Reproducibility of clinical research in critical care: a scoping review. BMC Med 2018; 16: 26 CrossRef MEDLINE PubMed Central
28.Stupple A, Singerman D, Celi LA: The reproducibility crisis in the age of digital medicine. npj Digital Medicine 2019; 2: 2 CrossRef MEDLINE PubMed Central