Making Decisions Clearer: Target Validity & The Estimands Framework Health Technology Assessments
At midnight tonight, the deadline will have passed to submit feedback on Canada's Drug Agency’s draft Methods Guide for Health Technology Assessment, which outlines a framework for appraising clinical evidence in health technology assessments (link). At CCS, we contributed comments emphasizing the need for methodological rigor and innovation to ensure assessments are both scientifically sound and practically relevant. Our submission highlights key areas where the guide can better align with the challenges of evaluating the complexity of drug products. You can read our full submission below and explore more about our perspective on the role of the estimands framework in evidence synthesis in our recent preprint.
January 28th, 2025
Comments on CDA-AMC’s Methods Guide Consultation (MG0030-000)
Dear CDA-AMC Methods Guide Committee,
Core Clinical Sciences Inc. (CCS) is submitting comments to the CDA-AMC’s Draft Methods Guide. CCS is a Canada-based healthcare analytics and modelling company that provides statistical, patient-engagement, and other methodological support to life sciences companies, academics, and non-profit organizations.
CCS is grateful for the opportunity to submit our comments to the CDA-AMC's Methods Guide. We submit the following comments to assist CDA-AMC in providing considerations for the preparation, content, and application of the Methods Guide. We have organized our comments into two themes of 1) target validity and 2) estimands.
We hope that our comments will contribute to the Agency’s efforts to increase transparency around the Agency’s process of appraising clinical evidence for their Drug Reimbursement Review Program.
Comments on target validity
This guidance document considers both the internal and external validity of clinical effectiveness evidence. For instance, despite their potentially limited external validity, the guidance highlights the value of explanatory trials, including well-conducted multi-center RCTs, for providing causal evidence in support of effectiveness. In tandem, the guidance recognizes the value of pragmatic trials for enhancing external validity and providing evidence of efficacy.
Efficacy describes whether an intervention works under ideal circumstances; data are generated from clinical trials which prioritize internal validity. Effectiveness assesses whether an intervention works in real-world settings trials; data are generated from study designs that prioritize external validity (p. 7).
Importantly, the current guidance approaches internal and external validity as separate and independent constructs. Yet, they are not standalone properties of a study. Instead, internal and external validity are interconnected concepts whose importance varies relative to the target question of interest. This is most clear when considering external validity. Without a target question of interest, how can a study be evaluated for external validity?
In addition to treating internal and external validity as separate constructs, the current guidance prioritizes internal validity over external validity. However, privileging internal validity can introduce challenges for decision-makers. Westrich and colleagues consider a hypothetical example of an intervention that has been studied in a well-controlled clinical trial conducted in an external population and in rigorous observational study conducted in the population of interest.1 If the two study results conflict, under the current guidance preference would be given to the clinical trial results even though these findings may be less relevant to the target question of interest.
Consequently, we propose that the methodological guidance consider adopting the concept of target validity which reframes internal and external validity as two components of a whole, rather than juxtaposed features of different study designs. A consequence of this framing is that biases associated with internal validity and external validity are not assessed separately. Instead target validity provides a way to assess bias as a whole, relative to the target question of interest.2 Existing causal inference methods provide reliable ways to quantitatively assess target validity. For example, transportability analysis integrates evidence from explanatory trials with real world data to generate causally valid effect estimates that reflect the population of interest.3,4 This method has been adopted by the National Institute for Health and Care Excellence (NICE) as part of their real-world evidence framework.5 Other methods of assessing target validity have also been proposed.1 These approaches generate evidence that speaks directly to the questions of interest in Health Technology Assessments (HTAs) – the comparative effectiveness and harms of a drug product in the local (i.e., Canadian) population.
Comments on estimands
We applaud CDA-AMC's recognition of the ICH E9(R1) addendum on estimands and sensitivity analyses. The estimands framework, now recognized by many global regulatory agencies including Health Canada, has been and will continue to be integrated into the design, analysis, and reporting of phase III and other trials that will be used for regulatory decison-making. However, there are myriad challenges that the estimands framework will pose for HTA and evidence synthesis.
Ambiguity in reported estimands
On Line 33 Page 10 to Line 4 on Page 11 of the technical document, it states that the target primary and alternative estimands must be clearly identified in the technical report.
Empirical work by Kahan et al. highlights important gaps in current trial protocols in that estimands are often not explicitly defined.6 When the authors tried to infer the target estimands from the trial protocols themselves, they noted that it was not always possibly to do so reliably. The CDA-AMC should provide more guidance on how estimands should be inferred in presence of poor reporting in clinical trials.
Use of PICO(T)(S) and PICOSI frameworks:
On Page 7 (Lines 27–28), CDA-AMC recommends the PICO(T)(S) framework—“population(s), intervention(s), comparator(s), outcome(s), time or time frame, study design or setting”—to determine the eligibility of studies for inclusion in clinical evidence submissions. Additionally, on Page 10 (Lines 23–26), CDA-AMC introduces the PICOSI framework—“population(s), intervention(s), comparator(s), outcome(s), summary effect measure, and intercurrent events”—to specify the target estimands of interest.
Systematic reviews serve complementary but distinct purposes to meta-analyses and other quantitative methods used for evidence synthesis. Broadly, the main goal of a systematic review is to provide a comprehensive summary of all available research on a specific topic. This differs from meta-analyses or other quantitative synthesis methods, where analysts should avoid mixing “apples and oranges” in a single analysis. For evidence synthesis, study heterogeneity is often described in terms of their differences in the PICO framework. The PICO(T)(S) framework is particularly suited for systematic reviews to ensure that all relevant studies are identified and included. However, not all studies included in a systematic review should be pooled and analyzed together quantitatively.
Using the estimands framework to specify intercurrent events and analytical strategies, as noted by CDA-AMC, is important to ensure that the study is designed and analyzed in a way that allowed quantitative assessment of the target estimand of interest. This is no different for meta-analyses. Alignment of the estimands framework is useful to ensure that we avoid combining apples and oranges. For this, the PICOSI framework is potentially valuable in deciding which subset of studies should be included in meta-analysis out of the totality of the evidence base captured by a systematic review grounded in the PICO(T)(S) framework.
More guidance is needed from CDA-AMC. Although specification of intercurrent events at the trial-level is an important first step, it alone does not avoid comparison of “apples and oranges” as different analytical strategies for the same intercurrent event can lead to different estimates of different target estimands. To ensure valid estimates are estimated in a meta-analysis, there is a need to go beyond specifying intercurrent events. Actual analytical strategies for intercurrent events should be accounted for in quantitative evidence synthesis. At present, the Methods Guidance does not address this crucial issue that can introduce bias.
For example, treatment switching (cross-over) in oncology, which is common in oncology trials, is an important intercurrent event for HTA decision making. In recognition of the importance of treatment switching for HTA decision making, NICE released two guidance documents (TSD 16 and TSD 24) to address treatment switching in the analysis of individual clinical trials.7,8 The guidance highlights that different analytical approaches for the same intercurrent event of treatment switching ultimately target different estimands and will, thus, produce different estimates.
If the goal of meta-analysis is to estimate the treatment effects that reflect the potential value of integrating a new novel therapy into the health system, given that the patients switched onto a new treatment that reflected the standard-of-care, it is likely of interest that we limit trials reporting on treatment policy estimands. If the patients in the trials switched onto a treatment option that is not locally available, trials reporting on treatment policy estimands are likely not of an interest for local HTA decision making.
For other estimands, such as hypothetical estimands, there are different analytical strategies for treatment switching and these will produce different estimates. This is not surprising as each analytical approach has different underlying statistical assumptions. For intercurrent events that occur as frequently as treatment switching, pooling trials that employ different analytical strategies should be avoided or be interpreted with major caution. Speaking to this, a previous simulation study showed that pooling estimates targeting different estimands resulted in pooled estimators that did not reflect the target estimand of interest.9
Summary
In sum, we propose the following:
Adopting the target validity framework and aligned methods, including transportability analysis, a NICE endorsed approach for real-world evidence
Providing further guidance on how to infer estimands from clinical trials given historically poor reporting
Further consideration of how to integrate the estimands framework into meta-analysis given that explanatory trials vary in their strategies for intercurrent events and this can in turn introduce bias relative to the meta-analytic estimand of interest.
Overall, we are heartened that the CDA-AMC has embraced the estimands framework as it enhances clarity about what is being measured and how. As methodologists, we appreciate the opportunity to provide feedback, and hope that it will help further strengthen the final methods guidance.
Sincerely,
Jay Park, PhD
Scientific Lead and Founder
Core Clinical Sciences Inc.
Rebecca Metcalfe, PhD
Principal Scientist, Advanced Epidemiology and Patient Centered Research
Core Clinical Sciences Inc.
References
1. Westreich D, Edwards JK, Lesko CR, Cole SR, Stuart EA. Target validity and the hierarchy of study designs. American journal of epidemiology. 2019;188(2):438-443.
2. Lesko CR, Ackerman B, Webster-Clark M, Edwards JK. Target validity: bringing treatment of external validity in line with internal validity. Current epidemiology reports. 2020;7:117-124.
3. Dahabreh IJ, Matthews A, Steingrimsson JA, Scharfstein DO, Stuart EA. Using trial and observational data to assess effectiveness: Trial emulation, transportability, benchmarking, and joint analysis. Epidemiologic Reviews. 2023:mxac011.
4. Inoue K, Hsu W. Transportability Analysis—A Tool for Extending Trial Results to a Representative Target Population. JAMA network open. 2024;7(1):e2346302-e2346302.
5. National Institute for Health and Care Excellence. NICE real-world evidence framework (ECD9). https://www.nice.org.uk/corporate/ecd9/resources/nice-realworld-evidence-framework-pdf-1124020816837
6. Kahan BC, Morris TP, White IR, Carpenter J, Cro S. Estimands in published protocols of randomised trials: urgent improvement needed. Trials. 2021;22:1-10.
7. Gorrod H, Latimer N, Abrams KR. Adjusting survival time estimates in the presence of treatment switching: An update to TSD 16. National Institute for Health and Care Excellence (NICE); 2024.
8. Latimer N, Abrams KR. Adjusting survival time estimates in the presence of treatment switching. National Institute for Health and Care Excellence (NICE); 2014.
9. Vuong Q, Metcalfe RK, Remiro-Azócar A, Gorst-Rasmussen A, Keene O, Park JJ. Estimands and Their Implications for Evidence Synthesis for Oncology: A Simulation Study of Treatment Switching in Meta-Analysis. arXiv preprint arXiv:241114323. 2024;