As a part of a debate on the last day of the 2024 VEITHsymposium (19–23 November, New York, USA), Robert E Lee, a medical officer recently retired from the US Food and Drug Administration (FDA), presented his argument in favour of when the use of randomised controlled trials (RCTs) for the regulation and approval of novel vascular access devices becomes important.
To kick off his presentation, Lee began by showing previously widespread therapies that have since been abandoned after RCTs called into question their effectiveness. From radical mastectomies for breast cancer to extracranial-intracranial bypasses, he made the case that without properly conducted, randomised trials that ineffective, or potentially harmful procedures, therapies or devices have been broadly implemented without being supported by convincing clinical data.
Lee’s core argument focused on the benefits of RCTs. “Randomisation,” he stated, “helps assure that participants in both treatment groups are similar in the distribution of prognostic factors. This minimises bias in statistical comparisons of patient outcomes when looking at both effectiveness and safety.” Continuing, he added that “RCTs yield the highest level of evidence to establish causal associations in clinical research, allowing outcome differences to be interpreted as the causal effect of treatment.” This, he argued, means that there is no need for statistical sleight of hand with propensity scoring or other complex statistical methodology to understand if a new therapy is more effective.
There are, however, some situations when single-arm studies may prove adequate; Lee stated that the main instance is “when the range of treatment outcomes for a given condition fall within a narrow range, such as is the case for treating AV [arteriovenous] graft stenosis with plain PTA [percutaneous transluminal angioplasty].”
One area that Lee highlighted were the major challenges that can be encountered when using performance goals. Selection bias was the first that he addressed: “Access trial subjects are often screened for study entry with baseline ultrasound exams,” he stated, “and so likely have better anatomy than those treated in historical reports.” He also addressed historical biases that can come into play, citing the example of real-world results from AV fistula creation being driven down by the ‘Fistula First’ initiative. Finally, Lee touched on statistical limits. Without controls, he argued, comparison to a performance goal cannot demonstrate superiority or non-inferiority of a new therapy. Performance goals can only be met. Quoting from the US FDA 2013 guidance document Design Considerations for Pivotal Clinical Investigations for Medical Devices, Lee said: “It is not generally recommended that a performance goal originate with a sponsor or be developed unilaterally by FDA for a particular submission.”
Lee also argued that there are specific situations when conducting randomised trials in dialysis patients is advisable to provide robust effectiveness data. He cited a systematic review of vascular access outcomes from the Journal of Vascular Surgery that showed a wide variation in patency outcomes. “When the range of access creation outcomes is broad,” he averred, “defining a meaningful performance goal is a challenge.” A head-to-head comparison in a RCT is more apt to make things clear, according to Lee.
Bringing his presentation and argument to a close, Lee summarised his conclusions for the gathered VEITHsymposium audience. Single-arm studies with performance goals are fraught with the potential for confounding, he argued, due to historical, selection, and investigator bias. Additionally, he argued that randomisation is “essential to minimise the plethora of both the known and unknown factors affecting AV access outcomes,” and that RCTs provide the highest level of evidence, allowing “observed outcome differences to be interpreted as the causal effect of treatment”.
Furthermore, he cautioned that drawing causal inferences from non-randomised, observational studies is “inherently speculative.” Continuing, he contended that RCTs may reassure patients, physicians and payors that new device therapies can be adopted based on robust, unconfounded clinical data. Finally, he ended on a question for the audience: “Doesn’t treatment of our vulnerable dialysis patients deserve to be supported by the highest level of clinical evidence possible?”
