This article in the New England Journal of Medicine (NEJM) reports on results from a clinical trial that evaluated the use of vedolizumab for Crohn’s disease. Media reported worldwide that this new drug is effective and without harmful drug side-effects. Takeda Pharmaceutical Company says they submitted a Biologics License Application (BLA) for vedolizumab to the U.S Food and Drug Administration (FDA). This license allows for selling and marketing vedolizumab. Apart from all the buzz, what’s in it for patients with Crohn? Not much , it seems.
Vedolizumab’s effect or an unblinding effect?
I reviewed the NEJM article and found some problems that may help interpret results. The most important issue is that trial participants may have known whether they got placebo or vedolizumab. Since the trial is a double blinded randomized trial, all participants, study personnel and outcome assessors should be blinded. This eliminates subjective, unconscious biases.
This is especially important since the study used the Crohn’s Disease Activity Index (CDAI) questionnaire. This survey gives extra weight to subjective complaints. To get valid results from it, any ‘breaking’ of the double blind must be avoided. Very small amounts of study unblinding can lead to misleading data. In fact, Hróbjartsson found that only 3% of patients need to become aware of their treatment (unblinded) to have an 36% exaggeration of treatment effects.
A complex study design was used with separate induction and maintenance phases. It followed that most of the participants gained experience with vedolizumab and became familiar with its subtle bodily cues during the subsequent maintenance phase. It is clear that unblinded participants may have more expectations for recovery when they find out to be on the new investigational drug.
Imagine what it would be to find out you are on placebo.
Your intestines go into a disillusioned spasm.
The study wanted to know how many patients have a clinical remission at week 52. This associated graph shows something remarkable:
Using this data, a doctor can say:
Dear patient, for 38 weeks it will not matter whether you take vedolizumab or placebo. In fact, placebo would save you a lot of money! If you plan to have symptoms after 38 weeks, it may then be better to switch to vedolizumab. Expect to find benefit from the drug at week 52.
Of course, this is complete nonsense. Upon inspecting the graph, the drugs’ performance remains more or less stable throughout the 52 weeks. Placebo remains stable too, but starts to degrade late in the study. This creates an apparently effective drug, caused by an knocked down placebo effect. Now is this a drug effect? Or, is this an unblinding effect due to bad study methodology?
Few Crohn patients will benefit, many will be harmed.
There is a statistical significant effect, despite a small difference (17%) between patients being ok on placebo versus those recovered on vedolizumab, every 8 weeks. Significance was reached because a large sample size was used. But in clinical practice, how relevant is this 17% difference? It would be helpful to report confidence intervals (CI), it is the standard method for presenting results of major findings. Then we would know how confident we can be about this 17% difference. In fact, the study’s protocol (can be found as additional download on NEJM article page) planned to report CI. Unfortunately, this vital statistic was not reported. A concern, indeed.
The good news is that we can calculate the number of patients needed to treat (NNT). This tells us how many patients should take the drug for one patient to benefit. In this case 6 with a 95% CI [4 to 14]. We are certain that the real-life NNT is somewhere between 4 and 14.
Are we done? No, things get worse. Assume that some patients became unblinded during the study, since they rate their own outcome this would exaggerate results. Assume 27% exaggeration (Risk Ratios estimate, taken from Hróbjartsson’s paper), after some calculating the true NNT is 9 (I am not 100% sure (or even 95% 🙂 ) I calculated this NTT correctly, so if anyone can check this?). This is indeed a very sobering exercise. Most probably this is what patients can expect form vedolizumab:
If 9 patients are treated with vedolizumab every 8 weeks, then after 52 weeks, 1 patient will have clinical remission of Crohn. During 52 weeks, 8 patients have nothing but side-effects.
Are we done now? No, things do get worse. The paper reports serious adverse effects for 15.3% of patients taking placebo and 24.4% taking vedolizumab. Reframed in understandable words:
If 11 patients get vedolizumab, we expect that 1 has serious adverse effects. Yes, that is 1 out of 11!
The first induction phase of the study lasted 6 weeks (see study design). Even with a large sample size, vedolizumab just reached statistical significance for clinical remission, but did not do so for CDAI-100 response. Again, the differences (8% and 6% respectively) with placebo are very small. Lacking a reported confidence interval, we are left in the dark as to the true estimated, clinically relevant drug effect. Yes, a concern. I calculated the NNT for you. NNT is 13 with a 95% CI [7 to 85]. We are certain that the real-life NNT is somewhere between 7 and 85.
Are we done? No, as stated before, most likely some patients became unblinded. The true NNT is 15. Pretty sobering figures, I would say.
If 15 patients are treated with vedolizumab, after 6 weeks 1 patient will have clinical remission of Crohn. 14 patients get nothing but side-effects.
Seriously, it is fair to say that if you have Crohn, you better seek alternatives for this ‘exiting new drug’. It is marketing talk.
Letter to the editor of NEJM
Titled ‘Vedolizumab for Crohn: impaired double blind?’, I wrote a letter to the editor of NEJM. A fellow scientist, told me my letter is too critical and would not get published since NEJM earns most of their money on pharmaceutical requests for article reprints.
On September 18 2013, the editor told me they would not print my letter, because space available for correspondence is limited. Fortunately there is plenty of web space, so I post everything here.
Dear Mr. Feys,
I am sorry that we will not be able to print your recent letter to the editor regarding the Sandborn article of 22-Aug-2013. The space available for correspondence is very limited, and we must use our judgment to present a representative selection of the material received. Many worthwhile communications must be declined for lack of space.
Thank you for your interest in the Journal.
Mary Beth Hamel, M.D., M.P.H.
What can be done?
Articles should report statistics that people can understand. The number need to treat is a way to do this. Therefore people are shocked that only 1 patient would benefit from treatment with vedolizumab, and 14 patients get nothing but harmful side-effects. That 1 out of 11 can expect serious harmful effects.
Another thing is that regulatory approval of drugs should be transparent. This concerns public health. The BLA request at FDA involves the submission of clinical trial data such as from this study. Unfortunately, from then, the FDA approval processes are less transparent. It is not clear what the FDA considers ‘a clinical relevant and safe drug effect’. Is this based on CI? Is study methodology a criterion? Does the FDA consider measures taken to secure trial blinding?
I propose that BLA and New Drug Applications (NDA) investigations are routinely doubled by independent evaluators who have full access to anonymized patient data. Even better is the conduct of a independent government sponsored trial. This would also allow more ethical study designs, such as the balanced placebo design. This design was already proposed back in 1981. It features better separation of placebo and true drug effects.
What do you think about this study findings? How would you interpret them? Leave a comment below and start discussion!
*This post is part of PhD Thesis. Illustrations reproduced with permission from Sandborn WJ, Feagan BG, Rutgeerts P, Hanauer S, Colombel J-F, Sands BE, Lukas M, Fedorak RN, Lee S, Bressler B, Fox I, Rosario M, Sankoh S, Xu J, Stephens K, Milch C, Parikh A: Vedolizumab as Induction and Maintenance Therapy for Crohn’s Disease. N Engl J Med 2013, 369:711–721., Copyright Massachusetts Medical Society.