Imagine you are a group of scientists and want to find out whether a novel drug X works on a specific problem Y. You run the following study:
-
You enroll a small sample of 14 participants who have the problem Y.
-
You give these 14 participants your novel drug, but you do not enroll a second group who received placebo medication, which means that any effect you find may be due to the placebo effect.
-
On top of that, all your 14 patients keep using a well-accepted medication for problem Y while taking the novel drug X, meaning that any changes you find may be due to their normal medication, not X.
-
You measure improvements of problem Y 240 minutes and 3 months after the treatment.
-
You find evidence of improvements for 7 of 14 people 240 minutes after treatment, and for 2 of 14 people 3 months later; it is unclear whether this is due to placebo, normal medication, or other reasons.
-
You write a scientific paper with the strong conclusion: “repeated doses of X rapidly and robustly decreased Y.”
This is exactly what what Ionescu et al. (2016) did in a new study entitled “Rapid and Sustained Reductions in Current Suicidal Ideation Following Repeated Doses of Intravenous Ketamine: Secondary Analysis of an Open-Label Study”, published in the Journal of Clinical Psychiatry.
Ketamine as treatment for depression or suicidal ideation The authors tested the viability of Ketamine as a treatment of suicidal ideation, as described in the summary above. Wikipedia sums up nicely what Ketamine is:
“Ketamine is medication mainly used for starting and maintaining anesthesia. It induces a trance-like state while providing pain relief, sedation, and memory loss. Other uses include for chronic pain and for sedation in intensive care.”
Ketamine has become a hot topic of study in the last years, both as depression treatment, and as rapid acting treatment for suicidal ideation. Three reviews so far provide very mixed evidence: Wilkinson & Sanacora (2016) concluded recently that evidence of Ketamine effects on suicidal ideation is “very preliminary”, because studies so far were very small, often not placebo controlled, and samples often do not contain patients with high levels of suicidal ideation due to ethical concerns. Iadarola et al. (2015) draw somewhat more positive conclusions, as do Reinstatler & Youssef (2015), but the actual studies they describe show that effects of Ketamine on depressive symptoms or suicidal ideation rarely last beyond 24 hours - it’s not secret that depressive symptoms and suicidal ideation do. A fourth meta-analysis by McGirr et al. (2015) of 7 studies on the efficacy of Ketamine treatment on depression shows consistent effects lasting up to 7 days; however, they find some evidence for publication bias, the average sample size per study was 26 (meaning 13 per treatment arm), and 7 days is not a sufficient time period to investigate treatment effects. To quote Prof Colleen Loo from the University of New South Wales, a Ketamine researcher herself: “Feeling better for a few days is no good with a chronic disease like depression”.
Ionescu et al. 2016 Let’s return to the study of Ionescu et al.: they showed that depressed patients with suicidal thoughts who were slightly sedated via Ketamine showed somewhat reduced levels of suicidal ideation minutes to hours after infusions, but that there were no lasting effects. It is possible that similar effects would be achieved with all sorts of psychedelic or psychotropic drugs – or even placebo – but that doesn’t make them effective treatment options for depression or suicidal ideation. Similarly, giving Ketamine to people with cancer may relief symptom burden for a few hours, but that likewise would not mean it is a reasonable cancer treatment.
While half of the patients showed improvements on suicidal ideation 240 minutes after infusions, only 2 of 14 patients showed improvements over the course of 3 months. Nonetheless, the authors conclude:
“Despite ketamine’s short half-life (approximately 2.5 hours), its antisuicidal (and antidepressant) effects are sustained well beyond the elimination of the drug, in some patients.”
There is no evidence at all for this conclusion in the paper, because there is no way to know whether this improvement was not a placebo effect. In fact, the placebo effect usually leads to improvements in 1 of 3 patients, meaning that 2 of 14 is less than would be expected in a placebo group (Khan & Brown, 2015). Moreover, there is no way to know whether this improvement was due to Ketamine because all patients continued to use their standard antidepressant medication. Finally, another very large concern is regression to the mean, which very likely contributed to the reduction of suicidal ideation, because patients were selected on suicidal ideation.
Despite these serious shortcomings, the main conclusion in the abstract reads:
“In this preliminary study, repeated doses of open-label ketamine rapidly and robustly decreased suicidal ideation in pharmacologically treated outpatients with treatment-resistant depression with stable suicidal thoughts.”
And the title reads:
“Rapid and Sustained Reductions in Current Suicidal Ideation …”
Sorry, as much respect I have for the authors - many of them are great names in the field, and deserve great respect for their brilliant scientific and clinical contributions - conclusion and title are simply incorrect, and I wish the reviewers or editor (or one of the 14 authors) would have pointed that out. There is no “sustained reduction”, and the paper provides little support for the viability of Ketamine as a “robust” or “rapid” treatment for suicidal ideation. Think about like this: if a doctor prescribed you a drug and told you it leads to “rapid and sustained reductions” of your condition, and you found out that it only shows a sustained effect in 14% of patients - for which the effect is very likely not even due to the drug - you would be a very upset patient.
The proper conclusion from the available evidence is: “In our underpowered study that is not representative of depressive patients, only half of the patients showed improvements, for only a few hours, and nearly nobody showed lasting improvements. Due to the lack of a placebo group, and because patients continued their standard medication, interpreting the results of this study as supporting the viability of Ketamine as treatment for suicidal is scientifically impossible.”
To be fair, the authors call the paper preliminary (albeit not in the title), and the sample was a group of treatment resistant patients, making this a difficult study. And obviously, helping even 1 in 10 depressed patients should make it worth to explore new treatments, and suicidal ideation is among the most relevant depression symptoms we need to address, and address quickly. The approach taken here - trying to address symptoms instead of syndromes - is consistent with a novel research framework focused on individual symptoms, and very much worth supporting. But how informative are studies without a placebo group? One may argue that they pose unacceptable ethical problems in terms of ratio of useful information to patient risk and resource waste, because literally 0 reliable information can be gained from a study in n=14 without a placebo arm. And the fact that a placebo group is ethically problematic in severely depressed patients is no counter argument - in that case, we can enroll a second group of patients into a second treatment arm, and give them, for instance, benzodiazepines.
For now, future studies will cite the study as evidence for the efficacy of Ketamine, which is inconsistent with the data of the study. Moreover, the study has been rather widely discussed by media as providing evidence for Ketamine efficacy, which, again, is inconsistent with the data. And this in part the authors’ fault – due to the strongly exaggerated conclusions throughout the paper, including the title.
A final concern is the stunning conflicts of interests section. Then again, it is driven primarily by one author, and the majority of authors report no conflicts of interests, so this should not be overinterpreted. Nonetheless, seeing that studies with conflicts of interest conclude 5 times more often than independent investigations that drugs are efficacious (Perlis et al., 2005), conflicts of interests should always be a concern and taken seriously.
General concerns about antidepressant trials It is helpful to understand this commentary in a much larger context - the scientific and methodological quality and rigor of antidepressants trials in general. The study by Ionescu et al. is the sixth study in only a few months that appeared in a high-impact psychiatry journal where the main conclusions are at best exaggerated (I wrote about the other studies here: 1, 2, 3, 4, 5), and my concern is that these studies are not really moving the field along towards the goal we all share: finding better treatments for people suffering from mental illness. What we need are studies with much larger samples, with placebo groups where possible, and otherwise with multiple treatment arms so we can compare the efficacy of drugs. We also need to check on people for longer periods - it is crazy that the majority of depression trials do only last 6-8 weeks, which is not an adequate timeframe for the large majority of patients. Finally, we need to follow Ionescu et al. in studying individual symptoms instead of syndromes. Depressed patients differ dramatically in their symptoms, and different treatments likely have very different positive and negative effects on different symptoms.
And I understand that getting funding for such studies is a problem, and the blame here is certainly not predominantly on the scientists and clinicians involved in conducting the clinical trials: we need journals that are much more rigorous in their peer review, we need better methodological standards for clinical trials, and we need funding agencies actually spending money on larger, high-quality studies, instead of multiple smaller studies without placebo groups.