The content of this blog has been adapted and expanded from comments I provided to Vaccines and Science this week.
The perfectly respectable MDPI journal called Vaccines made a mistake but they addressed it fast. Somehow an article assessing vaccine safety and effectiveness that was fundamentally flawed and made outrageous claims slipped through the system. Not the first time something like this has happened nor will it be the last. The unmentionable mother of them all was published in the prestigious Lancet in 1998 and it took over a decade for a full retraction. This one took under a week (See Here for Retraction Watch). Below I discuss why this article should never have left the editorial office.
During a time of unprecedented spread of misinformation and the challenges we face in immunising the global population against a pandemic virus it is even more critical that articles such as this receive rigorous appraisal by topic experts before entering the public domain, particularly those which could fuel vaccine concerns. I would like to acknowledge the overwhelming workloads that many currently face and the fact that mistakes can happen. The damage to vaccine confidence and trust that can occur through the distribution of pseudoscience in good quality academic journals cannot be underestimated. However, there is no use crying over spilt milk. I am so pleased to see the speed at which this has been addressed.
Some general comments about the article
The article titled The Safety of COVID-19 Vaccinations—We Should Rethink the Policy authored by clinical psychologist Harald Walach, physist Rainer Klement, and independent data scientist Wouter Aukema, is about the safety of COVID-19 vaccines. The study used data from a spontaneous adverse event reporting system and calculated a ‘number needed to vaccinate’ to cause harm.
The stated objective of the study was to determine the effectiveness of the vaccines and to compare them with the costs in terms of side effects. I think it is important to not that there is no inclusion of a topic expert among the authors. Also, (despite two of the three being anonymous) there is no indication that the referees who reviewed this work have experience in vaccine safety, the central theme of the paper.
The language also suggests that there is little familiarity with the topic and there are a number of instances of emotive and condescending comments such as: “Perhaps it might be necessary to dampen the enthusiasm by sober facts?” Such subjective language has no place in an academic manuscript.
The introduction is riddled with clues that the authors are unfamiliar with the topic.
The first sentence of the entire articles is a give away.
“In the course of the SARS-CoV2 pandemic, new regulatory frameworks were put in place that allowed for the expedited review of data and admission of new vaccines without safety data .”
Of course no vaccines were authorised for use without safety data. The number of participants providing safety data for each COVID-19 vaccines was consistent with other contemporary Phase III vaccine trials and provided robust information about the safety to the extent that any Phase III trial can. Ongoing data was provided. This is not ‘without safety data’.
In addition, the reference ‘(1)’ does not support this statement. Reference 1 is an opinion piece in a Swiss medical bulletin. It is in German but as luck would have it, I have a post doc who speaks and reads German fluently.
Then one comes to the second sentence.
“Many of the new vaccines use completely new technologies that have never been used in humans before.”
This is not true. Viral vector technologies have been used widely in populations (e.g. For Ebola). mRNA technologies have been used in humans for decades, beginning with cancer vaccines and progressing to infectious diseases. There have been early Phase I/II human trials for coronavirus mRNA vaccines prior to COVID-19.
The methods are fundamentally flawed – garbage in, garbage out
Again, we just need the first sentence, at which point I do face plant.
“We used a large Israeli field study  that involved approximately one million persons and the data reported therein to calculate the number needed to vaccinate (NNTV) to prevent one case of SARS-CoV2 infection and to prevent one death caused by COVID-19. In addition, we used the most prominent trial data from regulatory phase 3 trials to assess the NNTV.”
Vaccine effectiveness is never calculated by using an NNT/NNV. It cannot account for the effect on transmission and herd effects of vaccines. Vaccine effectiveness is calculated broadly using the formula below. It is expressed as a percentage, with confidence intervals (a measure of precision).
In other words, vaccine efficacy (measured in clinical trials) or vaccine effectiveness (measured in observed populations receiving the vaccine). Simply put these are measures of the chance of an unvaccinated person getting sick compared to the chance of a vaccinated person getting sick.
The authors refer to the cases identified in the EMA database as “side effects”. This should read suspected side effects. The source cited makes the following points:
Spontaneous adverse event reporting systems are for vaccine safety signal detection. If a concerning volume or unexpected type or pattern of reports are received this might represent the detection of a signal. These systems cannot determine causality. This type of data cannot be used to determine vaccine safety or the risk of vaccine side effects.
If a signal is detected it should be verified by determining the observed number of cases over the expected number (those anticipated as normal background rates of a condition). An example of this is the work in Denmark that assessed the observed over expected for blood clots after COVID-19 vaccines using a population based cohort study. This is a nice example of the next step.
If the signal is verified then risk should be assessed using methods such as the self-controlled case series or cohort studies. Only then can conclusion about risk be confirmed.
The results are completely meaningless because the data were inappropriately interpreted to represent vaccine reactions (events caused by administration of a vaccine) and because the methods used to calculate risk/benefit were also inappropriate approaches to address the research objectives. Garbage went in, got statistically abused, and what came out the end was nonsense.
The Discussion/Conclusions ignore the entire field of vaccine science
The article discussion ignores the body of vaccine safety science. There is no attempt to critically appraise the significant existing body of evidence for COVID-19 vaccine safety. None. Zip.
“Ideally, independent scientists should carry out thorough case reviews of the very severe cases, so that there can be evidence-based recommendations on who is likely to benefit from a SARS-CoV2 vaccination and who is in danger of suffering from side effects”.
My jaw dropped, particularly given the thousands of dedicated experts all over the world working long hours performing this task. This illustrates that the authors do not appear to understand how vaccine safety is assessed nor the fact that serious events are rigorously assessed.
The storm that followed…
When the article was published a Twitter storm ensued with some 14000 tweets in a few days. The attention score for the article soared. When one of the journal editorial board members saw the article they alerted the rest of the members. Outcry and mass resignations followed, including my own.
There was no way I could be associated with a journal that let something like this fly. And I was surprised that it had because my previous experience with this journal had been that they took considerable care over these things. Because retractions can take months to years and lobbying I initially wished to distance myself as this mistake was such a doozer. However, the Journal’s response has impressed me.
Prof Eugene van Puijenbroek from the Netherlands pharmacovigilance centre and topic expert wrote a critique to which the authors responded, basically reiterating their paper. The nugget below is from the author’s response to Prof van Puijenbroek:
“Ideally the consequence of this debate is that someone sets up a systematic observational post-marketing surveillance study in a large number of vaccinated persons under public scrutiny to really document the side-effects that can be causally related to the vaccine.”
This illustrates the complete ignorance of the authors about how vaccines are assessed after authorisation and by whom. This is like me saying that we really should recommend cyclists wear helmets and that safety belts should be considered in cars. The body of science encompassing post-marketing vaccine safety is called pharmacoepidemiology and lots of people spend their days doing it. Many countries have systematic observational post-marketing surveillance. Examples include Denmark, United States, Australia, Finland, Canada, United Kingdom. Many other countries have capability and conduct ad-hoc post-marketing vaccine safety studies. There are also multinational consortiums. In the EU this is VAC4EU which includes many European countries, and globally this is the Global Vaccine Data Network. There are hundreds of millions of persons under observation in systematic observational post-marketing surveillance and active studies are in play. How could the authors possibly overlook these facts?