misleading medical-science stories – & why most published research findings are false

By Alison Campbell 28/12/2009

It’s been pointed out to me that this post could be construed as dissing the EPP drug & its producers. This was not in any way my intention – the post was about the fairly poor reporting around a couple of medical science stories, ending with a wish that science journalists were more aware of the fact that much medical research tends (for a whole lot of perfectly good reasons!) to be ‘wrong’. (Hopefully this awareness would be followed by a decline in the hyperbolic reports about new breakthroughs & discoveries…)

Two stories in this morning’s Herald caught my eye – & made me sigh. One announced the imminent release of a ‘UV vaccine’, & the other elevated another fruit to the level of ‘superfood’. Both were examples of how not to present research findings to the public.

The first (originally from the UK paper The Telegraph) carries the headline UV protection drug prevents skin cancer & the first paragraph tells readers that [an] anti-sunburn drug that protects even the fairest-skinned people from skin cancer and ultra-violet rays could soon be made available to the public. It’s difficult to assess all the claims made in the article because, unfortunately, the actual research study on which this story appears to be based has yet to be published. However, there are a number of statements in the story that set my teeth on edge (beginning with ‘the results [of the study] will be published soon’).

The drug in question (afamelanotide) has been developed by an Australian company to treat people with a condition called EPP, or  erythropoietic protoporphyria. Porphyrins are chemicals involved in a number of important biochemical processes within the body, including the production of red blood cells. The metabolic pathway your body uses to produce porphyrins can be interrupted by any one of several different mutuations, with the result that intermediate chemicals build up in the body. In the case of EPP (a rare disorder that affects less than 1 in 200,000), a mutation sees an excess of the chemical protoporphyrin in patients’ blood & skin – & in the skin it absorbs and reacts with light to give an extremely painful burning sensation. Individuals with the most severe (& extremely rare) form, congenital EPP, can’t tolerate any natural light, as their skin blisters & burns on exposure. Not nice at all.

It sounds as if the drug – which has just gone through a 12-month trial in EPP patients – acts to stimulate eumelanin production in the skin (not ‘between light and a person’s skin’, as the newspaper story would have it), providing a protective barrier against the worst of the sunlight. However, I couldn’t find anything in the company’s own available documents to suggest that it might also protect against skin cancer – and anyway, while skin cancer is more common in fair-skinned individuals, it does still affect those with more heavily pigmented skins. (A tan is not a 100% guarantee of immunity from skin cancer.) For example, the frequency of melanoma in African Americans is 1 in 100,000 (compared to 22 in 100,000 for US Caucasians). In any event, if afamelanotide did ‘prevent’ skin cancer, then you might expect to see it licensed  approved for this use – but the US FDA has approved it for treatment of further clinical trials in people with EPP, & this is how the company concerned sees it being used[it] is proposed that afamelanotide will assist those patients who are most severely affected by UV and light, with the aim of improving their quality of life.

So the absolute statement that the new drug ‘prevents skin cancer’ is misleading, at the least. And while the drug may well reduce the rates of skin cancer in EPP patients, where any sun exposure is damaging, there is as yet nothing to demonstrate that it would have the same protective effect in the wider population. Nor should it be marketed to that wider population, as it sounds as if there is the potential for damage to immune & cardiovascular systems unless use of the drug is carefully monitored. Unfortunately, this information comes at the end of the news story, which otherwise hypes up the new pharmaceutical to the population at large.

The other story (also not yet on the Herald website, but carried in The Guardian in the UK) is called Fruit could be answer to hospital superbug. The latest addition to the ‘superfood’ pantheon is the pomegranate (which must be something of an acquired taste – one which so far I have yet to gain). We’re told that [s]cientists have discovered the power of fruit as a potential new weapon in the fight against MRSA and other hospital superbugs. Now, methicillin-resistant Staphylococcus aureus (MRSA) is a real nasty, resistant to methicillin & other related penicillin-type antibiotics, so the finding that a novel treatment may be available is cause for cautious optimism. And apparently a combination of pomegranate rind, vitamin C, & a metal salt ‘greatly enhanced [the] infection-fighting properties’ of this particular fruit. But not against Staph.aureus – the study on which this report seems to be based used the a multidrug-resistant strain of the bacterium Pseudomonas aeruginosa (Gould et al. 2009). A picky point, but how hard is it to be accurate? Would this detract from the story?

The article quotes one of the research team as saying that "It was the mix that fantastically increased the activity – there was synergy, where the combined effects were much greater than those exhibited by individual components…"  And then goes on to say that The tests were conducted using microbes taken from hospital patients. Scientists found that pomegranate rind mixed with metal salts was most effective against MRSA, while adding vitamin C helped tackle other common hospital infections.

But – and there are at least a couple of big ‘buts’ here… How well does this concoction compare to other currently available drugs (like vancomycin) that are still effective against most strains of MRSA (or of P.aeruginosa)? The story doesn’t tell us; the way it reads, the study looked at pomegranate rind, alone & variously combined with the other two ingredients. This is borne out by the abstract of the research paper itself, in which the authors conclude that [t]hese results warrant further investigation of PRE as a potential source of new antimicrobial agents. And – we don’t actually know if the pomegranate mixture would have the same antimicrobial effects in humans. Why not? Because these preliminary trials were in vitro ie on bacteria growing on agar in petri dishes. (To be fair, the article does – eventually – get around to pointing this out.) There’s a very long way to go indeed before this one makes it into regular use in hospital care – something the authors of the paper were careful to say, but which the article’s headline, & its first several paragraphs, gloss over.

All this makes me wonder whether a paper by John Ioannidis – published 4 years ago & freely available on-line – ought to be required reading for journalists writing stories about medical ‘breakthroughs’. Writing in PLoS Medicine, Ioannidis makes the point that in the field of medical research, ‘most published research findings are false’. (I need to emphasise here that he is talking about medical research – his essay has been used elsewhere in attemtps to support claims that most research in all fields of science is false, which is not necessarily the case.)

Ioannidis begins by noting that [p]ublished research findings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment. No surprises here, to anyone working in science; this simply reflects how science operates. Hypotheses are tested, & tested again, & it’s by no means out of the ordinary for an hypothesis to be overturned as a result of this rigorous evaluation. Unfortunately I suspect many non-scientists find this confusing. Having been sold the line that science gives us ‘the truth’; people can’t really be blamed for thinking, well, if scientists were wrong about this, what else are they wrong about? (They can be blamed if they abuse this misunderstanding in others to promote their own particular agenda, however.)

Anyway, Ioannidis goes on to point out that a number of factors can influence the production (& dissemination) of research findings that are false ie not a good reflection of what is actually going on in the biomedical system being studied. One of these is bias: the combination of various design, data, analysis and presentation factors that tend to produce researh findings when they should not be produced. (This can also at times obscure ‘true’ research findings, by the way.) Inefficient use of data, conflicts of interest (potentially a real problem with commercial development of pharmaceutical products), and a failure to notice when findings are statistically significant, can all have an impact on how research findings are presented.

There’s also an unfortunate tendency to trumpet findings by single teams of researchers, viewing their work in isolation at a time when there may be several different teams investigating the same topic. This can be exacerbated by pressure to publish & to get results out there. As Ioannidis says, [w]ith many teams working on the same field and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritise on pursuing and disseminating its most impressive ‘positive’ results.  In addition, size of the study (or size of its sample population) is important – studies with only a few patients involved may show a ‘positive’ outcome when in fact the apparent results are due to chance alone (or to some other factor that the design of the study failed to take into consideration).

Required reading? I think so – but I know it’s not likely to happen any time soon. "Gee whiz" & "latest scientific/medical/technological breakthrough" stories are perceived as more attractive to readers, & for whatever reason scientific accuracy seems to take a back seat. Which is a real pity, as it distorts the public view of what science is & how it operates, at a time when more than ever we need people to have at least some understanding of the nature & value of science.


S.W.Gould, M.D.Fielder, A.F.Kelly, W.El Sankary & D.P.Naughton (2009) Antimicrobial pomegranate rind extracts: enhancement by Cu(II) and vitamin C combinations against clinical isolates of Pseudomonas aeruginosa. British Journal of Biomedical Science 66(3): 129-132

J.P.A.Ioannidis (2005) Why most published research findings are false. PLos Medicine 2(8): e124. doi: 10.1371/journal.pmed.0020124


0 Responses to “misleading medical-science stories – & why most published research findings are false”

  • Greetings Alison Campbell,

    Regarding your commentary with respect to the article about melanotan-1 (referred to by its generic name “afamelanotide”) you are very much correct and in fact it is worse than you’ve described. The drug was in fact first described in a University of Arizona paper principally authored by Tomi K. Sawyer that was published in 1980 entitled: “4-Norleucine, 7-D-phenylalanine-alpha-melanocyte-stimulating hormone: a highly potent alpha-melanotropin with ultralong biological activity.
    The initial work of developing the peptide as a drug (including preclinical studies and phase I human trials) was primarily carried out by researchers at the University of Arizona and the subcutaneous administration method for the drug was established by additional U.S. researchers. Clinuvel (originally known as EpiTan) didn’t actually begin to work on the drug until they fully licensed it in 2000. Their primary role has been to conduct subsequent human trials on the drug in order to bring it to market. As EpiTan Clinuvel was initially pursuing the drug primarily as a photoprotection inducing agent against sunburns. It wasn’t until late 2005 / early 2006 that they began aggressively pursuing indications like EPP and CEP as well as potential additional light affected dermal indications . The terminology “UV vaccine” is I suspect a marketing ploy that is hopelessly unscientific see my commentary here: http://www.technologyreview.com/biomedicine/24267/page1/

    Thank you for sharing your thoughts concerning these matters.

    -Scott Stevenson
    Melanotan.org – afamelanotide Founder & Admin

    • Hi Scott, thanks for your comments
      I don’t have any issues with Clinuvel’s involvement in getting the drug to market, & certainly it looks as if it would be a valuable thing to have from the point of view of someone who has EPP or who is handling the treatment of someone with this problem – my concerns lie with the way that these stories are reported in the press. The ‘UV vaccine’ label can probably be laid at the feet of the media, as can the suggestion in the media story that the drug would ‘prevent’ skin cancer. They certainly attract attention & may raise expectations in the wider population that can’t be met; a positive spin-off there is that the story may raise awareness in people who have EPP so I guess there’s something of a trade-off there.