2 Comments

… consider testing if your idea is sound first using the review literature.

Perhaps one of the easier ways for readers (journalists, advocates, spokespeople, etc.) less familiar with a field to test for themselves if their claim is widely-accepted in the research literature[1] is to check if independent, well-regarded sources from the review literature make the claim.

A fairly common theme in my blog comments and elsewhere is interested parties enter comments to air their views on a topical issue.

Often it is clear that no-one conversing has enough ‘of-the-cuff’ background to be able to offer sound judgement, at least not without devoting considerable time to the effort.

I’d like share an approach that can sometimes be used to check if there is a clear, widely-accepted, conclusion on a particular claim that doesn’t involve the research literature directly.

It’s not perfect, but potentially it’s more straight-forward.

The idea is to leverage the review literature which is more compact, and sometimes easier to read, than the primary research literature, and covers a field more widely than any one research paper will.

It rests on that if a claim is important and widely-held, it is likely to be present in the review literature.

Conversely, if there are no statements supporting a claim in well-regarded review publications, then it is most likely that there is no clear, widely-accepted, conclusions on the claim in the research literature.[2]

Note that I’m using one type of literature, the review literature, to draw a tentative conclusion about another kind of literature, the research literature.

Let me elaborate.

It is relatively easy to dismiss papers that don’t match the criteria needed to prove a point.

Criticism is usually easy, and in most cases there are obvious pointers. On closer inspection, the paper is in fact about a different issue in the same general field. Obvious confounders have not been controlled. The effect size is far too small for the claim advanced. And so on.

Backing your own claim or a research paper’s argument is much harder.

Among the many difficulties of using the primary research are two key problems I consider relevant here:

  1. Individual papers in science aren’t of that much use taken on their own. It is exceptionally rare to find a single paper that covers all the issues of a topic, in fact it’d be essentially impossible in today’s science. (As if it ever were possible for that matter.)
  2. To examine a paper that offers promise requires considerable background, without which you really cannot draw more than a superficial judgement of the argument presented in the paper. Most research today draws on several quite different areas of expertise. Unless you are an expert in the niche topic – and even then – you are unlikely to know the nuances required to accurately interpret the data presented.

The up-shot is that to make a positive judgement, as opposed to getting rid of obviously irrelevant or flawed work, takes considerable effort and considerable background.

Few of us have the full background ‘ready-made’ to do it at short notice, or the time. (For the latter: obsessed retirees excepted!)

Perhaps one of the easier ways of judging if a field has a mature judgement on a topic is to see if there are several review papers by different authors – with no affiliations to each-other – from different reputable, ideally peer-reviewed, journals presenting essentially the same conclusion about a subject.

Here we are making use of the expertise of others, while being aware of the pitfalls.

Review papers are intended to advise those immediately outside of an area by providing a summary of the current state of the field in the judgement of the author. They are the judgements of the author(s), and with that it’s wise to be aware of any (academic) ‘slant’ the author might have.

Another ‘gotcha’ to watch out for is that some journals place more-or-less everything other than research reports under the title ‘review’, which means you need to weed out hypothesis proposals, general philosophical mutterings, and whatnot.

In the case of using PubMed add to your chosen keywords, ‘review’.

You should find that this locates considerably fewer articles that a search open to all types of articles.

To avoid missing reviews, try keep the keywords wider than your specific claim: some review papers will cover more than just your specific claim. Another benefit of this is that it allows you to do a one-pass search: multi-pass searches need care not to accidentally cut useful material out.

Eliminate false matches. Keyword-based invariably throw up dross. Some titles will obviously irrelevant from the title alone. Get rid of idle hypotheses or philosophical commentary or the odd older scientist reminiscing their past or other irrelevant articles. Take note of potential conflicts of interest. (Pharmaceutical and ‘alternative remedy’ companies publish reviews, sometimes slanted to their own interests: universities and independent research centres are more likely sources of neutral judgement.)

Often reading the abstracts will prove sufficient to show if there is widely-held support for a claim, or not.

While this is one useful tool, you cannot dispense caution over the sources of the review articles.

Reviews from less that stellar sources are notorious for being hypotheses in drag, opportunities for the author to spin their pet idea. (A recent example might be PalMD reviewing reader’s claim, presented from a review article, of broccoli’s cancer-fighting power.)

Bear in mind, too, that the same author is likely to repeat the same judgement: several citations of one person’s same judgement, is not several independent judgements of the claim.

This approach will not remove the need to understand how conclusions are presented, e.g. the distinction between absolute and relative risk.

If you can’t find substantiation of your claim in the review literature, it likely means one of several things:

  1. The science is simply too new for there to be an accepted conclusion. A frequent issue is media reporting each new scientific paper as if it were the last word, without regard for the state of play. If there is no firm conclusions in the review literature, the state of play may be ‘work in progress’.
  2. The claim is too historic or too well-accepted it may be assumed as part of the reader’s background knowledge. This is very unlikely to be the case for claims about alternative remedies or anything in the current events media – they’re too new, for one thing.
  3. The claim is too far-fetched to be seriously considered. Science often uses a concept of prior plausibility. It essentially asks that new ideas follow fairly directly from established ones. If new ideas require large ‘leaps of faith’ or contradict too much established science, they will (rightfully) be regarded with deep skepticism until substantial evidence is presented. Many – I would say most –reviewers are likely to stay away from unsubstantiated claims, or at best politely point out the lack of substance backing them.

Footnotes

[1]

The comments that initiated my writing involve claims by others that high-dose intravenous vitamin C is a sound treatment for someone in critical care with pneumonia. I would present this as a worked example as it’s a fairly good illustration of the approach, but I would prefer this article focus on the general approach, rather than it be distracted by any one issue.

[2]

Strictly speaking you can’t prove an absence of something, but pragmatically if the claim is of interest to researchers this approach will work often enough to justify it’s use. Furthermore, when you do find several independent, soundly-based reviews backing your claim you have a useful point to enter the research literature.


Other articles on Code for life:

Gene patents, an amicus curiae

Fainting kittens – feline myotonia congenita?

Vaccine promotion – the medium matters too

Paul Nurse on ‘anti-science doubters’ and the blogosphere

Thoughts towards a human brain neural connection map