"But a Study has Shown …"

By Michael Edmonds 23/04/2013 4


It’s probably familiar to many of us … you are debating some issue when someone with an opposing comes out with “but I read a study that showed … ” and then goes on to explain how the so called study justifies their position. You then track down the study (assuming they can tell you where to find it) and it does nothing of the sort.

This has happened to me twice in the past week.

First, someone trying to justify homeopathy by attacking conventional medicine comes out with the suggestion that a study has shown that during a doctor’s strike in New York, mortality dropped, which demonstrates that without doctors prescribing their “poisons” less people died.

Tracking down the paper I quickly discovered

1) Five doctor’s strikes had been studied

2) None were in New York

3) Only two out of five showed a decrease in mortality

4) The authors suggested that this drop in mortality was most likely due to no elective surgeries taking place during the strike (i.e. so no relation to the prescription of drugs)

 

Then in another debate about same sex marriage I had someone tell me of a study they “remembered” that showed that same sex marriages only lasted one and a half years on average, with eight partners during that time (not sure if that is supposed to be each, or a combined score).

I did a bit of reading on line, as the person pushing this idea couldn’t remember the source, and discovered it is a major distortion of research into HIV transmission. A good summary can be found here, however key points include

1) The study was carried out before gay marriage was legal in the Netherlands

2) Monogamous gay men were excluded from the study

3) Did not involve lesbians

 

This is why I am so wary when people use the statement “But a study has shown …”

If you do want to use a study to support whatever position you are arguing could I please suggest the following:

1) Make sure the study actually exists (and ideally you should be able to provide any interested person with a reference to it)

2) Read it and make sure it does support your position

3) Make sure it is from a reputable source (reputable scientific journals – yes, magazines – no)

4) Make sure you understand the difference between causation and correlation

5) Make sure that there aren’t ten other reputable studies refuting your position (if this is the case then some care has to be taken to work out why there is disagreement, and which source(s) are likely to be more reliable)

6) If you can’t fulfill steps 1 to 5 above then don’t use it with me in a debate, especially if, like so many people, you can’t even get past step 1!

 

 

 


4 Responses to “"But a Study has Shown …"”

  • I would also add an extra step for the intellectually honest:

    7. Understand the subject under study well enough that you can suggest confounding factors for studies even when you like/agree with the results.

  • Good point Darcy,

    And understanding the limitations of what the study can and cant tell us

  • It often seems to be the case with point 5. The more heavily studied an area the more likely there are studies you can pick out studies that go against the otherwise overwhelming evidence. Flouridation of water supplies and vaccination are good examples where people ignore hundreds to tens of thousands of studies to pick a few that kinda support their cause. To the lay person it seems to validate their position, but to someone familiar with the field it makes them look like an idiot.