By Grant Jacobs 07/02/2019 18


A recent study found most of the 10 most popular ‘health’ articles of 2018 were inaccurate. Similar but less skewed results were seen for the top 100 articles. I dislike pointing fingers. For these things, they tend to point at journalists and social media. Instead, I’d like to highlight something less talked about when discussing inaccurate health and science news: could editors help?

Many newspapers position themselves as fighting fake news. That’s mostly for politics, but there’s a lot that news outlets might do for science and health news, too.

Ideally these inaccurate stories would never published in the first place. Then we wouldn’t have to worry about what’s in social media! In reality there will always be some inaccurate stories in social media, but news outlets can play their part.

It’d be great to see more editors offer strong gatekeeping. Perhaps also encouraging writers to do better. Not an easy thing to do in these times with non-existent budgets. But perhaps it’s a necessary thing? Science and health stories affect people too, after all.

The study that brought this mind looks at health news. The brief for health overlaps science, so they’re taken together here. Many of the issues from one spill over to the other.

The Health Feedback and Credibility Coalition study

They have chosen the ‘most popular’ articles from news outlets judged by the number of social media engagements (comments, likes, how often they were shared), then checked their accuracy. For the top 10 articles they invited experts to review them; for the top 100 they used Health Feedback’s science editors.

If you’re curious you can scan their list of all 100 articles. The most popular and not credible article is Federal Study Finds Marijuana 100X Less Toxic Than Alcohol, Safer Than Tobacco.

The results for the top ten most popular articles are skewed towards poor quality articles. Having said that with a sample size that small it’s hard to say much. It might be a useful reminder that because something is popular, it’s not necessarily sound.

For the top 100 articles, it’s split down the middle. Factually inaccurate stories spread roughly as readily as accurate material. (Bear in mind reviewing the top 100 is less in-depth than for the top 10.)

(Click to enlarge.)

I’m not surprised to read that food and nutrition articles did especially badly, “only 5 out of 18 articles received a positive credibility rating.”

The strongest category was ‘disease and disease treatment’,

Although 4 of the 21 articles originated from websites of questionable credibility (these 4 accounted for all the negative ratings in this category), the rest of the articles came from established news organizations, such as Time, The Atlantic, CNN and ABC News. This indicates again that well-established news sources are more likely to report science accurately, likely as a result of higher journalistic standards and stronger oversight over quality of sources and content.

They don’t say this directly, but editorial standards may be a factor. It would be interesting to examine the role of editors in these stories.

A role for editors: better gate-keeping and ownership?

As newspapers increasingly step up to fake news, it’d be great to the see the same for science and health news. You have to empathise with editors, though. They’ll be frantically marshalling everything as it is. Any solutions want to be practical.

A few loose thoughts include,

  • The survey notes use of dubious websites. A simple check of the source(s) used may go a long way.
  • If you can’t hire specialist editors, can you out-source checking of individual articles?
  • Perhaps attending Science Media Centre training weekends for editors? (I have no idea as to what extent they offer these or not.)
  • Editors might try use specialist writers for specialist topics. (Related to this a simple rule of thumb I use is ‘can I critique this?’ If I can’t it’s either a topic I should avoid, or I’m looking at serious amount of backgrounding and checking. Editors might ask the same of writers: are they able to critique the claims?[1])
  • To what extent do editors ask journalists to present fact-checking material? Do they list the source backing each factual statement? Are those sources sound?

Whatever the ‘solution’ to poor quality material in health coverage, it’d be great to see editors step up to meet the challenge. We all (writers and editors) can do so much better.

Feel free to add your thoughts or suggestions in the comments.

Other articles on Code for life

Popularity does not mean effectiveness or sensibility

Corrections and clarifications

Map shows New Zealand with lowest death rate on earth in 1856, over 11 in 1000 dying

Dear journalists and editors, (Also: Dear journalists and editors, (again))

Three kinds of knowledge about science and journalism

Science journalism–critical analysis, not debate

Scientists can’t write?

Footnotes

The survey authors point at The Guardian for running opinion pieces, including book excerpts. Often these are only clear if the reader checks the blurb immediately after the article. It might help if these were at the top!

They also briefly point at headlines (and by proxy ledes). This is something they have tackled previously. They found that clickbait headlines correlate with poor credibility. Headlines (and ledes) are often written by sub-editors, not the journalist. Ideally they’d reflect the article in a balanced way, but an aspect of headlines is to ‘trap’ readers skimming a page, and start them reading the article. As a consequence headlines easily veer towards clickbait.

1. I’m a bit leery of what may be a dodge. Many articles read as if quotes—from a person, website, scientific paper, etc—are used to ‘get around’ not knowing the science. Potentially that leaves that journalism as uncritical transcription.

About the featured image

The title page of the world’s first newspaper,

Title page of the Relation aller Fürnemmen und gedenckwürdigen Historien from 1609. The German-language ‘Relation’ had been published by Johann Carolus at the latest since 1605 in Strassburg, and is recognized by the World Association of Newspapers as the world’s first newspaper.

Source: Wikipedia. Public domain.


18 Responses to “Taming inaccurate health and science news: could editors help?”

  • I doubt there has ever been a time in history where it has been more important for authors everywhere to own their work and face a consequence if they write crap. Once a two bit rag on the outer most edge of the galaxy could print fake stuff and only the local inhabitants would see it. Today fake news can be written by a dog with a computer from Nenetsia ( I hear that is hard to get to), and it ends up viewed by millions. Maybe we can’t control Russian trolls and bots but I hope the move to crush fake news will grow so that our social norms become such that misinformation is not tolerated. Maybe I am an optimist 🙂

  • It’d also help if outlets like Netflix wouldn’t support the likes of Gwyneth Paltrow’s Goop — Netflix have announced a deal with Goop to run a show on their streaming service. As this opinion piece put it,

    Its [Netflix’s] deal with Goop both validates and amplifies pseudoscience. Which should make everyone feel very queasy indeed.

    https://www.theguardian.com/commentisfree/2019/feb/08/goops-deal-with-netflix-is-a-dangerous-win-for-pseudoscience

    News outlets publishing pseudo-science or inaccurate material do much the same. They validate and amplify stuff that’s at kindest unhelpful, and at worst harmful. People in positions to manage or control this have a great opportunity to help.

    It’s easy to laugh at Goop’s nonsense, as the exchange below from the comments at The Guardian illustrates, but Paltrow’s nonsense ultimately encourages choices that end up hurting people.

    “This sort of thing makes me really mad. I’ll have to get my chakras realigned now.”

    “You can’t align chakras until you steam your vagina and take two pills of Infowars BrainFreeze. It has to be done precisely in this order otherwise your frequencies won’t level up.”

    “I don’t even have a vagina. Is that why I’m so tired all the time?”

    There is an argument that exposure will bring the ridicule that will turn most people away. I’m not if that ridicule reaches the people it needs to, though.

    (Maybe I ought to expand on this as another blog post… I’ll leave it at this for now.)

  • Hi Grant

    Have you looked at this article in that study?

    Because of Anti-Vaxxers, 37 People in Europe Have Died of Measles This Year

    It was given a high credibility rating but yet it provides no evidence to support the bold headline.

    • The article does actually draw the line between anti-vaccine groups and the outbreaks – it cites a PLOS One paper for that. E.g from the summary points of that article,

      A social movement of public health vaccine opposition has been growing in the United States in recent years; subsequently, measles outbreaks have also increased.

      Our findings indicate that new foci of antivaccine activities are being established in major metropolitan areas, rendering select cities vulnerable for vaccination-preventable diseases.

      (I haven’t time to break down the full research paper – sorry; that stuff takes a lot of time.)

      As for the score, the scores are for the full article. A quick skim shows the writer has backed pretty much every claim, with links in almost every paragraph to sources, incl. WHO, CDC, etc. I’m not surprised it scored well.

      The title isn’t wrong though. Bear in mind that piece is about the USA, drawing on Europe as examples. The regional drops in vaccination in the USA are what lies behind the the rise of measles there. Lying behind that are “concerns” over the MMR vaccine pushed a small number groups, notably Wakefield’s supporters and ‘that’ Vaxxed movie. From what I’ve read Wakefield is directly involved in some of it, too.

  • Thanks Grant.

    With respect, the headline says that the deaths of 37 people in Europe were because of antivaxxers but it doesn’t provide any evidence that that is the case. The article draws on USA observation.

    • I notice you’ve not repeated your complaint about it being “given a high credibility rating”. That seemed to be your main point. I take that’s been addressed – the score is for the full piece.

      Sorry if I misdirected my answer re the title in and off itself. I’m tired, and am trying to help many people asking different things in different places.

      Note there two different things here:

      – if the claim is correct
      – if the claim is directly cited

      Not every claim in a piece is going to be directly cited, it’d be unreadable. This isn’t academia!

      Yes, it’s not been repeated in the body of the piece (which it’d need to be if it were to be cited*) or a source lifted up so that’s it’d be directly apparent. Sources cited though lead to WHO sources saying both lack of access and refusal to vaccinate are responsible for the drop in Europe. So it’s a mix. I suspect if you keep going both will be backed up further down; WHO tends to build off data, not anecdote. You’ll have to check this yourself, and what the balance of the two is and where they apply (are the deaths are in areas where the issue is refusal not access?), etc., as I simply haven’t time. It’s your interest after all, not mine!

      (* It’s very rare to cite stuff that’s in title itself outside of academia as you’d have to put a citation mark in the title itself and footnote it somewhere. See also the second paragraph of my footnotes. How headers (and ledes) are written is often a nuisance.)

  • Grant, you say that WHO tends to build off data, not anecdote.

    WHO says vaccination rates are at record levels.

    I work in the health system. In my experience most parents who don’t vaccinate their children are not antivaxxers, but just complacent, or ‘just don’t care’ parents, or can’t afford to get to the doctors. My understanding is that antivaxxers make up less than one percent of those who don’t vaccinate their children.

    Regarding the credibility of the article. The body of the story lacks credibility of two counts. The story does not support the headline claim relating to antivaxxers being the cause of the deaths, or Europe itself. It relies on anecdote and unsubstantiated opinion and not facts.

    • Hi,

      Do you have idea just how incredibly tedious it is to see for the (seemingly) several-hundredth time a commenter write and from the get-go be almost certain what they are ‘really’ after has nothing to do with the article and topic at hand, that they are pushing a barrow, but that you’re ‘supposed’ to give them a chance? (for ‘give them a chance’ read: ‘give them enough rope to hang themselves’.)

      See Aston’s comment. If you don’t get it, search ‘sealioning’.

      Please take this somewhere else. At the very least it will want to be a vaccine-related post.

      Pro-tip: around here you’re supposed to interact with the author’s topic. My piece isn’t about any of the articles used in the study in and of themselves, never mind vaccines; the study was an opportunity to think about what we might do to tackle inaccurate science and health news.

      So, one long closing paragraph and we’re done. The title does not say ‘37 children of parents who refused the MMR vaccine died of measles this year’. It refers to what caused the outbreaks. I think that’s obvious to anyone. re ‘complacency’ and ‘lack of access’, many would suggest in all but a tiny number of cases it’s (a hell of) a western privilege to not take make the effort. One post I wrote has a photo of grandmother who carried two of her children 5 hours to get them vaccinated because they were missed when their village was vaccinated. BTW you can’t generalise to elsewhere (as you have), and ‘your experience’ = anecdote. (And you’re doing it to the wrong person; I’ve covered the NZ survey on this.) Forgiving is/seems/feels very kind – except when there are poor outcomes. Your final paragraph tries to multiply one claim to two by pointing at two outcomes of the same thing; that doesn’t work and doesn’t still include the article itself. But all of this is well off-topic, and pushing a barrow.

      OK, we’re done! No more off-topic, please. Any efforts to will likely to be removed.

  • Grant, my apologies. I too thought your piece was about thinking about what we might do to tackle inaccurate science and health news.

    I took you at your word and looked at the study & articles. You’d highlighted no surprise to read that food and nutrition articles did especially badly, “only 5 out of 18 articles received a positive credibility rating.” And then commented that “The strongest category was ‘disease and disease treatment’.”

    So I looked at the articles.

    The one I highlighted (there are others as well) demonstrated the study you were using wasn’t very reliable.

    Nevertheless, you are quite right. These inaccurate studies should never get published in the first place. Then we wouldn’t have to worry about what’s in social media!

    Best!

    Rob

    • Looks like a non-apology that “tries again”. I wrote, “My piece isn’t about any of the articles used in the study in and of themselves, never mind vaccines; the study was an opportunity to think about what we might do to tackle inaccurate science and health news.” – but you prefer to ‘overlook’ that, and drag on…

      Please just move on.

  • I agree wholeheartedly. Too often a juicily headlined release is published without, apparently, any reasoned, informed oversight. In an ideal world editors would take the line you suggest: actively access informed advoice. Sadly too many in the “news” space don’t know what they don’t know.
    The responses were a fine example of one part of an article that, to me, wasn’t the message being discussed ad nauseum.

    • @Anne

      I assume this is addressed to me, and is about my article not previous comments. (Sorry – it’s just not clear!)

      Headlines are a nuisance, as I noted in the Footnotes. In some ways they’re almost a separate issue or a separable element from the body content.

      My usual advice to people is to discard headline (and ledes), and try read the piece as if you’ve never seen them. It’s not easy to do though, as they tend to set a context that you read the piece in.

      When they are written by a (sub-)editor there are conflicting goals. On one hand they aim to stop people skimming other reading options and dive into that article. There’s some science behind this, too. People don’t read pages, they skim headlines. One ‘trick’ website designers use (and I presume newspaper layout people too) is to defocus the page and view it – you’ll see better how a person will skim the key features and where their eyes will track. Headline writers will try drop in words or phrases that gets people to stop skimming, which is a blessing and a curse.

      (The case of press release services is different; there headlines are the author’s.)

      People often ‘share’ on social media on the basis of not much more than the headline, too, which is unhelpful!

      The point about not knowing what you don’t know is a real catch-22 I think, and problematic. I like the notion of out-sourcing to specialists, but there’s money involved… I’m reminded of this when I see a few magazines calling out for a specialist to cover a particular topic. It‘s good to see; I’d love to see more of that, but it’s probably something magazines can do more readily that daily news. Short deadlines being one catch.

      In-house specialist editors are now rare, except in a few of the major outlets.

      I’ve got a lot empathy for editors, who are no doubt rushing around madly managing it all!

      So I guess the question is given the current state of things, how might you tackle this?

      It’s a tough question, but I like the thought that one editor can influence a lot of content, so they’re in a pivotal position to help. It’d want the newspapers as a whole to be on board in the way that they say re fake (political) news.

      But enough from me for one day (and I must get to other writing).

  • Some interesting points you’ve raised here, Grant. People might be interested to know that research a few years ago traced back the ‘hype’ in health stories and placed some of the blame at the feet of the institutes’ press releases: https://www.bmj.com/content/349/bmj.g7015

    It’s easy to blame journalists and editors, but the way the current media landscape works there are a lot of time-poor journalists who might be misled by press materials especially when the papers themselves are difficult to obtain, or the journalist isn’t equipped to analyse them.

    FWIW, we don’t do weekend training for editors, but we do hold two-day workshops for scientists engaging with the media 🙂

    • Hi Sarah-Jane,

      Thanks. I’d have to look back, but I think I covered that study when it came out. I did think of including it (and a dozen other aspects), but I try have blog posts focus on one thing. Also it seemed to me that editors are—potentially at least—in a position to help there too, e.g. picking up that a piece might need independent checks, etc.

      Just to be clear I am not blaming anyone (as I’ve tried to say several times), and I do appreciate how busy journalists are, etc. I just see that editors might have a role too, but I never see much discussion about their part of things when this topic is discussed. I’m just lifting up one aspect that doesn’t seem to get much time. My piece runs off that the study summary pointed at journalists; I thought “what about editors, too?”

      re

      the way the current media landscape works there are a lot of time-poor journalists who might be misled by press materials especially when the papers themselves are difficult to obtain, or the journalist isn’t equipped to analyse them.

      One solution might be to make better use of specialist writers? (I put this in as one of the bullet-point ideas.) There’s more to this, but I haven’t time right now. Besides I have to head off and find lunch! 🙂