Evaluating Science Outreach?

By Elf Eldridge 24/09/2013

Given the audience of Sciblogs.co.nz, I doubt I’ll be going out on a limb by saying that science communication and outreach is important. The issue I’ve been grappling with recently is how effective, and important the outreach we actually do is.

Let me set the scene. In Wellington* we have a huge number (circa 70) of different organisations that are in some way (or have been) involved in science outreach and communication programmes. They cross the spectrum from non-profits to CRIs and Universities and community groups, and the list is still growing – please get in touch if you know of any omissions! Most of these operate independently from one another, most struggle daily for funding and people to continue their work, and all approach the issue of science outreach and communication in their own personal ways.

So, put yourself in the shoes of a potential sponsor for a moment (or a volunteer with time wanting to help out). With limited money, people and time, try and decide which one(s) should you support. Part of it will come down, of course, to personal interest, for example if you’re a Porirua-based company you might be more interested in supporting Porirua focussed programmes. Yet once you have whittled down the list by preference, often there are still tens of programmes to choose from. If your goal (like mine) is to promote general science literacy across the board – then most of them seem like great causes worthy of your support.

How would you weigh your choice? – Image via Wikimedia Commons

But what if you dig a little deeper? How could you compare the ‘efficacy’ of something like a Royal Society sponsored lecture that attracts 200 people to a bi-weekly meeting of parents and kids akin to what is currently happening at The Clinic at Ngaio school (check it out here – it looks awesome!)?

Here’s my list of factors so far (again it’s incomplete and suggestions are encouraged):

  • How many times has it occurred over the last 3 months? (n)
  • How many people does it attract (on average) per session? (p)
  • How much does it cost to run – to both the participants and organisers? (C)
  • How long does each session take? (h)
  • What is involved in each session? (is it simply a talk? Or is there creativity involved? Or hands-on demonstrations?) (Q)
  • What audience is it reaching? (d)

Taking these variables (and any others you feel are relevant) how would you choose to combine them to give some ‘measure’ of a particular activity. My first instinct is to combine them as  = (nphQ)/(Cd) **.

Personally, I would take d as the average decile value of the 5 schools closest to the location of the event (as a point to start from – this is hardly a robust metric!), Q as 1 for a talk, 2 for a hands on demo, 3 for an actual experiment and 4 for the creation and design of something. Finally I would estimate on a scale from 1-10 with 1 being a free event to arrange and 10 being something that requires attendance fees or large degrees of sponsorship to occur.  Interestingly this particular formulation scores community efforts that encourage face-time and hands on experimentation higher than more traditional outreach methods such as lectures (which is hardly surprising as that’s the way I constructed it to act!).

So would you agree? Or is this approach simply too coarse grained to have any meaningful comparative value? If so, how then would you choose what to support?

UPDATE: NZCER (NZ Council for Educational Research) has also collated  a bunch of their science education research here for anyone that’s interested in NZ specific publications. They can be a bit lengthy, but many of them contain some valuable insight into how, when and where we might be best placed in applying our science outreach/education efforts for the maximum social effect.

* the majority of my work has a strong Wellington focus simply because of the amount of time I have available to dedicate to outreach. I dearly hope that getting detailed information about one location might generate some insight about how to proceed in other areas of the country at some stage in the future.

** Yes I apologise for committing the cardinal sin of putting a formula in a blog post. It’s also laughably simplistic to expect an accurate comparison from such a simple exercise and there are plenty of ways it could be expanded. For instance the Q and h terms are linked and should suffer from diminishing returns. There’s also the issue of personal weighting of particular variables (my big issues are obviously demographic (d) and quality (Q)), but it may still be an interesting way to start considering this problem.

0 Responses to “Evaluating Science Outreach?”

    • Oh absolutely Rachel! Gathering feedback/info on attendees is absolutely key to studying efficacy atm the moment as well and it’s certainly something I dont do enough of (yet). Ultimately I would like to find a way to look at both for all these outreach programmes, and I think the quality factor (Q) that I talk about above would ultimately be dictated by how the participants respond.

      Talking of studying attendees – I’m always split between deciding whether to attempt to collect data on memory, perception, enjoyment, understanding or interest. It very quickly become extremely easy to end up spending more time asking the attendees questions than actually communicating science to them. My own personal experience is that people will essentially report on whatever they think I want to hear – rather than being honest about their experience. I’ve tried to come up with some positive and negative ‘control’ questions to ask for comparison, but haven’t had much sucess so far. That’s why I hope to work a little closer with some of the psychology researchers at VUW in the coming months.

  • There’s another point I should have made in this post – the reason I place a relatively low value on lectures/talks isn’t just the lack of interactivity which is often linked to information retention – but mostly because of the audience they attract. These are (from my observations alone) typically the sort of people that already come to science talks or watch science programmes – so while they’re hugely interesting and I love going along to them, they ignore the fact that they don’t add much to the general science knowledge in the wider community. They simply teach scientifically literate people a little more science!

  • I think you’re right, there, Elf – a lot of those who come to Cafe Scientifique events are already interested in science & want to find out more. We used to get some ‘walk ins’ but usually the audience was regulars. (Had a huge group for a CS in 2006 about fluoride, but most of them were one-offs.)