Test your Science Literacy Skills

By Darcy Cowan 22/01/2013 31

Last week an interesting paper was published outlining the development of a test designed to evaluate science literacy. The basic idea of the test is to examine a number of different skills that are involved in evaluating scientific claims and facts.

I liked this idea so much I decided to put together an online version of the quiz that people could take and get instant feedback for. This could also be adapted for use in an educational environment as intended in the original paper.

To that end I need beta testers, people who would like to take the test and give me feed-back on how to improve it. A couple of things I need to mention at this point:
First, the test will ask for your email address in order to send a summary of how well you did on the test. Your email address will not be used for any other purpose and indeed I’ve set up the back-end so that the address will be deleted as soon as your email is sent – your private information will remain that way.
With this in mind – If you have feed-back about your performance, there is no way for me to tell which entry is yours unless you give me the time you submitted the form.

Secondly, I have used some custom html code in creating the form which doesn’t integrate seamlessly with the google drive document that runs most of the functionality. This means if you miss a required question (all of them are required FYI) than you will be re-directed to the original form of the test, this will look a bit different and will not contain the pictures (but will have urls you can use to get to them).
You can either continue with this version of the form or hit your browser’s Back button and fill out the questions you missed there.

Finally, If you have and comments or suggestions leave them here and I’ll see what I can do. Also I am actually fairly mediocre at html coding and such like so if there is anyone who wants to volunteer their services to upgrade the quiz I’m happy to share the load :).

Ok, here’s the link to get you started, have fun.

[Edit: Here’s a link to some initial analysis of the results]

Enhanced by Zemanta

Filed under: Sciblogs, Science Tagged: Scientific literacy

31 Responses to “Test your Science Literacy Skills”

  • 21 Is a little off, I think.

    The videogame enthusiast argued that “playing violent video games (e.g., Doom, Grand Theft Auto) does not cause increases in violent crimes as critics often claim.”

    The answer “The decreasing trend in violent crime rates may be caused by something other than violent video games” is a flaw with the argument that playing violent video games causes a decrease in violent crime, but that wasn’t the argument that was made.

    To me, none of the answers, even though some were true, is a flaw with his argument. The major flaw with the argument being made is that the graph shows when violent video games came out, but doesn’t actually show how many people were playing them. If the blogger wants to make the argument, that a video game came out isn’t relevant nearly as much as the numbers playing violent videogames. We can assume that’s increasing, but who knows?

    In 12, the lack of an oxford comma in the definition of tertiary gives a misleading definition of its meaning. And although I’m not a scientist, the idea that something only counts as a primary source if its published in a peer reviewed journal seems off. It might not be a good source if all you have is an unpublished test, but it’s still primary and not none of the above.

  • @Graeme, I think the the secondary cause subsumes your point. If the numbers are low then clearly there is another cause and if numbers are high then there could still be another cause.

    for Q12, I’m not sure what you are getting at about the comma, could you elaborate?
    -as for the second point, well it is a test of scientific literacy and that’s what it means to be a primary scientific source. So it’s correct by definition.

    thanks for the feedback

  • ah, I think I get it now. but there aren’t too many government published encyclopaedias…so I’m not sure how generally misleading this could be?

  • @Graeme, re-reading the question and answer for 21 again I think you are right that the answer should not be worded like that. It implies the argument is the violent video games cause a decrease in violence but this is not what is said in the background info.

    Better would be if the background info say the blogger claims violent video games cause violence to decrease. As it is stated it’s an implicit rather than explicit claim.

  • In Q3, we are told that the mean life expectancy for men and women are different and asked what we might do to increase our certainty that there is a difference. The correct answer, apparently, is to do a statistical test. But what kind of test? If we look at the expectancy data for men and then for women what should we look for? If we measure, say, the variance, or skew, or kurtosis of the distributions this wouldn’t change the fact that the means are different. Since we’ve already been told that the means are different what test should we doing that would increase our confidence that the means, are in fact different? I don’t get it.

    The answer for Q26 appears to be wrong. The journal has published a number of positive articles and no negative articles about a drug made by the company funding the journal. The correct answer, apparently, is that this journal is not credible because the journal only publishes positive studies. But there isn’t enough information given to determine this. We have no way of knowing whether negative or unflattering studies were even submitted to the journal. Since the articles were apparently peer-reviewed by credible experts that means the articles are credible. They may not necessarily be correct but that’s a separate issue from credibility (and a standard problem with drug studies). At best, the question is badly worded

  • Hey,

    Chiz I agree with you on 26 (one of the two I dropped…). But Q3 makes sense to me. The question might be better worded as “what might you do to ascertain the population mean is indeed higher for men”. The important distinction being the different between what our sample says and what’s going on out there in the world. (Good old t-test would be an obvious one here).

    • If you did it in the last few minutes it could have been while I was updating the spreadsheet. That may have cased issues.

      Couldn’t say at what point the problem happened could you? ie on first loading the form, between the first and second pages, on submit?

  • Completed the test, and 26 of 28. Questions 12 and 15 wrong, and that’s not a surprise to me.

    The detailed breakdown by reasoning type shows only one question missing (evaluating and distinguishing sources 4 of 5). I think it probably should be 0 of 1 for making a graph, or maybe 3 of 4 for interpreting graph information, both show as 100%. I suspect that’s a coding error.

  • OK, added question references to the summary. That should help people interpret the blow-by-blow, and see that everything is working right.

    In response to a commenter elsewhere (between two posts and two blogs it’s getting hard to remember what’s where) I’m considering adding demographic questions to the test.
    Any suggestions? (ie wording, I’m planing on science/non-science background questions but fitting that into the multi-choice format is a bit restricting.)

  • Yeah – that was it. And we do have a government funded encyclopaedia: http://www.teara.govt.nz/

    Wasn’t saying it was a problem with the test, but if you can write the answer so that no-one comes away thinking government-funding is a component of what counts as a tertiary source, that’s probably better.

  • Key for me is that where people get a question wrong it’s because they don’t know the answer, not because they had difficulty interpreting the question.

    For the two I got wrong:
    The first was relating to primary/secondary/tertiary. I answered primary, the answer was tertiary. I interpreted the question to be whether the study was a primary/tertiary source, not the newspaper article. I’m sort of OK with that – I know what is a primary or tertiary source, I thought about whether you meant the article or the study was primary/tertiary, I choose wrong. But it seems to me that the question could be less ambiguous. Problem is, a question that says “is the newspaper article a primary or tertiary source” is kind of giving the answer in the question.

    The other I got wrong was the graph relating to stress. I thought the graph showing breakouts for stress level 0-9 was a better representation than high stress/low stress. Again, I thought about both of these, the question did ask which was better to show high stress v’s low stress, so I was tempted to pick the graph with the two buckets (meets the question). But I thought the graph that shows correlation across all levels was a better representation, because if that graph wasn’t mostly linear you’d have question about the relationship – if for example those with very low stress had a higher score you might think this was a J curve rather than a correlation. Again, I feel like I got this wrong through question interpretation rather than misunderstanding the concept of the graphs – and I don’t think your aim is to ascertain how well people interpret the question wording…

    • Exactly right on the point of the test there Paul and that there’s a balance in wording the questions so that you are clear without giving things away. I noticed in the original paper that they changed one of the question about sources to omit the term “Peer review” because people knew that was good without really knowing why or what it was.

  • Oh, and on the question of when I did the test – it was last night, but I hadn’t been in my e-mail to look at the results.

    • On test timing, bugger, that means it was an existing flaw. Oh well. Have looked through it all again and it should be good, Now with 100% more demographic questions!!!

  • A “Scientific Literacy Test” ought to be more precise in the language selected and employed. I stopped after the first question. It is quite obvious that the answer you are looking for as the correct answer is option two; however, option two is as incorrect as the other three options. This is due to the misuse of the term essential. The gene is not essential (i.e., necessary); the gene is sufficient, and the mouse, as is, contains no other genes, or groups of genes, that are sufficiently capable to switch on the reproductive capacity. To assert that said gene is essential, you must assert that there are no other genes in the universe that are sufficient for the consequence.

    I would suggest scrapping whatever demographic data you have compiled, since it has been compiled in connection with a faulty test. I would then suggest that you make certain that you are scientifically literate prior to producing any new tests.

  • I received 24 out of 28. Not the best that I’ve ever done on a science test, but decent. However, I do have something to say about 1 or 2 of the test questions.

    Number 14 was a little confusing to me as, at first, I couldn’t tell if it was asking me if the researcher failed to do one of those listed things (or all), or if it was asking which one of those (or all) was an attribute that would not have increased the validity of the results. Which is why I picked “Randomly sampling NYC residents,” thinking that it would be better to also test people who don’t live in an urban area or possibly test people who live in other states. I hope I didn’t confuse anyone with how I stated that. I suppose I could have figured it out if I had worked on that one a little bit longer (in retrospect, it seems clearer), but I was thrown off initially.

    For number 21, I answered “The graph only shows data up to 2003. More current data are needed.” I was not under the impression that the game blogger was trying to prove that violent video games causes a decrease in violent crime, only trying to indicate that critics of violent video games are essentially barking up the wrong tree because violent crime started to go down around the time more violent video games were being published. I suppose the critics of violent video games could argue that the crime rate would have been even lower if the games had not been published, but that sounds more like conjecture than anything else. As a gamer myself, I saw that the data ended in 2003 and I know very well that more violent video games have been released in the last 10 years. That being said, it’s noteworthy that with the development of technology, the way violent video games depict blood and gore has also developed, compared to the days where games were more pixelated (and that could be significant). So I thought that the answer that I chose was correct, but I’m interested in your thoughts on this. Otherwise, it was a very well put together test. Thank you.

  • @Jim, interesting interpretation. I ask that you might show your reading comprehension skills in not attributing the test wording to me when it I have stated that the test is NOT my design. But thanks for your feedback.

  • @Christopher, if you click on the “initial analysis” link at the end of the post you’ll see I also had issues with Q14, as well as Q12 (and others are suggested all the time).

    As for Q21, I admit I must have glossed over the question the first time as I chose the “correct” answer. I has since been made clear to me that what you are saying is correct – the “correct” answer turns what could be an oblique implicit argument of the blogger into an explicit argument which doesn’t exist in the question. I still happen to think the answer works as “least wrong” but opinions may vary.

  • “I would then suggest that you make certain that you are scientifically literate prior to producing any new tests.”

    Amazing how rude people can be thanks to the anonymity of the blogosphere

    Darcy, While I agree that the questions need to be as precise and valid as possible, I would also suggest that in the wider world people are likely to come across information about science that isn’t perfectly precise or which may contain minor accuracies. If one has a good level of science knowledge one can still work out which is the “most right” argument.

    As Jim himself pointed out, the answer in the first question was “obvious” even if he found it imprecise

  • Michael, quite true. People are most often exposed to science in the news and through conversations (where they are free to quibble about word usage). They should be able to interpret what they see and here to get to the most reasonable conclusions of the information they have.

    I have also see a complaint that the test includes too many statistics questions and so must be confusing science with statistics. A possibly valid complaint but as many of the conclusions in science are arrived at and communicated using mathematics and statistics it also seems a bit beside the point.

    All that said – there is definitely room for improvement in the wording of the test and I am providing reasonable feedback to the authors in the hope that it might get better in future.

  • I was a little thrown by the use of “essential”, but still picked it as the “best” option. Certainly in real life, science isn’t always so obvious & clear cut, as has been pointed out. Part of the fun of it :)

    In any case, this is the point of testing the test, to iron these things out, right?

  • Nano, yes and no, I mean I have no control over what the authors do with the test. The initial testing in my mind was to ensure the mechanics of the online version worked correctly. Then it became clear that the questions could do with a tune-up. I contacted the authors and passed along feedback.

    What is done with that feedback isn’t in my hands, unless I create an independent version of the test.
    As the work in creating the test was no my own in the first place I feel a bit uncomfortable re-working it myself.

    After all that, I think I haven’t made clear the intent of the test (and nobody reads the paper) as an educational aid. So long as the questions are worded well enough that the test measures what the authors think it measures (and post-education increases in scores imply that it does at least partially do this) then that is the main thing.

    The test was not created as a novel internet quiz (which is what I’ve inadvertently turned it into).

    Well, it’s out there for better or worse and what happens to it and it’s effectiveness remains to be seen.

  • @Amy, appologies. I was alerted that the quiz no longer submits results (so no data for me either) about a month ago but I have been unable to determine the problem.
    If anyone is familiar with google forms, esp publishing custom scripts as web apps to collect the data please step forward.

  • I’ve just tried rolling back versions of the form and even the earliest working version does nothing when the “Continue” button is pushed. I have to assume Google has changed something that has made the code non-functional. Sorry.

Site Meter