SciBlogs

Posts Tagged bibliometrics

What science are Australians doing? Shaun Hendy Aug 02

3 Comments

By popular request our intern has put together a subject area tag cloud for Australia from their 2009 publications in the ISI database.  As she observed, Australia is poorly designed.  So much so that it is hard to squeeze Canberra’s tag cloud in between those of Sydney and Melbourne.  In fact, you’ll see in the map below it has drifted out to the southern coast of New South Wales in a most aesthetic manner.  It may be in fact that many of the residents of Canberra would be in favour of such a move …

australiasubjects09 copy

Medicine and medical sciences dominate the clouds over Sydney, Melbourne, Perth and Adelaide.  Hobart looks a lot like Wellington with its emphasis on oceanography and marine biology.  Canberra seems to have a broader focus albeit with a strong contribution from the physical sciences and engineering.  Brisbane stands out with a very strong signal from biochemistry.

Who are we collaborating with? Shaun Hendy Jul 30

1 Comment

Our talented intern from MIT has produced another tag cloud.  This time she has taken a look at who we collaborated with in 2008 based on our co-publication preferences in the ISI database.   The resulting map is shown below:

citycity08 copy

It’s clear we like working with Australians.  Those in Auckland, Palmerston North and Christchurch prefer to work with Sydneysiders, while those of us in Wellington prefer Victorians.  Hamiltonians have more exotic tastes with a clear preference for Californians.  And although Dunedin is often said to be the Edinburgh of the South, our southern scientists show a strong preference for London.

What science are we doing? Shaun Hendy Jul 26

4 Comments

What science are New Zealanders working on?  To help me answer this question, I have an intern from MIT here for her summer break.  Luckily for me, she hadn’t heard about Wellington’s winter.  (Not that our spring or summer are up to much either, although we can put on a decent autumn.)

She is a very bright cookie, and she mastered the ISI bibliometric database and our network analysis software in no time at all.  She is mainly studying the bibliometric performance of the Centres of Research Excellence (CoREs), but she has found time to look into other aspects of New Zealand’s bibliometric record.

Inspired by visualisations of the Twitter universe (such as trendsmap), last week we produced a “tag cloud” of subject areas Kiwis are publishing in across the main centres.  We picked the top five ISI subject areas in each of the main centres, scaling the text by how often it occurred (i.e. by the total volume of papers published in each subject area).  The 2009 cloud is shown below:

whitebackground09 copy

In Auckland and Dunedin, pharmacology dominates, presumably due to their university medical schools.  In Christchurch and Hamilton, environmental science dominates; in Wellington, it is marine biology; and in Palmerston North, it is veterinary science.

The map clearly shows New Zealand’s strong specialisation in health sciences, the environment, and food and agriculture.  As I pointed out in a previous post, the proportion of articles that Kiwis publish in the health sciences is similar to the rest of the world.  Where we differ from the international norm is the high priority we give agricultural and environmental science and the low priority we assign to the physical sciences.

National bibliometric report released Shaun Hendy Jul 05

1 Comment

MoRST has released its national bibliometric report covering papers published during the years 2002-7. The team from MoRST and the Royal Society that put the report together used the Scopus database.  The major findings include:

• The rate and impact of New Zealand publications has increased during the period 2002-2007. This is especially so in the Tertiary Education sector, which appears to be associated with changes to Tertiary Sector research funding.

I looked at the rate of publication in a previous post (New Zealand’s recent bibliometric productivity) and have recently had a paper published in the New Zealand Association of Scientists’ journal, the New Zealand Science Review, “New Zealand’s bibliometric record in research and development: 1990-2008” [1], available here.

The MoRST report reaches similar conclusions to mine, except that I was not as confident that the introduction of the Performance Based Research Fund for the universities has driven the change in citation impact, as it seems to have occurred in the CRIs as well.

• While the impact of New Zealand publications is generally average for an OECD nation, there are certain disciplines (especially in the medical sciences) where New Zealand research has a higher than average impact. This is the same as in previous bibliometric findings.

Here are the top five subjects by citation impact relative to the OECD according to the report:

Nat Report - top five
This type of calculation is a bit more difficult for me to do, because I have access to a different type of data set than the researchers at MoRST and the Royal Society.  Nonetheless, you can do something similar using the Scimago website.  For instance, if you are pleasantly surprised that Physics and Astronomy make the top five, you can use Scimago to see where New Zealand ranks in Physics and Astronomy by sorting by citations per document in a country comparison of papers.  As you can see below, New Zealand ranked ninth in citations amongst countries which publish more than 100 per year for Physics and Astronomy papers published between 1996-2008:

image
It’s always nice to be ahead of Australia!

• New Zealand is a cost effective place to do research. It has a comparatively high rate of publication per dollar of R&D expenditure.

This is also consistent with my findings on patents, where New Zealand appears to produce more patents per dollar than a number of other countries.  The figure below (taken from the report) shows that Kiwis are very cost effective indeed, although productivity per researcher FTE is middle of the road.   Bibliometric efficiency

My data shows that neither of these measures has changed much in New Zealand over the last twenty years.  Productivity (in papers per FTE) and publications per dollar have remained static.  As I suggest here and in my NZ Science Review article, the national increase in publication rates has been driven by increases in researcher FTE in the tertiary sector.

There is plenty of other material in the report to talk about, including some nice network diagrams illustrating collaborations between institutions.  Also, the authors have made some institutional comparisons of citation impact.  I will comment on that in a future post.

[1] S. C. Hendy “New Zealand’s bibliometric record in research and development: 1990-2008”, New Zealand Science Review 67, 56-59 (2010).

CRI bibliometric performance: Part III Shaun Hendy Feb 19

1 Comment

Last week, John Key signalled in a speech to Parliament that there would be changes to the way the Crown Research Institutes are funded.  Indeed, the debate over CRI funding has continued pretty much unabated since they were created.  In earlier posts, we looked at the growth in the total bibliometric output of the CRIs and at the increase in their citation impact relative to the rest of New Zealand.  In this post, I will look at the relationship between CRI funding and bibliometric output.  The data suggest to me that the growth in bibliometric output has been driven by the development of new revenue sources.

CRI total revenueFirst, I want to look at CRI revenues since 1994.  It is clear that CRI revenue has increased by about 30% over this period, once adjusted for inflation (figures are given in 2008 $).  Not all CRIs seem report their levels of public good science funding (or PGSF, which I will define here as the level of FRST and capability funding), but for those that do (most), I also plot PGSF revenue after adjusting for inflation.  Note that the PGSF revenue, at least for those CRIs that report it, has remained static over this period.

This is especially interesting given statements made when the CRIs were established.  Here is Sir James Stewart, Chair of the CRI Implementation Steering Committee:

“The science staff surpluses are not an outcome of the restructuring, but in part stem from chronic underfunding of science … Science departments had carried too many people for the money available.”

So how have staffing levels changed at the CRIs?  Statistics NZ collects FTE data from the CRIs, assigning research FTEs to the categories of researcher, technician and support staff.  Here is how Statistics NZ defines the different categories:

Researchers
Researchers are defined as those staff engaged in the conception and/or creation of new knowledge/products; personnel involved in the planning or management of scientific and technical aspects of R&D projects, and software developers.

Technicians
Technicians are defined as staff engaged in technical tasks in support of R&D, normally under the direction and supervision of a researcher.

Other Supporting Staff
Other Supporting Staff are described as staff providing specific information acquisition and treatment (for example drafting, typing, maintaining libraries etc. or specific administrative support such as bookkeeping, personnel services etc.)

CRI staff ratiosThe COMU website reports that just over 80% of CRI staff were involved in research in 2008.  On the right, the plot shows how the numbers of these research staff in each of the Stats NZ categories have changed according to the Stats NZ R&D survey.  (Note – in an earlier post, I reported on the numbers of researchers at CRIs, but there I used government sector researchers as a proxy, as not all the CRI data has been published.  The data on the right is the actual CRI data kindly supplied to me by MoRST.)  From the plot, we see that research staff FTEs have steadily increased at the expense of technical and support staff.  The decline in support staff since the mid-1990s is particularly dramatic.  This is something that has been very noticeable to me during my time as a CRI scientist.

CRI publications per dollar Now let’s look at how the revenues above scale with staff FTE and bibliometric output.  In the plot on the left, I give the total revenue (in 2008 $) per Researcher FTE (not research staff). This has remained relatively stable since 1994, fluctuating at around $400k per Researcher FTE.  On the other hand, revenue per paper published declined sharply in the 1990s, but then stabilised at roughly $500k per paper over the last decade. Of course, a good fraction of the research conducted in the CRIs will not lead to a publication, so this number does not reflect the true cost of a publication.

As we have seen, the CRIs’ bibliometric output has risen since their creation, and their citation impact has grown faster than the rest of New Zealand.  It also seems that they have become much less dependent on PGSF funding since they were created, with total revenue growing by 30% while PGSF revenue remained static.  Researcher FTE levels have risen, albeit at the expense of support and technical staff (although this may be typical of many businesses?), while the revenue per researcher FTE has remained static. Thus, the generation of revenue from non-PGSF sources, has led to increases in researcher staffing levels, which has in turn lifted the bibliometric output of the CRIs. To go any further, we will need to look more closely at the performance of individual CRIs.

CRI bibliometric performance: Part II Shaun Hendy Feb 10

9 Comments

In a post a few weeks ago, I looked at the total published output of the CRIs from 1993. Now I want to look at the citations to CRI papers. I will use two citation measures. The first is a two year impact factor, which is a measure that is often used to rank journals. The impact factor of a CRI in 2008, for example, is the average number of citations in 2008 for papers published by authors at that CRI in 2006 and 2007. The second measure I will use is a 5-year impact factor i.e.  the average number of citations to papers in 2008 that were published between 2003-2007 is the 2008 5-year impact factor.

Now, the analysis I am going to give below is somewhat naive. I should really be breaking down the citations by subject area (as pointed by Crikey Creek’s Daniel Collins in a comment last year). This is important because rates of citations differ considerably between disciplines – unfortunately I haven’t had the time to do this, except in a few special cases such as my own Institute. Thus, differences in impact factor between Institutes will depend on the areas in which they work. Changes in that difference over time may reflect changes in focus within Institutes, rather than changes in impact of the research conducted.

Why do citation rates differ between disciplines? At least part of the difference comes from the degree of empiricism within a discipline. Medical science frequently makes use of the aggregation of meta-data from many studies, some of which may be too small to have statistical significance on their own. So if your small study suggests that  smoking is a risk factor for diabetes, it will be important to cite as many other studies of smoking and diabetes as possible to give your reader context. Mathematics on the other hand relies on mathematical proof. To prove the Reimann hypothesis, you may only need to cite a handful of papers that contain results you rely on in your proof. You hardly need to cite every paper on the Reimann hypothesis that has appeared in print. Not surprisingly, journals in medical science typically have much higher impact factors that mathematics journals.

CRI Impact vs NZ On to the results. Firstly I have plotted the CRI (2 year) impact factor from 1995 to 2008 (on the right) against the New Zealand impact factor as calculated from the Thompson Reuters database. Firstly, we note that both data series show large increases over this time period. However, in 1995 the CRIs trail New Zealand as a whole, whereas in 2008 the CRIs lead New Zealand. The data is sufficiently noisy that one can’t to assert that the CRIs are significantly different from the rest of the country with much confidence however.

CRI 5yr Impact However, with the 5-year impact factor, the trend seems clearer: the 5-year impact factor of the CRIs is below those of New Zealand as a whole at the end of the 1990s, but by the mid 2000s it surpasses those of the rest of the country. As I mentioned above, there could be a number of explanations for this. CRI citations per paper have grown faster than New Zealand as a while. For example, I wonder if this could reflect a diversification of research activities at Universities, where disciplines with lower impact factors have started publishing more, perhaps as a result of the Performance Based Research Fund.

Unfortunately, without breaking down citations by discipline we can’t really tell whether this does reflect an increase in relative impact by CRI researchers. However, the data does suggest that this would be a worthwhile exercise: why has CRI impact surpassed that of the rest of New Zealand in the last decade?

CRI bibliometric performance: Part III

CRI bibliometric performance: Part I Shaun Hendy Jan 26

5 Comments

In a post last year, I looked at New Zealand’s bibliometric productivity in the university and government research sectors using data from the SCImago bibliometric site.  Over the next few weeks, I will report on some further bibliometric analyses using the Thompson Reuters Web of Science.  While providing substantially the same New Zealand-wide results as SCImago, the Web of Science database also allows me to break down publication data by research institute (and by individual author if needed).  Unfortunately, it is not freely accessible – I have institutional access through Victoria University of Wellington.

I’ll start this series of posts by looking at the total published outputs of the Crown Research Institutes (CRIs).  The CRIs were established in 1992 by scientists from the Department of Scientific and Industrial Research (DSIR), the research division of the then Ministry of Agriculture and Fisheries, and the New Zealand Forestry Service.  Shortly thereafter, a significant portion of the Crown funding for science became contestable through the Public Good Science Fund, open to the CRIs, universities, and businesses or other organisations conducting research and development.

At the time, the restructuring of government science into the CRIs was highly controversial‘Is New Zealand shooting itself in the brain?’ wrote New Scientist magazine.  ‘A small country does something like this at its peril’ said John Stocker, chief executive of Australia’s main research organisation, the CSIRO*.  The DSIR had given the world Marlborough Sauvignon Blanc, earthquake resistant lead-rubber bearings for building foundations and high-temperature superconductors, yet the Government of the day thought that the new Institutes would be better placed to contribute to New Zealand’s economic growth.

Almost two decades later, our new Government is wondering how the experiment went.  While the government scrutinises CRI balance sheets closely, other aspects of CRI performance receive very little attention.  This is surprising, since the reason the crown owns such research institutes has nothing to do with their balance sheets at all.    CRI total publications

Here I will look at how the CRIs have performed bibliometrically, starting with their total published output from the year following their establishment. The figure on the left shows the number of papers in the Web of Science database published by scientists at the CRIs since 1993. It can be seen that the annual number of publications doubled from 600 in 1993 to 1200 in 1997, a level where it has remained to the present.  The increase in output from 1993 to 1997 was substantial, but how was it achieved?

CRI productivityThe next figure shows the total researcher FTEs in the CRI sector from 1994-2006, and the corresponding productivity (in papers per FTE) over the same time period.  Researcher FTEs increased from 1996 to 2002, but have then declined by 20% since their peak in 2002.  Note that the productivity of researchers, in papers per FTE, remains relatively static over the period in question.  This largely reflects the New Zealand situation as a whole, where productivity has remained steady, and changes in levels of published outputs have been driven by changes in FTEs.

(Update: I received some better FTE data from Statistics NZ so the figure above was replaced on 18 March 2010. The data is similar to that shown in the original figure, but with the addition of the 2008 data, we see there was large jump in researcher FTE from 2006 to 2008, reversing the decline since 2002).

In my next post on the CRIs, I will look at how the number of citations of their papers have changed over time.  I will then look at how the CRIs have been able to lift their researcher FTEs from 1300 in 1994 to over 1800 in 2006.  After that I will move on to the Universities.

* Australia still has the CSIRO, and although some reforms have taken place since John Stocker made his comments, Australia has resisted introducing the direct competition between CSIRO and university scientists that has characterised our science system.

CRI bibliometric performance: Part II
CRI bibliometric performance: Part III

What’s ahead in 2010? Shaun Hendy Dec 29

No Comments

2010 is shaping up to be a defining year for New Zealand’s RS&T system. We will be hearing how the Government will set its RS&T priorities and what these priorities will be. The CRI task force will be reporting back, and we will find out how the Government is going to encourage R&D in the business sector. As I discussed in a post last month, the money spent by the business sector on R&D is strongly correlated with patenting.

What will I be blogging about in 2010? I have devoted quite a bit of time this year discussing data that we have extracted from the OECD patent database. There is still a lot of information to be mined from this database and I will be continuing to discuss this in the New Year. For instance, I will have more to say about inventor networks and how their structure changes with network size. I also want to look at some specific networks in more detail, comparing their size and structure between countries.

I will also follow up on some of my earlier posts on bibliometrics. It took me a while to get permission from Thompson Reuters to start publishing citation data, but this has now come through, so I will be looking at how the impact factors of New Zealand institutions have changed over the last 20 years. I also want to follow up with more detail on the surprisingly large co-author networks that exist within the New Zealand science community.

Of course, from time to time, I will blog on other matters that interest me throughout the year. I have been following the progress of some new types of collaborative research: mathematicians have been learning how to use the mass collaboration that blogging allows to prove theorems and solve original research problems. This is, after all, the reason that the web was created in the first place.

Collaboration, whether through blogs or other means, may be the key to New Zealand taking its own R&D to scale.

In the meantime, I hope you are all enjoying your holidays!

The University co-author network Shaun Hendy Oct 27

5 Comments

Uni coauthor networkIn an earlier post I looked at the 2008 CRI co-author network. Now let’s turn to the University network. Using the Thomp­son Reuters Web of Sci­ence again, I found 5116 publications in 2008 with authors from New Zealand universities. In total 13930 authors contributed to these papers. The network is shown on the right.

Again, a remarkably large fraction of authors belong to the giant component. In the 2008 CRI co-author network, 2325 of the of the 4496 authors belonged to the largest connected component. Here, 9771 of the 13930 authors belong to the largest component – that’s a remarkable 70%.

We can make some other comparisons between the CRI  and the university networks. In the university network, on average each author has 8.4 collaborators; in the CRI network, each author has 5.1 collaborators. Apparently, university authors are more collaborative.

Degree distribution However, just comparing the average numbers of co-authors is misleading. I’ve graphed the distribution of co-author numbers for the universities and the CRIs on the left i.e. the proportion of authors with certain numbers of co-authors. From the graph it’s apparent that the difference between the university and CRI networks lie in the tails of the distributions. There are a number of university authors that participate in very large collaborations. For instance, there are a dozen or so authors in the network whose only published work in 2008 was one with 343 co-authors. Big science!

It is probably not surprising that university researchers are more likely than those in a CRI to participate in very large overseas collaborations. This skews the average number of co-authors for university researchers relative to CRI researchers, making the mean number of co-authors larger.

New Zealand’s RS&T priorities Shaun Hendy Oct 23

2 Comments

MoRST have just released a discussion document which introduces a new structure for RS&T investment, aimed at allowing for greater strategic priority setting. Submissions are due by November 18th. Get it here.

What do you think of the feedback document and where would you put our rather modest science dollar?

World bibliometric output

While you are pondering this, here is where the rest of the world puts its intellectual grunt: the pie chart below shows the proportion of papers published by subject area over the last ten years. The physical and medical sciences account for two thirds of the world’s publications.

In contrast, here is where New Zealand puts its efforts:

NZ bibliometric output

Setting priorities is clearly nothing new for New Zealand – as the charts show, our science system strongly emphasises agricultural and environmental sciences at the expense of physical sciences. Have we got the balance right?