Winning research funding roulette can come down to chance

By Tim Dean
Wednesday, 28 September, 2011

Apparently Lady Luck has a seat on health and medical research funding review panels, with a new study finding that the peer review process for research funding involves a high degree of randomness in allocating funding.

Professor Nicholas Graves, from the Queensland University of Technology’s Institute for Health and Biomedical Innovation (IHBI), has found that this makes the current research funding system overly costly and somewhat random.

“The key problem is that the current process is overloaded with information and a reliable ranking based on peer review is difficult to achieve,” he said. “The quality of peer review is known to be variable at the best of times.

“The composition of peer review committees themselves is somewhat arbitrary, depending upon who has been invited to take part and who is available. This affects funding decisions because the personalities, preferences and knowledge of committee members will vary.

“In many cases in order to get funded an applicant will have to write a really good proposal and be lucky.”

Writing in the British Medical Journal, Professor Graves looked at 2705 grant proposals to the National Health and Medical Research Council (NHMRC) in 2009 and found that chance played a role in many of the funding decisions.

Each proposal sent to the NHMRC was assessed by one of 45 discipline-specific peer review panels consisting of between seven and 13 members. Each panel member scores the proposal, and the average of all panel members’ scores is tabulated with the proposal receiving funding if its average score crosses the funding threshold.

Professor Graves looked the individual proposals and grouped them into three categories depending on whether none, some or all the reviewers rated the proposal above the threshold score, and compared this to whether the proposal was actually awarded funding.

He then looked at the variability in the panel members’ scores, and how many of the proposals that were funded had been rejected by some panel members, and how many that were not funded were given the tick by some panel members.

He found that of the 23 per cent of proposals that eventually received funding, 59 per cent were sometimes not rated highly enough for funding by some on the review panel.

This effectively separates funding proposals into three groups: those that are a cut above and which are very likely to get funding; those that are sub-standard and are very unlikely to get funding; and a sizable group in the middle for which funding is allocated with a high degree of chance.

As a part of the study, Professor Graves also assessed the cost of the funding process, calculating that each proposal cost a considerable amount in man-hours and dollars.

“The current funding process is also costly and time consuming to participate in,” he said. “With the average grant application taking 22 days to write at a cost $17,744 each, a total of 180 years of research time was used up in 2009 to prepare applications at a cost of $50 million, most of which was borne by applicants.”

Professor Graves’ recommendation is that the funding system could be improved in a number of ways that would reduce cost and increase the likelihood of the best research being funded.

One suggestion is to impose a quota on how many proposals any individual researcher can submit, which would reduce the load on review panels. This might result in more risk-averse proposals being submitted, but Graves suggests the current system favours less risky proposals already anyway.

Another suggestion is to let researchers know the probability they had of seeing their proposal funded, with the hope that researchers that consistently fell below the threshold might refrain from resubmitting the proposal.

Graves also recommends simplifying the proposals, which are currently between 70 and 120 pages long, with only nine dedicated to a research plan.

Larger review panels were also found to have lower variability and lower chance in determining which proposals received funding.

Graves also suggests that funding could be allocated more based on past performance and research productivity using quantifiable evidence rather than being based on promises of future research that may never manifest.

He also suggests that more research ought to be conducted into how health and medical research is funded and how to improve the system.

“While it would cost money to conduct this research it's likely to be only a drop in the ocean compared with total research money allocated by research funding bodies in any one year," Professor Graves said.

Chair of the NHMRC Council Professor Michael Good said: “It would be interesting to think about other ways of allocating research funding, by assessing an applicant's actual research performance, including published papers, new policies or observable improvements in health, as opposed to promises of research to come.”

The paper was published in the British Medical Journal today.

Related News

Microgravity causes changes in gene expression rhythms

Astronauts exposed to microgravity experience changes to physiology, including immune...

NSW, Qld announce RSV immunisation programs for infants

The government-funded programs have been designed to ensure infants and young children at the...

Gut bacteria appear important for overcoming milk allergy

Gut bacteria are thought to help reduce allergic reactions to some foods, but little is known...


  • All content Copyright © 2024 Westwick-Farrow Pty Ltd