Several countries decide on search applications by lottery

Brussels / Bergen (Khrono): in September The British Academy informed They will for the first time test a lottery for the distribution of research funds. A lottery will be held for orders over £10,000, if they pass the initial quality check. They are not the first in the UK, the Natural Environment Research Council conducted a similar experiment.

Nor are British funders the first to test it lotteryor randomization. Tested in countries such as New Zealand, Austria, Germany and Switzerland. The debate is going on in many countries, in Denmark and Canada, it is being discussed to test similar systems. also in Sweden discussion goes.

It may also be related to Norway. as such Khrono wrote recentlyAnd the The Research Council is looking at a number of different changes regarding research funding. There are 12 suggestions on the table. One of them is the lottery among the applications that are equally rated at the top level.

“In the past there was a lot of skepticism about this, but I don’t experience the situation that way now,” said Marie Sundley Tveit, director of the research council for Khrono.

However, it is not completely uncontroversial. Professor Bertil Tunguden told Chrono that It is believed that lotteries can reduce the quality of applications and lead to dissatisfaction among researchers.

will remove bias

the magazine temper nature Recently, in a leadership position, he advocated that more people should try the lottery. They referred, among other things, to the Schweizerischer Nationalfonds zur Förderung der wissenschaftlichen Forschung (SNF), the largest research funder in Switzerland, which annually distributes nearly one billion Swiss francs.

The Swiss have gone further than anyone else. After a pilot project from 2018-2020, it became clear in 2021 that they will use the lottery as a public method.

When the Swiss started the pilot, the goal was to get rid of what’s called bias, or bias, when millions were distributed to various research projects.

According to Martina Hogg, Research Officer at SNF, it was also about ensuring greater application processing efficiency.

– More and more orders came, but the budget did not increase at the same pace. There were group meetings with long discussions, in which it was finally not possible to distinguish some of the proposals, she told Gronow via a screen from Switzerland.

vote

Hogg describes how they wanted to make sure that no one other than official objective standards were allowed to play a role in these long discussions. She says the quality of the discussions has not necessarily improved in line with the length of the meetings.

“The idea then was to try a lottery for a small percentage of the apps, where it was difficult to differentiate using the official criteria,” she says.

SNF started small, in the trial period it was only about Postdoctoral Mobility Scholarships, scholarships that are awarded to postdoctoral scholars who spend two years at a host institution abroad. Therefore, not all prizes are drawn by lottery. In the trial period, this was applied to about four percent. Since the end of the trial period, 5.1 percent of all applications for these mobility grants have been subject to randomization.

Hogg says some project applications have already been rejected in the first round as they are all independently evaluated by two different members of the panel, who give the projects scores from one to nine. A project that gets a score of three on two different assessments can actually be rejected, for example, here. The rest is then discussed in a physical meeting where committee members vote on each application, which determines which projects are directly funded and which go to the lottery.

paint spots

So how is the lottery done in practice?

“The problem with the electronic system is that you can’t be completely sure that it’s random, so we decided to actually do it,” Hogg says.

It describes a process in which, for example, they could have seven projects, three of which would be funded. The seven-name paper cut-outs are folded into small balls, which are then placed in a bowl. Then someone comes from another room and pulls out. Everything is recorded on video.

Many were skeptical at first, but that has changed, according to Hogg.

– It’s better now, people need time to see the benefits and the alternative, which is to discuss proposals for hours, has not been better, she says.

less controversial

Kristen Oxley researches peer review processes on her PhD project at the TIK Center for Technology, Innovation and Culture at the University of Oslo. She says the lottery isn’t necessarily a red cloth for those vying for research funding.

– It’s getting more and more controversial with lotteries or randomness as we call it in professional parlance. New Zealand was the first, and there it is used by both the Health Research Council and the Science, Technology and Innovation Council, which have included this as a methodology in some of their programs, Oxley says.

It also says that private institutions, such as the Volkswagen Foundation, use partial randomization when selecting research projects for funding and support.

– Those who have normalized the lottery to the fullest are the Swiss National Science Foundation. Oxley says they use this in all of their schemes as a so-called “tie breaker” between applications that would otherwise be exactly equal.

more radical

– So the word lottery in such a case does not mean that any application can win?

– No, in these cases it is not so radical, because it is about choosing between applications that are very similar in quality and level. This is fairly uncontroversial even among those who have been subjected to this type of choice, says Kristen Oxley.

Still, there are those who go so far as to let opportunity prevail when choosing which research projects to fund.

Volkswagen has made other moves that are perhaps more radical in terms of choice. You also have an Austrian science institution that uses randomization, Oxley says, and the same is true of the British Academy.

When using more radical forms of sweepstakes, it is about smaller assignments and in areas where you are looking for innovative ideas and projects that really push the boundaries.

Some research suggests that peer review in these cases works relatively conservatively. These panels often avoid risk. That’s why randomness is used to reduce such biases when it comes to finding truly innovative projects, says Kristen Oxley.

drawn from the pond

She explains that experiments with randomization are scientifically based, and that those who test the method also conduct research on how the method works. Volkswagen ran this as an experiment from 2017 to 2020/21, and the plan is to report on results during 2022. Twelve apps were selected by their peers, while 12 were randomly selected. The results of these projects are then evaluated against each other.

– Radical forms of randomness How is it implemented?

– Sweepstakes are not only used to differentiate other similar applications. In these cases, projects are eligible for a pool, which is then drawn through a lottery. This means that a decision is made between projects that may be slightly different in quality and level. But Oxley says this is still a solid project.

She explains that there aren’t too many examples where randomness has been used across the entire application stack.

“There are a small number of academic articles where this is offered as an alternative, but the vast majority see the need for a first round of quality assurance of applications before making a random selection among the strongest,” Oxley says.

Not among the best

Research in this area shows that peer review is well suited to distinguishing between good and bad applications. But to differentiate the best apps, peer review is not good.

– Now we are in a situation where success rates are dropping, fewer applicants get projects. Then, by and large, very good applications must be distinguished. It’s tough. A survey from the USA also showed that among the most powerful applications, there is less connection between peer review and how projects proceed afterwards. You might as well have a lottery, Oxley says.

– This is indisputable among the applicants?

– If these methods are tested, the results show that randomization is acceptable by the majority. But this applies to relatively small grants in areas where highly innovative projects are being sought. Here, research shows that peer review tends to favor safer projects.

Difficult multidisciplinary

“Many researchers today say they feel like they’re in a lottery when they apply for Research Council funding. So, it’s a good idea to draw the lottery?

Yes, as long as there is a guarantee of the quality of the applications selected to participate in the randomization. Also, many of the applications are interdisciplinary. Then it becomes difficult to make a fair comparison.

Some of the distinguished researchers we spoke to argue against radical randomness, and that more emphasis should be placed on historical and quality achievements. what do you think ?

– If a group is selected from the top 20 percent of the apps, then of course the ones with the highest quality will be chosen. That also includes history and past benefits, Oxley says.

Does not save work

She stressed that the lottery can to some extent undermine confidence in the selection system. But this is true to some extent as long as the choice is made between projects at the top level.

There is talk now of simplifying the system for search applications, with the goal that researchers spend less time writing requests and fewer resources being used to process requests. What do you think will happen here?

– Anyway, there is no argument for using randomness. There is less you will save administratively by running randomization compared to the current model. To some extent, time is spent rating powerful apps. But it’s not that this is what steals resources.