香港六合彩官网

Author: Deidre Hudson Reuss

Senior Marketing Manager

A recent white paper - - drew on the voices of over 6,600 researchers globally, across all career stages and disciplines. One of the largest surveys of its kind, it provided a lens through which we can better understand the state of research evaluation and assessment, as well as the varied experiences, perceptions and wishes of researchers navigating this environment.

Nearly 1,300 early career researchers (ECRs) from 105 countries completed the survey, and this article focuses on their views on evaluation and the future of research evaluation.  

These ECRs represented many disciplines and the research they undertake ranges from wholly fundamental through to entirely applied research, but most (81%) do both fundamental and applied research.

ECR involvement in research evaluation

Nine out of 10 ECRs responding to this survey have experience of being evaluated in the past 5 years, with 6 out of 10 being involved in research evaluation.

ECRs most frequently experienced research evaluation for grant applications (57%), but their research is commonly evaluated for new jobs, career advancement and in internal institutional assessment processes (43%).

Around a third also reported involvement in research evaluation through membership of panels for internal evaluation and staff recruitment, though they're much less frequently involved in regional or national research evaluation exercises (14%).

Key Findings: What do ECRs say about the evaluation of their research?

As a group, ECRs are generally positive about their experiences of being evaluated over the last 5 years – mirroring the results of the wider survey. However, they do identify significant concerns and desires for future change. Three key themes stand out in their responses:

> A desire to move beyond narrow publication metrics

ECRs expressed concerns that their experiences of evaluation encouraged or incentivised quantity (of publications) over quality, and that this is based on publication metrics to the exclusion of rounded qualitative assessments. For over half of the ECRs responding to the survey, evaluation is experienced as something that is entirely or mostly based on publication metrics (18% reported it as being entirely based on metrics, with 38% reporting that assessment was mostly based on metrics).

"The evaluation process is principally quantitative and bibliometric, there is no possibility for qualitative evaluation and impact of research."
- Early career researcher from Argentina

Only one in three of the responding ECRs reported evaluations that were approximately equally based on metrics and qualitative approaches.

> Incorporate wider contributions and impact

ECRs want to see their wider contributions to research recognised, alongside consideration of the impact of their work for society, the economy and organisations.

“The actual metrics in my country do not reflect the full work I have developed. The assessment is mainly based on published papers, but little is given to the effort put into supervising students, workshops and even review assessments of national or international (EU) panels. The results from the application of my research within local populations are also not taken into consideration.”
- Early career researcher from Portugal

Many types of research outputs used by ECRs are not included in evaluations. One in four said they had produced outputs like datasets, reports for non-academic audiences, and software code that weren't considered in their evaluations. Even highly measurable outputs like patents are often overlooked.

> Transparency and fairness

Transparency and fairness of the evaluation process is the third area of high priority for ECRs, for whom evaluations come at a critical and defining stage in their development. The survey data shows that ECRs whose research is evaluated through both metrics and qualitative inputs respond more positively about fairness and whether evaluation represents a good use of their time.  

“Ensuring that the assessment framework is fair, transparent, and inclusive of diverse forms of research output is essential.”
– Early career researcher from India

What do ECRs want the future of research assessment to look like?

Research assessment reform has been discussed at length within the academic community, through initiatives like the , the and the . While ECRs' views align with these reforming initiatives, 41% of responding ECRs didn't know if their institution had signed up to one of these initiatives. For those who knew that their institutions had signed up, nearly 7 out of 10 reported perceiving change as a result of this.

In an ideal research evaluation process, 54% of ECRs want an even mix of qualitative and metrics-based assessment. A significant number (just over 1 in 4) still feel that metrics should feature strongly or entirely in their ideal evaluation process.

Whilst publications remain the most highly weighted element, ECRs reported that their ideal evaluation is one that moves towards a greater weighting on their contributions to society (referred to in the survey as “the public good”), and wider contributions to research and research culture.

“It is essential to move beyond traditional metrics like impact factor to more comprehensive evaluations that include societal impact, interdisciplinary contributions, and real-world applications.”
– Early career researcher from India

What ECRs recognise is that qualitative elements, especially relating to societal impact, can be much more difficult and time-consuming to develop. For ECRs, there's an additional challenge that research impact can take years to materialise.

Embracing change in research assessment

The data presented in this white paper provides a strong evidence base to help understand early career researcher perspectives on existing and future evaluation practices.

Overall, ECRs are positive about the ways that they are currently evaluated. However, there are some notable concerns, alongside a desire to move towards a system which places greater weight on positive contributions to society, the wider research community, and research culture.

Key questions emerged from this research that particularly affect early career researchers: How can assessment practices better represent researchers' input and effort? What is the optimal balance between quantitative and qualitative indicators? How can contributions to society be embedded into evaluation frameworks rather than just highly regarded? And how can processes minimize administrative burden on researchers?

As an early signatory of DORA, 香港六合彩官网 recognises that traditional metrics can't capture the full spectrum of a researcher's contributions. Through initiatives like DORA and CoARA, the research community is making progress towards more holistic assessment. However, our survey findings highlight gaps between current practice and ECRs' preferences - pointing to continued work needed in research assessment reform.

Read the white paper  and find tools and resources for early career researchers here.

Also of interest:

• How do you determine the value of a journal? Challenging assumptions about journals with lower impact factors
• Publishing to a Collection: Impact for early career researchers

Don't miss the latest news and blogs, sign up to The Source Monthly Digest!