Thursday, November 19, 2026

Scientific peer review should rate quality and impact separately, study argues

I would strongly argue it is easier, more effective and more objective to assess relevance and quality of a research paper!

I would also guess, in the near future machine learning & AI will tremendously help to improve peer review of new research papers.

For example, the relevance of a research paper is clearly diminished if it addresses only a very tiny aspect or very technical detail of the subject. Unfortunately, the publish or perish paradigm implies the wrong incentive to publish often and in large numbers.

The quality of a paper can also be assessed. E.g. was anything important omitted or neglected, were the presumptions tested and correct, are the presented comparisons and related research very complete and correct and so forth.

Impact is much too speculative and nebulous!

From the abstract:
"Peer review serves as the cornerstone of scientific quality control. Yet, the current journal-centric system is hindered by long timelines, high publication costs, inconsistent review quality, systemic biases, and editorial gatekeeping.
Notably, the system relies on misaligned measures of impact that are tethered to journal branding and conflate scientific rigor (Quality) with perceived significance (Impact).
Here, we report findings from the Discovery Stack Pilot Study, which tested a scientist-designed, journal-independent peer review model. The Discovery Stack model integrates in-line reviewer comments to promote constructive feedback and separately evaluates scientific Quality and Impact using defined criteria.
To examine feasibility and effectiveness, manuscripts were reviewed in parallel with traditional journal review. A total of 162 reviews were completed, and survey data from 86 participants were analyzed.
The results showed that reviewers effectively evaluated Quality and Impact as separate dimensions, with Quality scores being more consistent across reviewers than Impact scores.
Participants strongly supported the core elements of the Discovery Stack model and expressed enthusiasm for its broader adoption to enhance transparency, efficiency, and value in peer review.
Future studies will explore integrating this model into a digital platform for reviewing and curating scientific discoveries to improve the production and dissemination of high-quality research."

Scientific peer review should rate quality and impact separately, study argues | Science | AAAS

No comments: