A key question for any evaluation process is whether it actually changes anything. Do researchers engage with the feedback they receive — and do they revise their work as a result?
We have begun tracking this systematically across our portfolio. The results so far are available here:
Did Authors Respond to Unjournal Evaluations? (interactive table and analysis)
What we found so far
Based on manual coding of 57 tracked evaluations (last updated April 2026):
- 19 papers received a formal or informal author response to the evaluation
- 16 papers received a formal written response
- 5 papers show clear evidence of substantive updating in response to the evaluation
- 7 authors stated their intention to update
- 15 papers show at least some positive signal — evidence of updating, a stated intention, or minor revisions
The remaining 35 papers are either too recent for follow-up or pending assessment.
To supplement the manual coding, we also ran automated PDF comparison on a subset of 34 paper pairs, detecting line-level changes between pre- and post-evaluation versions. Six papers showed substantial revisions (more than 50 net line changes); one paper had over 3,000 net line changes post-evaluation — consistent with a major revision.
Ongoing work
This is an evolving project. We plan to continue extending coverage to more papers, refining the automated change-detection pipeline, and ultimately linking observed revisions to specific evaluator suggestions. We also intend to complement the automated and manual coding with further human “sentry” checks and structured discussion of the findings. The table is updated as new data comes in.
The methodology combines two sources: (a) manual assessment by Unjournal staff of author responses and paper revisions, and (b) automated PDF-comparison that detects line-level changes between before- and after-evaluation versions.