How do Text-driven Metrics score over Publication/Citation Metrics?
Pit Pichappan
Digital Information Research Labs
Ilford. UK & Chennai. India
pichappan@dirf.org
Abstract:
The text represents a disciplinary knowledge contribution and reflects the content and, in turn, the knowledge-building activities. Words with context are the signals of concepts in texts. Common and overused words represent shallow research, whereas unique words and phrases reflect novelty in content. Text is a better yardstick than other indicators for assessing the knowledge contained in scientific papers. Publication and citation indicators deploy bibliographic data that fails to portray scientific knowledge. Even the San Francisco Declaration on Research Assessment (DORA) has strongly criticised journal impact factors (JIFs), a sentiment that resonates with many in the academic community; yet, they continue to be extensively utilised by research administrators, assessors, and publishers in evaluating the performance of individual researchers and scientific journals. When knowledge is embedded in texts, and if the texts are tapped, it signals the research rather than the publication and citation numbers.
In this exercise, we provide a few mechanisms for text-based measurement and indicate the future directions for text metrics. In the current discussion, we will demonstrate the significance of a few text measures, such as Lexical diversity, Lexical density, Text uniqueness, and Phrase occurrences. Earlier keywords were used to map scientific concept relations and to study domain formation. However, keywords are minimum features that signal text ambiguity; instead, we advocate “keyphrase analysis”.
We present evidence of converting peer review text into scores using sentiment analysis. Our journal translates the peer review reports into metrics based on the review scale, review scores, and inter-reviewer consistency. This presentation underscores the immense potential of text-based systems as a superior alternative to publication and citation-dependent metrics, offering a promising future for research assessment.
References
- I very much enjoyed your presentation “Translating Peer Review Reports into Measurable Scores and Their Relationship with Citations: A Cloud-based Sentiment Analysis Approach”- personal mail from Henk Moed dated 11 December 2020.
- Henk F. Moed. Toward New Indicators of a Journal’s Manuscript Peer Review Process, Frontier in Research Metrics and Analytics, 02 August 2016
- Mehmeti, L., Maria Formia, E., and Lupo, E. (2024) Paradigm shifts in research assessment for scientific publishing: emerging models in a pluriverse perspectives, in Gray, C., Ciliotta Chehade, E., Hekkert, P., Forlano, L., Ciuccarelli, P., Lloyd, P. (eds.), DRS2024: Boston, 23–28 June, Boston, USA. https://doi.org/ 10.21606/drs.2024.849.
- Waltman L., Traag V.A. (2021). Use of the journal impact factor for assessing individual articles: Statistically flawed or not?. F1000 Research, 9, 366 Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain, R., Kerridge, S., Thelwall, M., Tinkler, J., Viney, I., Wouters, P., Hill, J. and Johnson, B. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. London: Sage