Novel approaches for evaluation of research impact

Research evaluation plays a vital role in the management of science and technology. This thesis is motivated by the fact that manipulation of citation data is becoming a common issue in scientific research evaluation. Thus, we aim to utilise multiple data resources, including scientometric and altmetric data, to develop evidence-based quantitative evaluating methods for research impact to increase the reliability of assessment results.

To achieve the research goals, we first examined various altmetric data sources and platforms through a comprehensive literature review and an empirical study to investigate the quality of altmetric data. We suggested ways to utilise altmetric data in research impact evaluation studies. Then, we developed an evidence-chain-based evaluation method for research impact. We explored the methods for evaluating the research impact of research publications. We analysed the impact generation cause-effect chains, which put together the academic and broader impacts (economy, society, culture, public policy or services, health, and more) of the research output as a final effect, and illustrated the cause-effect chains of the impact generation process across different logical stages. We collected a range of evidence data as indicators of the impact generation process that were carefully selected from the most representative database within their category, and quality was verified by small sample tests, clarifying the origins of data sources and the breadth and comprehensiveness of the database's publication coverage. We utilised the structural equation modelling approach to establish the research impact aggregation models. Finally, all observed publications' impact scores were calculated using the factor weights from disciplinary structural models for research impact.

This research offers several significant contributions to the evaluation of research impact. Firstly, it proposes an innovative quantitative method based on evidence chains, effectively addressing the limitations of traditional results-driven evaluation systems and yielding more reliable insights. Secondly, it pioneers a full-chain evaluation method that captures both academic and broader dimensions of impact by incorporating every stage of the impact generation process, providing a novel perspective on impact assessment. Thirdly, the study develops a new way of thinking that employs cause-effect chains to analyse research impact, improving attribution and consistency. Additionally, an empirical analysis of altmetrics platforms offers a detailed understanding of altmetric data quality, reliability, and disciplinary differences, serving as a valuable reference for future use of the altmetric data. Lastly, while recognising the limitations of altmetrics as direct tools for impact evaluation, this research highlights their role in deriving meaningful indicators that propose a fresh approach to their application in research impact evaluations.