Ranking of library and information science researchers: Comparison of data sources for correlating citation data,and expert judgments

This paper studies the correlations between peer review and citation indicators when evaluating research quality in library and information science (LIS). Forty-two LIS experts provided judgments on a 5-point scale of the quality of research published by 101 scholars; the median rankings resulting from these judgments were then correlated with h-, g- and H-index values computed using three different sources of citation data: Web of Science (WoS), Scopus and Google Scholar (GS). The two variants of the basic h-index correlated more strongly with peer judgment than did the h-index itself; citation data from Scopus was more strongly correlated with the expert judgments than was data from GS, which in turn was more strongly correlated than data from WoS; correlations from a carefully cleaned version of GS data were little different from those obtained using swiftly gathered GS data; the indices from the citation databases resulted in broadly similar rankings of the LIS academics; GS disadvantaged researchers in bibliometrics compared to the other two citation database while WoS disadvantaged researchers in the more technical aspects of information retrieval; and experts from the UK and other European countries rated UK academics with higher scores than did experts from the USA.     

Google Scholar,Web of Science,and Scopus: A systematic comparison of citations in 252 subject categories

Despite citation counts from Google Scholar (GS), Web of Science (WoS), and Scopus being widely consulted by researchers and sometimes used in research evaluations, there is no recent or systematic evidence about the differences between them. In response, this paper investigates 2,448,055 citations to 2299 English-language highly-cited documents from 252 GS subject categories published in 2006, comparing GS, the WoS Core Collection, and Scopus. GS consistently found the largest percentage of citations across all areas (93%–96%), far ahead of Scopus (35%–77%) and WoS (27%–73%). GS found nearly all the WoS (95%) and Scopus (92%) citations. Most citations found only by GS were from non-journal sources (48%–65%), including theses, books, conference papers, and unpublished materials. Many were non-English (19%–38%), and they tended to be much less cited than citing sources that were also in Scopus or WoS. Despite the many unique GS citing sources, Spearman correlations between citation counts in GS and WoS or Scopus are high (0.78-0.99). They are lower in the Humanities, and lower between GS and WoS than between GS and Scopus. The results suggest that in all areas GS citation data is essentially a superset of WoS and Scopus, with substantial extra coverage.     

Using the Web for research evaluation: The Integrated Online Impact indicator

Previous research has shown that citation data from different types of Web sources can potentially be used for research evaluation. Here we introduce a new combined Integrated Online Impact (IOI) indicator. For a case study, we selected research articles published in the Journal of the American Society for Information Science & Technology (JASIST) and Scientometrics in 2003. We compared the citation counts from Web of Science (WoS) and Scopus with five online sources of citation data including Google Scholar, Google Books, Google Blogs, PowerPoint presentations and course reading lists. The mean and median IOI was nearly twice as high as both WoS and Scopus, confirming that online citations are sufficiently numerous to be useful for the impact assessment of research. We also found significant correlations between conventional and online impact indicators, confirming that both assess something similar in scholarly communication. Further analysis showed that the overall percentage for unique Google Scholar citations outside the WoS were 73% and 60% for the articles published in JASIST and Scientometrics, respectively. An important conclusion is that in subject areas where wider types of intellectual impact indicators outside the WoS and Scopus databases are needed for research evaluation, IOI can be used to help monitor research performance.     

Convergent validity of bibliometric Google Scholar data in the field of chemistry—Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere,using Google Scholar,Science Citation Index,Scopus, and Chemical Abstracts

Examining a comprehensive set of papers (n = 1837) that were accepted for publication by the journal Angewandte Chemie International Edition (one of the prime chemistry journals in the world) or rejected by the journal but then published elsewhere, this study tested the extent to which the use of the freely available database Google Scholar (GS) can be expected to yield valid citation counts in the field of chemistry. Analyses of citations for the set of papers returned by three fee-based databases – Science Citation Index, Scopus, and Chemical Abstracts – were compared to the analysis of citations found using GS data. Whereas the analyses using citations returned by the three fee-based databases show very similar results, the results of the analysis using GS citation data differed greatly from the findings using citations from the fee-based databases. Our study therefore supports, on the one hand, the convergent validity of citation analyses based on data from the fee-based databases and, on the other hand, the lack of convergent validity of the citation analysis based on the GS data.     

A systematic analysis of Hirsch-type indices for journals

A theoretical model of the dependence of Hirsch-type indices on the number of publications and the average citation rate is tested successfully on empirical samples of journal h-indices.     

Definitions of time series in citation analysis with special attention to the h-index

The structure of different types of time series in citation analysis is revealed, using an adapted form of the Frandsen–Rousseau notation. Special cases where this approach can be used include time series of impact factors and time series of h-indices, or h-type indices. This leads to a tool describing dynamic aspects of citation analysis. Time series of h-indices are calculated in some specific models.     

Is Google Scholar useful for the evaluation of non‐English scientific journals? The case of Chinese journals

This study determined how useful Google Scholar (GS) is for the evaluation of non‐English journals based on a sample of 150 Chinese journals listed in the Report on Chinese Academic Journals Evaluation of Research Center for Chinese Science Evaluation (2013–2014). This study investigated two disciplines: Library, Information & Documentation Science and Metallurgical Engineering & Technology. We collected data from GS and the Chongqing VIP database to evaluate GS as a citation database for Chinese journals on its resource coverage, journal ranking, and citation data. We found that GS covered 100% of the sample journals but indexed 22% more article records than the number of articles published. The ranking of Chinese journals by GS Metrics was not suitable to present a dependable ranking of Chinese journals. GS appeared suitable to provide an alternative source of Chinese citation data, even though there existed coverage problems, including article duplication and citation omission and potential duplication. The GS Metric average citation provided results highly correlated to traditional citation results, showing that it would be suitable for evaluating Chinese journals.     

Publication point indicators: A comparative case study of two publication point systems and citation impact in an interdisciplinary context

The paper presents comparative analyses of two publication point systems, The Norwegian and the in-house system from the interdisciplinary Danish Institute of International Studies (DIIS), used as case in the study for publications published 2006, and compares central citation-based indicators with novel publication point indicators (PPIs) that are formalized and exemplified. Two diachronic citation windows are applied: 2006-07 and 2006-08. Web of Science (WoS) as well as Google Scholar (GS) are applied to observe the cite delay and citedness for the different document types published by DIIS, journal articles, book chapters/conference papers and monographs. Journal Crown Indicator (JCI) calculations was based on WoS. Three PPIs are proposed: the Publication Point Ratio (PPR), which measures the sum of obtained publication points over the sum of the ideal points for the same set of documents; the Cumulated Publication Point Indicator (CPPI), which graphically illustrates the cumulated gain of obtained vs. ideal points, both seen as vectors; and the normalized Cumulated Publication Point Index (nCPPI) that represents the cumulated gain of publication success as index values, either graphically or as one overall score for the institution under evaluation.The case study indicates that for smaller interdisciplinary research institutions the cite delay is substantial (2–3 years to obtain a citedness of 50%) when applying WoS for articles. Applying GS implies a shorter delay and much higher citedness for all document types. Statistical significant correlations were only found between WoS and GS and the two publication point systems in between, respectively. The study demonstrates how the nCPPI can be applied to institutions as evaluation tools supplementary to JCI in various combinations, in particular when institutions include humanistic and social science disciplines.     

Citation impact analysis of top ranked computer science journals and their rankings

Citation based approaches, such as the impact factor and h-index, have been used to measure the influence or impact of journals for journal rankings. A survey of the related literature for different disciplines shows that the level of correlation between these citation based approaches is domain dependent. We analyze the correlation between the impact factors and h-indices of the top ranked computer science journals for five different subjects. Our results show that the correlation between these citation based approaches is very low. Since using a different approach can result in different journal rankings, we further combine the different results and then re-rank the journals using a combination method. These new ranking results can be used as a reference for researchers to choose their publication outlets.     

A comparative study of cross-domain research output and citations: Research impact cubes and binary citation frequencies

Across the various scientific domains, significant differences occur with respect to research publishing formats, frequencies and citing practices, the nature and organisation of research and the number and impact of a given domain's academic journals. Consequently, differences occur in the citations and h-indices of the researchers. This paper attempts to identify cross-domain differences using quantitative and qualitative measures. The study focuses on the relationships among citations, most-cited papers and h-indices across domains and for research group sizes. The analysis is based on the research output of approximately 10,000 researchers in Slovenia, of which we focus on 6536 researchers working in 284 research group programmes in 2008–2012.As comparative measures of cross-domain research output, we propose the research impact cube (RIC) representation and the analysis of most-cited papers, highest impact factors and citation distribution graphs (Lorenz curves). The analysis of Lotka's model resulted in the proposal of a binary citation frequencies (BCF) distribution model that describes well publishing frequencies. The results may be used as a model to measure, compare and evaluate fields of science on the global, national and research community level to streamline research policies and evaluate progress over a definite time period.     

Chinese-language articles are not biased in citations: Evidences from Chinese-English bilingual journals in Scopus and Web of Science

This paper examined the citation impact of Chinese- and English-language articles in Chinese-English bilingual journals indexed by Scopus and Web of Science (WoS). Two findings were obtained from comparative analysis: (1) Chinese-language articles were not biased in citations compared with English-language articles, since they received a large number of citations from Chinese scientists; (2) a Chinese-language community was found in Scopus, in which Chinese-language articles mainly received citations from Chinese-language articles, but it was not found in WoS whose coverage of Chinese-language articles is only one-tenth of Scopus. The findings suggest some implications for academic evaluation of journals including Chinese-language articles in Scopus and WoS.     

Missing author address information in Web of Science—An explorative study

Bibliometric analysis is increasingly used to evaluate and compare research performance across geographical regions. However, the problem of missing information from author addresses has not attracted sufficient attention from scholars and practitioners. This study probes the missing data problem in the three core journal citation databases of Web of Science (WoS). Our findings reveal that from 1900 to 2015 over one-fifth of the publications indexed in WoS have completely missing information from the address field. The magnitude of the problem varies greatly among time periods, citation databases, document types, and publishing languages. The problem is especially serious for research in the sciences and social sciences published before the early 1970s and remains significant for recent publications in the arts and humanities. Further examinations suggest that many records with completely missing address information do not represent scholarly research. Full-text scanning of a random sample reveals that about 40% of the missing address articles have some address information that is not indexed in WoS. This study also finds that the problem of partially missing address information for U.S. research has diminished dramatically since 1998. The paper ends by providing some discussion and tentative remedies.     

多源数据的期刊复合引文分析均值方法探析

[目的/意义] 鉴于Scopus、Web of Science(WoS)与Google Scholar(GS)3个单源数据库在计量学研究和应用中的局限,发展综合利用多源数据信息的复合型引文分析方法可为计量学分析提供一种补充思路。[方法/过程] 以期刊总被引次数这一底层参量为切入点,用图书情报学国际期刊2009-2014年数据为实证基础,尝试对多源数据复合引文算术均值、几何均值及调和均值方法进行比较讨论。[结果/结论] 结果发现,算术均值、几何均值和调和均值虽有差异,但亦有线性关联,应用中可选其一;累计算术均值、几何均值和调和均值与期刊数之间未证实存在布拉德福定律,但在期刊数量等同的三分区内,累计被引均值满足形如n²:n:1的经验分布;多源数据复合引文与单个数据库引文的期刊排名差异呈现"两端小、中间大"之现象;期刊的算术均值排名与GS排名结果更接近,几何均值排名则与Scopus相似度更高,而调和均值与WoS的排名差异最小。     

Citations to scientific articles: Its distribution and dependence on the article features

The citation counts are increasingly used to assess the impact on the scientific community of publications produced by a researcher, an institution or a country. There are many institutions that use bibliometric indicators to steer research policy and for hiring or promotion decisions. Given the importance that counting citations has today, the aim of the work presented here is to show how citations are distributed within a scientific area and determine the dependence of the citation count on the article features. All articles referenced in the Web of Science in 2004 for Biology & Biochemistry, Chemistry, Mathematics and Physics were considered.We show that the distribution of citations is well represented by a double exponential-Poisson law. There is a dependence of the mean citation rate on the number of co-authors, the number of addresses and the number of references, although this dependence is a little far from the linear behaviour. For the relation between the mean impact and the number of pages the dependence obtained was very low. For Biology & Biochemistry and Chemistry we found a linear behaviour between the mean citation per article and impact factor and for Mathematics and Physics the results obtained are near to the linear behaviour.     

专利引文分析法与共词分析法在揭示科学-技术知识关联方面的差异对比

揭示基础研究学科与技术发明领域之间存在知识关联,预见二者间可能的相互推动或启示作用。科学计量学和文献计量学提供的途径是对科研论文和专利文献进行共词分析或引文分析。共词分析面临科学概念与技术术语部分不对应、专利技术核心特征词难以确定、语种差异引发部分术语缺失等问题。专利引文分析直接观察知识主体间发生的知识交流活动,跳出繁琐的语境细节,通过追踪专利对论文的引证线索将无形的知识关联显性化。     

Journal coverage of the Emerging Sources Citation Index

The Emerging Sources Citation Index (ESCI) was created recently, in 2015, but few assessments of its journal coverage have been made. The present study tries to fill that gap by comparing its coverage with that of other international abstracting and indexing (A&I) databases. Using this measure, it is feasible to benchmark this index against the other citation indexes for acceptance criteria. We analysed 6,296 ESCI‐indexed journals, 8,889 Science Citation Index Expanded (SCIE), 3,258 Social Science Citation Index (SSCI), 1,784 Arts and Humanities Citation Index (AHCI), and 22,749 Scopus journals as indexed in July 2017 to determine their inclusion in 105 databases. We found that 19.3% of the ESCI journals are not covered by any other A&I databases, a high figure compared with only 0.5% SCIE, 0.3% SSCI, 0.3% AHCI, and 5.5% Scopus journals. This low coverage suggests that the selection criteria for ESCI journals are not consistent with the overall trend in the other classical citation indexes.     

基于“反向引用”视角的中国专利技术与世界科学关联分析

[目的/意义] 论文引用专利的“反向引用”也是科学与技术知识关联的重要体现。针对现有“反向引用”研究的不足,分析中国专利技术对世界科学研究的影响。[方法/过程] 搜集整理中国专利被世界科学论文引用的数据,利用文献计量学方法分析“反向引用”的引用频数、技术关联度、引用时滞及技术循环周期等指标,利用社会网络分析方法构建“反向引用”网络,探讨技术领域到科学领域的知识流动情况。[结果/结论] 中国专利技术对世界科学研究的影响力越来越大。科学与技术的双向互动在化学领域体现的最为明显,化学、工程、材料科学三个学科领域较多地吸收了专利知识,技术关联度较高。前1-4年的技术对当前科学影响最大,中国专利被科学文献引用的循环周期较短,平均为4.69年。     

A Comparative Citation Analysis of Web of Science,Scopus, and Google Scholar

This article presents the results of a comparative study of Web of Science (WoS), Scopus, and Google Scholar (GS) for a set of 15 business and economics journals. Citations from the three sources were analyzed to determine whether one source is better than another, or whether a new database such as Scopus, or a free database such as GS could replace WoS. The authors concluded that scholars might want to use alternative tools collectively to get a more complete picture of the scholarly impact of an article.     

Overlaps of multiple database retrieval and citation tracking in dementia care research: a methodological study

Objective:We aimed to determine overlaps and optimal combination of multiple database retrieval and citation tracking for evidence synthesis, based on a previously conducted scoping review on facilitators and barriers to implementing nurse-led interventions in dementia care.Methods:In our 2019 scoping review, we performed a comprehensive literature search in eight databases (CENTRAL, CINAHL, Embase, Emcare, MEDLINE, Ovid Nursing Database, PsycINFO, and Web of Science Core Collection) and used citation tracking. We retrospectively analyzed the coverage and overlap of 10,527 retrieved studies published between 2015 and 2019. To analyze database overlap, we used cross tables and multiple correspondence analysis (MCA).Results:Of the retrieved studies, 6,944 were duplicates and 3,583 were unique references. Using our search strategies, considerable overlaps can be found in some databases, such as between MEDLINE and Web of Science Core Collection or between CINAHL, Emcare, and PsycINFO. Searching MEDLINE, CINAHL, and Web of Science Core Collection and using citation tracking were necessary to retrieve all included studies of our scoping review.Conclusions:Our results can contribute to enhancing future search practice related to database selection in dementia care research. However, due to limited generalizability, researchers and librarians should carefully choose databases based on the research question. More research on optimal database retrieval in dementia care research is required for the development of methodological standards.     

Comparing alternatives to the Web of Science for coverage of the social sciences’ literature

The Web of Science is no longer the only database which offers citation indexing of the social sciences. Scopus, CSA Illumina and Google Scholar are new entrants in this market. The holdings and citation records of these four databases were assessed against two sets of data one drawn from the 2001 Research Assessment Exercise and the other from the International bibliography of the Social Sciences. Initially, CSA Illumina's coverage at journal title level appeared to be the most comprehensive. But when recall and average citation count was tested at article level and rankings extrapolated by submission frequency to individual journal titles, Scopus was ranked first. When issues of functionality, the quality of record processing and depth of coverage are taken into account, Scopus and Web of Science have a significant advantage over the other two databases. From this analysis, Scopus offers the best coverage from amongst these databases and could be used as an alternative to the Web of Science as a tool to evaluate the research impact in the social sciences.