most cited research article

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Environ Res Public Health

The Top 100 Most Cited Scientific Papers in the Public, Environmental & Occupational Health Category of Web of Science: A Bibliometric and Visualized Analysis

Vicenç hernández-gonzález.

1 Human Movement Research Group (RGHM), University of Lleida, Plaça de Víctor Siurana, 25003 Lleida, Spain

2 Physical Education and Sport Section, University of Lleida, Av. De l’Estudi General, 25001 Lleida, Spain

Josep Maria Carné-Torrent

Carme jové-deltell, Álvaro pano-rodríguez, joaquin reverter-masia, associated data.

The Web of Science (WoS) data can be accessed through theWoS’s official website: https://www.webofscience.com/wos/alldb/basic-search (accessed on 14 March 2022).

(1) Background: The main basis for the public recognition of the merits of scientists has always been the system of scientific publications and citations. Our goal is to identify and analyze the most cited articles in the Public, Environmental & Occupational Health category. (2) Methods: We searched the Web of Science for all articles published in the “Public, Environmental & Occupational Health” category up to March 2022 and selected the 100 most cited articles. We recorded the number of citations, the journal, the year of publication, quartile, impact factor, institution, country, authors, topic, type of publication and collaborations. (3) Results: 926,665 documents were analyzed. The top 100 had 401,620 citations. The journal with the most articles was the Journal of Clinical Epidemiology and the one with the highest number of citations was Medical Care. The year with the highest number of articles in the top 100 was 1998. The country with the highest percentage of publications was the USA and the most productive institution was Harvard. The most frequent keywords were bias, quality, and extension. The largest collaboration node was between the USA, Canada, Germany, Spain, Australia, France, and Sweden. (4) Conclusions: This bibliometric study on Public, Environmental & Occupational Health provides valuable information not only to identify topics of interest in the analyzed category, but also to identify the differences in the topics they study.

1. Introduction

Bibliometrics is a science that uses statistical and mathematical procedures to track the general trend of research in a specific field [ 1 ]. Various authors have targeted the participation of researchers in scientific activities, as well as differences and conditioning factors from the different fields of scientific knowledge [ 2 ]. These authors attribute different frequency and different publication practices between scientific disciplines [ 2 ]. The Web of Science (WoS) online database includes all important research papers and provides integrated analysis tools to produce representative figures, that is, it is the reference database of institutions, researchers and actors linked to science [ 3 , 4 , 5 ].

Within bibliometrics, citation analysis is one of the most used tools to assess the academic impact of an article in a specific area of knowledge [ 6 ]. The number of citations a publication receives does not necessarily reflect the quality of the research or the relevance of its authors [ 7 ], but it has been suggested that articles with the highest number of citations may have the ability to generate changes in practice, controversy, discussion and more research [ 6 , 8 , 9 ], or, as suggested by Zhu et al. [ 1 ], the number of citations can measure the article’s influence and merit. In addition, WoS search results could be exported to software for later analysis such as VOS-viewer [ 10 ], which could provide important information associated with collaboration networks between countries, institutions or authors.

Although there have been bibliometric analyses of articles in the field of food safety [ 11 , 12 ]; environmental health [ 13 , 14 ]; health promotion [ 15 , 16 ]; health education [ 17 ]; mental health [ 18 , 19 ]; sport health [ 20 ]; and occupational health [ 21 , 22 ], the entire category of Public, Environmental & Occupational Health has never been studied worldwide.

Few studies have a standardized measure of the wide range of dissemination activities in a scientific category that allows a detailed observation of production, collaboration and interrelation in a scientific field. No explorations have been performed based on quantitative methodologies aimed at building indicators on which to be able to empirically test the scientific productivity of the Public, Environmental & Occupational Health category.

To our knowledge, there is no study that bibliometrically analyzes high citation articles that evaluate the Public, Environmental & Occupational Health category. Therefore, this study aimed to identify and analyze the 100 most cited articles in the Public, Environmental & Occupational Health category to understand the historical perspective and promote discussion and scientific progress in this specialty.

2. Materials and Methods

2.1. search strategy and eligible criteria.

Bibliometric analysis was performed on 14 March 2022. Two independent researchers, who searched the Web of Science Core Collection (Clarivate Analytics), a research platform that provides a substantial bibliographic database through of Science Citation Index Expanded (SCIE) using the search category, identified articles. The search strategy was performed through the “Public, Environmental & Occupational Health” category. We refined the research by selecting original research articles and reviews. The 100 articles with the most citations were eligible for bibliometric analysis, arranged in descending order of citation count. Any disagreement between the reviewers was discussed between them to reach a final decision. Author in descending order according to the number of citations ordered these articles.

2.2. Data Extraction

Two authors independently retrieved information from all articles. Through the Web of Science, the 100 articles with the highest number of citations were selected and exported. Later, they were exported into an Excel document where the following were recorded: the number of citations, name of the journal, year of publication, first and last author and co-authors, total number of authors, geographical location, origin and associated institute, the title of the article, type of document (article or review), abstract and corresponding author. For the analysis of authors, all the authors who participated in the study were counted. In the bibliometric analysis by country, each country that participated in the study was taken into account and the citations received were counted. Citations received by a country more than once were not counted if several authors from different institutions but from the same country had participated in the same study. The number of articles per country was counted as long as there was an author from the country in the study. If the first author was affiliated with two institutions, then the first institute was selected for inclusion.

2.3. Statical Analysis

We used IBM SPSS Statistics for Windows, Version 27.0 (Armonk, NY, USA: IBM Corp.) for correlation analysis. Correlation was determined using Pearson’s correlation coefficient (r), and when p < 0.05, the difference was considered statistically significant. We used a popular bibliometric analysis tool, VOSviewer 1.6.18 software (CWTS, Leiden, The Netherlands) [ 23 ], for cooperative network identification and keyword co-occurrence analysis. In addition, it could generate visual maps of knowledge. We also used the MapChart program [ 24 ], a platform from which a personalized map of different regions of the world was created, using colors and descriptions.

The study flowchart is shown in Figure 1 , and included studies that were published from 1900 to 2022 for the Public, Environmental & Occupational Health category of Web of Science. The search topic, after applying the strategy, produced 926,665 documents. For the analysis of the study, only articles or review articles were taken into consideration, which led to the exclusion of 294,361 documents. Of the remaining 632,304, the 100 documents with the highest number of citations were considered for the study. A total of 632,204 documents were excluded.

Flowchart of study.

3.1. Publication Year, Citation and Bibliometric Analysis of the Keywords

The 100 most cited publications in the Public, Environmental & Occupational Health category were published between 1938 and 2020, of which 70% were published after 2018. We performed an analysis of publication trends by 6-year intervals based on a ranking of publication dates. Between 1998 and 2003, 29 documents were published, with the year 1998 ( n = 11) being the year of greatest production. There has been a visible improvement in the quantity of the data, since, of the 100 articles, before 1998 a total of 29 documents were published, making a big difference with the period 1998 to 2003, when 29 articles were published ( Figure 2 ).

Pattern of distribution of top-cited articles (number of articles per year).

The top 100 articles were cited 401,620 times in total, and the average total number of citations was 4016 citations (ranging from 1846 to 30,229). No significant correlation was found between the total number of citations and the age of the articles (r = −0.121, p = 0.229). The most cited article (30,229 citations) was “A new method of classifying prognostic co-morbidity in longitudinal-studies-development and validation” by Charlson et al. [ 25 ] published in the Journal of Chronic Diseases . Based on the number of publications in the 100 articles, and analyzing the citations per publication, 1998 was the most productive year with 11 articles (42,320 citations and an average of 3847 citations/article) in the top 100 list ( Table 1 ).

The top 100 articles with most total citations in Public, Environmental & Ocupational Health category.

The oldest study included in the list was published by Miles et al. [ 26 ] in 1938 entitled “The estimation of the bactericidal power of the blood”, with 3246 citations. The last study included was published in 2020 by Wang et al. [ 27 ], the paper entitled “Immediate Psychological Responses and Associated Factors during the Initial Stage of the 2019 Coronavirus Disease (COVID-19) Epidemic among the General Population in China”, with 2485 citations, published in the International Journal of Environmental Research and Public Health ( Table 1 ).

Eighty-six of the 100 publications were original research, and the remaining 14 were reviews. The average number of citations per article in the review works was 3285 citations/article compared to 4135 citations/article in the original works ( Table 1 ). The most common important keywords included quality of life, comorbidity, disease, cancer, clinical-trials, bias and epidemiology, and the keywords that appeared the most were “bias” (total link strength of 14), “quality” (total link strength of 11) and “extension” (total link strength of 10), which had a strong link with “epidemiology”, “metaanalysis” and “cancer” ( Figure 3 ).

The co-occurrence network of keywords. Note : The size of the nodes indicates the frequency of occurrence. The curves between the nodes represent their co-occurrence in the same publication. The smaller the distance between two nodes, the higher the number of co-occurrence of the two keywords.

3.2. Authors and Bibliometric Analysis of the Co-Authorship

A total of 487 authors contributed the 100 most cited. The number of authors in an article ranged from 1 to 26 (mean 5.53). Analysis of the 10 most productive authors based on their number of articles in the top 100, regardless of their authorship positions, showed that Ware, J.E., Altman, D.G. and Horan, T.C. were the authors with the highest number of articles.

Ware, J.E., from the USA, had a maximum of 46,062 citations with five articles listed and an h-index of 100. The average number of citations/article was 9212 citations. However, Altman, D.G. from England, published four papers, the total index of citations was 36,420 and the average per article was 9105; their h-index was 182. The third position is for the researcher Horan, T.C. from the USA, with four published documents, an h-index of 25, and with more than 13,500 total citations ( Table 2 ).

The top authors with the most articles in the top 100.

The total number of citations was not related to the number of authors (r = −0.118, p = 0.058). However, the average number of citations per article was associated with the number of authors (r = 0.210, p < 0.001).

There was low collaboration between most of the main authors, creating only one cooperation research network. Authors with a minimum of three papers per author were considered for analysis. Of the 487 authors, seven reached the threshold ( Figure 4 ). Altman, D.G. formed a collaborative network with five other researchers with a link strength of 15.

The author collaboration network. Note: The collaboration map of authors reflects the scientific research cooperation between them. The circle/node signifies the authors; size of the circle/node signifies the number of articles. The lines denote the authors’ collaboration strength, and each color signifies a cluster.

3.3. Countries, Institutions and Bibliometric Analysis of the Collaboration

A total of 26 countries published the 100 most cited articles in the Public, Environmental & Occupational Health category. Table 3 shows the twelve most productive countries, with the USA being the one that contributes the most, with 65 documents, followed by England with 21 and Canada with 17 articles. These three same countries obtain also the greatest number of citations. However, the country with the highest rate of citations per article is Italy, with an average of 5151 citations/article, followed by England with an average of 4724 citations/article.

The top countries with the most highly cited articles.

Two collaboration nodes were established. A larger one, involving seven countries and where the USA had the most active partnership (a liaison force of 45 and collaborated on 57 documents); its major research cooperators included Canada, Germany, Spain, Australia, France, Sweden. The other node was where England (with a link strength of 37 and 20 documents) had a strong collaboration with mainly European countries such as Denmark, The Netherlands and Switzerland. We found that Italy and Norway rarely cooperated with other countries in investigations. ( Figure 5 ).

The country collaboration network.

The world map revealed that the articles were mainly concentrated in North American and western Europe, and less so in Oceania. Specifically, the USA was the country with the highest production of documents, followed by England and Canada ( Figure 6 ).

The distribution map of the number of published articles worldwide for countries (MapChart).

In total, 228 institutions participated in the 100 articles. The number of institutions per article ranged between 1 and 21. The average institutional collaboration was 3.8 institutions/article. The article with 21 participating institutions was a review on toxic equivalency factors (TEFs) published in 1998 with 2592 citations. The World Health Organization, with eight articles included in this bibliometric analysis, was the institution with the greatest scientific representation ( Table 4 ). In four of the eight papers, it was included as the main institution of the study. The total number of citations was 20,339 and the average number of citations per article was 2542 citations. The second institution was the University of Harvard in the USA with six documents and one as the main institution. The total number of citations was more than 25,000 with an average of 4209 citations per article.

The top institutions with the most highly cited articles.

There was a strong and significant correlation between the number of institutions and authors (r = 0.848, p < 0.001). There was a negative correlation between the total citations and the number of participating institutions (r = −0.115, p = 0.286).

In the collaboration network analysis ( Figure 7 ), a minimum of three collaborations between institutions were established, 19 reached the threshold and three cooperation network nodes were formed. In the first of them, McMaster University cooperated with institutions such as Harvard University, University of Washington and the University of Toronto and collaborated on five articles. The University of Washington had a strong partnership and cooperation with the University of Minnesota, NCI, and Wake Forest University. Collaboration network analysis also highlighted the institutional collaboration network that the World Health Organization has with University of Toronto, Wisconsin University and US EPA, with more than seven documents shared.

The institution collaboration network.

3.4. Journal Analysis

The Web of Science “Public, Environmental & Occupational Health” category had 204 indexed journals, of which 35 made it to the top 100 most cited articles list. A total of 16 journals were in the first quartile (approximately 45%), 9 journals in the second quartile, 3 in the third quartile, and 5 were from Q4. Two journals were out of print or had changed their name. A total of 79% of the studies were published in high-impact journals (Q1 or Q2).

The IFs of the 35 journals ranged from 0.875, Malawi Medical Journal , to 59.769, MMWR Surveillance Summaries . There were up to 22 journals with an IF < 5.000, seven between 5.000 and 10.000, and four journals with an IF > 10.000.

Table 5 shows the top nine journals that published three or more articles.

The top journals that published the top 100 highly cited literature in Public, Environmental & Occupacional Health category.

Journal of Clinical Epidemiology was the most productive journal ( n = 15), followed by Medical Care and Statistics in Medicine ( n = 12). The top five journals published 54% of the articles and account for more than 59% of the total citations. The self-citation rate for the top nine journals ranged from 1.7% for the Bulletin of the World Health Organization to 10.3% for the Journal of Clinical Epidemiology . The journal with the highest number of citations was Medical Care ( n = 74,189) and its mean number of citations per article was 6182 citations/article.

4. Discussion

This is the first paper that analyzes the 100 most cited papers in the Public, Environmental & Occupational Health category of Web of Science. This article identifies the authors, journals, countries, institutions, etc., with the greatest impact in this category from the beginning of the 20th century to the present. The sample size was set at 100 manuscripts to provide a manageable and significant number of articles to be analyzed, in accordance with several published works [ 1 , 6 , 8 , 9 , 28 , 29 ].

The period of the greatest publication of articles starts in 1998; a clear upward trend in the production of works started during the period 1989–2010, but then it disappears in the last decade. A stochastic process is observed. The Mann–Kendall trend test ( Figure 3 ) revealed a significant positive trend towards a greater number of articles over the years starting in 1985 ( p = 0.055, Kendell’s Tauβ). Our results would be in line with those found by the authors of [ 6 , 29 , 30 ]. They contrasts with recent reviews, on other topics and specialties, in which most of the most cited papers were published earlier in the 1980s [ 31 ] or later, from the year 2000 [ 32 , 33 ]. The socioeconomic growth of recent years may be one of the causes of the advancement in scientific research, an evolution that the dissemination and communication of science has already been experiencing as an exponential change for some time. To understand these changes, it is necessary to know how science spreads. In the professional field, one of the main ways that the research community has to disseminate its work is the publication of scientific articles; however, on the other hand, they also use social networks and all Internet options (scientific forums, blogs, etc.). These tools are also protagonists in recent years, which encourage more dissemination of science and, therefore, more knowledge of what is published [ 34 ]. Some experts believe that studies that are more recent are cited today due to the advancement in scientific dissemination [ 10 ]. It may seem surprising that the studies with the highest number of citations are recent studies; among other factors, this could be due to the appearance of scientific journals in an electronic format, facilitating access and thus favoring circulation in the scientific community [ 6 ].

Some specialists consider that research goes further, suggesting the publication of an article ends when it is read and understood by a large part of society, that is, it is not enough just to publish, it is necessary for the audience to clearly understand its content and, thus, be able to cite it [ 35 ].

The keyword co-occurrence analysis found that the words “cancer”, “quality of life”, “comorbidity”, “epidemilogy” and “disease” had the highest frequency of co-occurrence in the research in the analyzed category. Our work reflects, in part, a growing trend in public health research. Studies on quality of life, comorbidity or cancer have been the focus of research in the scientific community and specifically within the Public, Environmental & Occupational Health category [ 36 , 37 ]. Performing a quick bibliographic search in WoS, these terms occupy the fourth, eighth and fourteenth position, respectively, with more records among the different categories.

Metadata from all documents were used to reveal the most productive authors and the most impactful sources. The high number of authors (487) contributing to the 100 articles, with more than an average of five authors per article, made it difficult to determine the individual contribution and, consequently, the role of each author [ 38 ]. As suggested by Bruni et al. [ 33 ], traditionally, in multi-author articles, the first position is occupied by the main contributor, while the last position is reserved for the supervisor. The authors with most impact in the studied category generally held relevant positions, either as main author or as supervisor. This is becoming more common due to the influence of experimental sciences, considering the same importance to the first and last author, based on the author/director relationship. This interpretation is known as the FLAE approach, an expression of first last author emphasis [ 39 ].

The h-index quantifies the research performance of individual scientists, incorporating both the number and visibility of publications [ 40 ]. In the work, we can see an unequal distribution of the h index among the authors of the 100 articles, where the number of citations that a scientific subcommunity grants to a manuscript is undoubtedly and directly related to the number of researchers that make up such a sub-community [ 41 ]. The analyzed category is a very broad field of knowledge; therefore, the number of citations of the articles will be very different depending on the topic analyzed.

As indicated by Jung et al. [ 42 ], in a context in which there is great interest in intensifying international collaboration within scientific practice, this paper proposes an approach on how to measure and visualize international collaborative work at the institutional level. The low collaboration observed between the different authors in our work contrasts with that found in studies such as by Zhu et al. [ 1 ] or Yu et al. [ 10 ]. The joint analysis of the collaboration indexes of the relationships between the different authors of the documents allows us to make a better interpretation of the structure of international scientific collaboration networks in the category of study [ 43 ]. One of the variables handled in our work was the possibility of identifying whether there was a high level of international and potentially multinational collaboration with other institutions that could affect the visibility of the research and the frequency of citations of a category [ 44 ]. This was not the case, but we have been able to map and identify the existing collaborations within the Public, Environmental & Occupational Health field, as well as the main citation sources.

The most relevant works were mainly in North America, specifically the USA and Canada, and Western Europe. Similarly, citation analysis showed this same trend in previous studies [ 45 , 46 ]. This trend can be explained by several reasons, first of all, by the cumulative geographical advantage, since citations originate more frequently from institutions located in the same country as the place of residence of the author [ 47 , 48 ]. Second, as suggested by Wang et al. [ 49 ], the USA can count on a broad scientific community and generous science funding policies. In fact, the most productive institutions in our study are geographically located in the USA and Europe. A third reason may be that larger universities provide greater opportunities for scientists to collaborate and work on similar topics, and co-authorship may lead to higher citation rates.

Most items originated from two major advanced economies: North America and Western Europe. These are undoubtedly economically developed continents with more access to early research and they can support medical research [ 29 ]. This can be seen in the data published by WHO in a report published in 2020, in which high-income countries spend a higher percentage of GDP on R&D in the health sector [ 50 ].

The cooperative network of research institutions can reveal the distribution of research forces in the field of Public, Environmental & Occupational Health. The USA has the most extensive cooperative relationships and prefers to cooperate with Canada and some European countries. England, with the second highest number of co-authored articles, prefers to work with other European countries. Our results would be in line with those found by Song et al. [ 51 ] in the Entrepreneurship research area. In the field of science, collaborative work, institutional and disciplinary structures face the challenge of a global context. This challenge has led to the creation of initiatives such as e-Science in the United Kingdom, which was announced as a global collaboration program in key areas of science, and the development of the next generation of infrastructure. These types of initiatives show that contemporary scientific practice is characterized by being very collaborative, multidisciplinary, global work with intensive data management [ 52 ].

According to the results, more than half of the classified articles were published in only five journals, collecting more than half of the total citations. These results demonstrate that a significant number of studies concentrated on a limited core of journals, in accordance with Bradford’s law [ 53 ]. As indicated by Highhouse et al. [ 54 ], authors tend to send their work to the most prestigious journals, attracted, according to Bruni et al. [ 33 ], due to the greater visibility in the search results, as well as the greater probability of being cited.

If we look at the quartiles of the journals, works mostly appears on first and second quartile journals. As stated by Torres-Salinas and Cabezas-Calvijo [ 55 ], publication in high-impact journals generates benefits, starting with the fact that a scientist who regularly publishes in these journals will be able to advance smoothly in his scientific career and will be recognized as an expert in his field. Other authors affirm that publication in high-impact journals helps to develop one’s own criteria, increases self-esteem, strengthens the confidence of the researcher, and feeds the desire to continue researching and publishing, in addition to guaranteeing quality through arbitration, such as peer review demonstrates [ 35 , 56 ].

There is no doubt that the use of the JIF as an evaluation measure generates debate, but today it is a useful way to measure the prestige and importance of scientific journals in the international system, as well as for their researchers [ 57 ]. Many authors have pointed out that the JIF has some limitations such as: (a) a built-in bias that favors American journals (in the case of our study, six of the nine journals in the ranking are published in the USA), (b) scoring highly variable IF between fields and specialties within fields, (c) vulnerability to inflation due to self-citation of journals, (d) vulnerability to inflation due to the publication of review articles and meta-analyses, and, finally, (e) an arbitrary citation window that penalizes some fields or specialties within them [ 54 , 57 ].

Finally, we wanted to also compare the JIF without the self-citations. In this sense, the level of self-citation of the analyzed journals was relatively low, with some exceptions. The abuse of self-citations is another element that can substantially affect the JIF. The self-citation rate in the presented list was low (8.1%) compared with other studies [ 33 , 58 , 59 ]. This is a bias that many platforms have been working on for years to solve [ 60 , 61 ].

5. Conclusions

The work allows the identification of relevant aspects in order to encourage scientific mapping in the Public, Environmental & Occupational Health category. The analysis can help the governance of specific areas or it can outline an institution’s research. The category analyzed has very varied topics; however, it allowed us to identify the most cited authors, institutions with greater visibility, and the most notable articles.

It has also made it possible to analyze the researchers who are forming national and international collaboration networks, as well as to identify the most collaborative authors and institutions.

Currently, the publication rate of American researchers is the highest in the category studied and its institutions are among the most productive. In addition, the collaborative network of countries, institutions and authors shows the influence of European and American countries in the Public, Environmental & Occupational Health category.

Keyword analysis was an effective method to identify interesting topics among researchers and mark research trend lines.

The results of this research open up new possibilities to identify new strategies and institutional policies that allow them to consolidate their research networks.

Although there has been an exponential growth in work, greater efforts are still required from both researchers and institutions.

In this article, valuable information is provided not only to identify topics of interest in the analyzed category, but also to identify the differences on topics studied between the areas that form the category.

Funding Statement

This research was funded by Human Movement Research Group: SGR-Cat 2021 grant number SGR 1463. Generalitat de Catalunya.

Author Contributions

Conceptualization, V.H.-G. and J.M.C.-T.; methodology, V.H.-G., C.J.-D. and J.M.C.-T.; formal analysis, V.H.-G. and J.R.-M.; investigation, J.M.C.-T., C.J.-D. and Á.P.-R.; data curation, V.H.-G. and J.M.C.-T.; writing—original draft preparation, V.H.-G. and J.R.-M.; writing—review and editing, V.H.-G., Á.P.-R. and J.R.-M.; visualization, J.M.C.-T. and Á.P.-R.; supervision, J.R.-M.; project administration, J.R.-M.; funding acquisition, V.H.-G., C.J.-D., Á.P.-R. and J.R.-M. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Data availability statement, conflicts of interest.

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Google Scholar Metrics

Google Scholar Metrics provide an easy way for authors to quickly gauge the visibility and influence of recent articles in scholarly publications. Scholar Metrics summarize recent citations to many publications, to help authors as they consider where to publish their new research.

To get started, you can browse the top 100 publications in several languages , ordered by their five-year h-index and h-median metrics. To see which articles in a publication were cited the most and who cited them, click on its h-index number to view the articles as well as the citations underlying the metrics.

You can also explore publications in research areas of your interest. To browse publications in a broad area of research, select one of the areas in the left column. For example: Engineering & Computer Science or Health & Medical Sciences .

To explore specific research areas, select one of the broad areas, click on the "Subcategories" link and then select one of the options. For example: Databases & Information Systems or Development Economics.

Browsing by research area is, as yet, available only for English publications. You can, of course, search for specific publications in all languages by words in their titles.

Scholar Metrics are currently based on our index as it was in July 2023 .

Available Metrics

The h-index of a publication is the largest number h such that at least h articles in that publication were cited at least h times each. For example, a publication with five articles cited by, respectively, 17, 9, 6, 3, and 2, has the h-index of 3.

The h-core of a publication is a set of top cited h articles from the publication. These are the articles that the h-index is based on. For example, the publication above has the h-core with three articles, those cited by 17, 9, and 6.

The h-median of a publication is the median of the citation counts in its h-core. For example, the h-median of the publication above is 9. The h-median is a measure of the distribution of citations to the articles in the h-core.

Finally, the h5-index , h5-core , and h5-median of a publication are, respectively, the h-index, h-core, and h-median of only those of its articles that were published in the last five complete calendar years.

We display the h5-index and the h5-median for each included publication. We also display an entire h5-core of its articles, along with their citation counts, so that you can see which articles contribute to the h5-index. And there's more! Click on the citation count for any article in the h5-core to see who cited it.

Coverage of Publications

Scholar Metrics currently cover articles published between 2018 and 2022 , both inclusive. The metrics are based on citations from all articles that were indexed in Google Scholar in July 2023 . This also includes citations from articles that are not themselves covered by Scholar Metrics.

Since Google Scholar indexes articles from a large number of websites, we can't always tell in which journal a particular article has been published. To avoid misidentification of publications, we have included only the following items:

• journal articles from websites that follow our inclusion guidelines ;
• selected conference articles in Engineering and Computer Science.

Furthermore, we have specifically excluded the following items:

• court opinions, patents, books, and dissertations;
• publications with fewer than 100 articles published between 2018 and 2022;
• publications that received no citations to articles published between 2018 and 2022.

Overall, Scholar Metrics cover a substantial fraction of scholarly articles published in the last five years. However, they don't currently cover a large number of articles from smaller publications.

Inclusion and Corrections

If you can't find the journal you're looking for, try searching by its abbreviated title or alternate title. There're sometimes several ways to refer to the same publication. (Fun fact: we've seen 959 ways to refer to PNAS.)

If you're wondering why your journal is not included, or why it has fewer citations than it surely deserves, that is often a matter of configuring your website for indexing in Google Scholar. Please refer to the inclusion manual . Also, keep in mind that Scholar Metrics only include publications with at least a hundred articles in the last five years.

• Privacy & Terms

The top 10 journal articles

Vol. 53 No. 1 Print version: page 26

1. COVID-19 disruption on college students: Academic and socioemotional implications

Tasso, A. F., Hisli Sahin, N., San Roman, G. J.

This study in Psychological Trauma: Theory, Research, Practice, and Policy (Vol. 13, No. 1) reveals that college students experienced emotional distress on many levels during the COVID-19 pandemic. Researchers surveyed 257 students at a U.S. college who all participated in remote learning off campus during the spring of 2020 because of the pandemic. Students reported being afraid of contracting COVID-19 and even more afraid of people within their social network contracting the virus. They also reported worrying about themselves or loved ones becoming severely ill, academic-related distress following the transition to remote learning, and COVID-19-related mental health distress, including interpersonal disengagement, struggles with motivation, and boredom, as well as anxiety, depression, and sleep disturbances. DOI: 10.1037/tra0000996

2. COVID-19 and the workplace: Implications, issues, and insights for future research and action

Kniffin, K. M., Narayanan, J., Anseel, F., Antonakis, J., Ashford, S. P., Bakker, A. B., Bamberger, P., Bapuji, H. Bhave, D. P., Choi, V. K., Creary, S. J., Demerouti, E., Flynn, F. J., Gelfand, M. J., Greer, L. L., Johns, G., Kesebir, S., Klein, P. G., Lee, S. Y., Ozcelik, H., Petriglieri, J. L., Rothbard, N. P., Rudolph, C. W., Shaw, J. D., Sirola, N., Wanberg, C. R., Whillans, A., Wilmot, M. P., Vugt, M.

This article in American Psychologist (Vol. 76, No. 1) presents possible workplace trends resulting from COVID-19, including remote work, virtual teamwork and management, social distancing, and unemployment. The analysis suggests that working from home will continue and expand post-pandemic. As for effects on workers, the authors predict increases in economic inequality, loneliness, stress, burnout, and addiction. Other workplace changes the authors forecast include virtual work arrangements that may foster more participatory relationships, new performance management and evaluation systems for remote workers, and new modes of surveillance by companies to check in on employees working remotely. DOI: 10.1037/amp0000716

3. A closer look at appearance and social media: Measuring activity, self-presentation, and social comparison and their associations with emotional adjustment

Zimmer-Gembeck, M. J., Hawes, T., Pariz, J.

This Psychology of Popular Media (Vol. 10, No. 1) study presents a tool to assess youth’s preoccupation with their physical appearance on social media. Researchers administered a 21-item survey about social media to 281 Australian high school students. They identified 18 items with strong inter-item correlation centered on three categories of social media behavior: online self-presentation, appearance-related online activity, and appearance comparison. In a second study with 327 Australian university students, scores on the 18-item survey were found to be associated with measures of social anxiety and depressive symptoms, appearance-related support from others, general interpersonal stress, coping flexibility, sexual harassment, disordered eating, and other issues. The researchers also found that young women engaged in more appearance-related social media activity and appearance comparison than did young men. DOI: 10.1037/ppm0000277

4. When social isolation is nothing new: A longitudinal study on psychological distress during COVID-19 among university students with and without preexisting mental health concerns

Hamza, C. A., Ewing, L., Heath, N. L., Goldstein, A. L.

In this study in Canadian Psychology (Vol. 62, No. 1), researchers examined the psychological impacts of COVID-19 on the mental health of postsecondary students with and without preexisting mental health concerns prior to the pandemic. The researchers surveyed 773 college students in Canada in May 2019 and again in May 2020 about recent stressful experiences and their mental health status. They found that students with preexisting mental health concerns showed improving or similar mental health during the early pandemic compared with 1 year prior. By contrast, students without preexisting mental health concerns were more likely to exhibit declining mental health during the pandemic, perhaps because they had less experience with social isolation than did students with preexisting mental health issues, the researchers suggest. DOI: 10.1037/cap0000255

5. Trauma-focused cognitive-behavioral therapy (TF-CBT) for interpersonal trauma in transitional-aged youth

Peters, W., Rice, S., Cohen, J., Murray, L., Schley, C., Alvarez-Jimenez, M., Bendall, S.

This pilot study in Psychological Trauma: Theory, Research, Practice, and Policy (Vol. 13, No. 3) indicates that trauma-focused cognitive behavioral therapy (TF-CBT) is an effective treatment for young people who have experienced post-traumatic stress disorder (PTSD) following interpersonal trauma such as child physical or sexual abuse, maltreatment, or neglect. Researchers delivered 15 TF-CBT sessions over 25 weeks to 20 youth ages 15 to 25 (­transitional-aged) in Australia, 16 of whom had a PTSD diagnosis. They found that following treatment, 15 of 16 participants no longer met criteria for a PTSD diagnosis, and self-report measures of PTSD, depression, and anxiety showed improvement, though some participants reported transient increases in symptoms. The researchers plan to conduct a larger randomized clinical trial to examine the effectiveness of TF-CBT for PTSD and other frequently co-occurring symptoms, including anxiety, depression, and substance use. DOI: 10.1037/tra0001016

6. Social media use and friendship closeness in adolescents’ daily lives: An experience sampling study

Pouwels, J. L., Valkenburg, P. M., Beyens, I., van Driel, I. I., Keijsers, L.

Adolescents who use social media apps such as Instagram more frequently than their peers feel closer to their friends, suggests this study in Developmental Psychology (Vol. 57, No. 2). Researchers asked 387 adolescents ages 13 to 15 in the Netherlands to report six times per day for 3 weeks their Instagram, WhatsApp, and Snapchat use in the previous hour, as well as their momentary experiences of friendship closeness. They found that participants who used WhatsApp and Instagram with close friends with whom they felt a sense of trust, support, and intimacy more frequently throughout the 3 weeks experienced higher levels of friendship closeness during the study than their peers. However, participants felt less close to their friends after they had used Instagram or WhatsApp in the previous hour, perhaps, the researchers suggest, resulting from unmet expectations that friends would immediately provide feedback on their posts. Neither association was found with Snapchat. DOI: 10.1037/dev0001148

7. Every (Insta)gram counts? Applying cultivation theory to explore the effects of Instagram on young users’ body image

Stein, J.-P., Krause, E., Ohler, P.

This study in Psychology of Popular Media (Vol. 10, No. 1) suggests that young people who frequently browse Instagram in a highly engaged way are more critical of strangers’ bodies and indulge more often in disordered eating—even if their own body image is unaffected. Researchers asked 228 participants ages 18 to 34 in Germany about changes in weight-related knowledge, attitudes, and self-reported dietary restraint. They found that participants, especially women, who browsed Instagram’s content more actively than their peers formed harsher views about the weight of strangers as well as an increased risk for disordered eating, but not a reduction in satisfaction with their own bodies. DOI: 10.1037/ppm0000268

8. Nonverbal overload: A theoretical argument for the causes of Zoom fatigue

Bailenson, J. N.

This review article in Technology, Mind, and Behavior (Vol. 2, No. 1) combines theory and prior research to derive four explanations for “Zoom fatigue,” the feeling of exhaustion brought on by video calls: excessive close-up eye contact with speakers, constant self-evaluation of one’s own image on the screen, remaining in a fixed position in view of the camera, and the increased cognitive load of sending and receiving nonverbal communication. The author offers the following solutions: reduce the size of the Zoom window to minimize face size, hide “self-view,” position the camera further away to allow for moving beyond a fixed sitting position without disrupting the call, and take “audio-only” breaks by both turning the camera off and turning away from the screen. DOI: 10.1037/tmb0000030

9. Coping during the COVID-19 pandemic: Relations with mental health and quality of life

Shamblaw, A. L., Rumas, R. L., Best, M. W. 

During the COVID-19 pandemic, people using avoidance coping strategies experienced increased depression and anxiety, while those using approach coping strategies, such as positive reframing, received the largest mental health boost, suggests this study in Canadian Psychology (Vol. 62, No. 1). In April 2020, researchers surveyed 797 online participants in the United States and Canada about 14 different approach or avoidance coping strategies as well as symptoms of depression, anxiety, and quality of life. One month later, 395 of the participants took the survey again. The researchers found that avoidance coping was associated with higher depression, higher anxiety, and lower quality of life at baseline and increased depression and anxiety 1 month later. Approach coping was associated with lower depression and better quality of life at baseline but not over the 1-month period. Of the specific coping strategies examined, reframing negative aspects of the pandemic was the most beneficial. DOI: 10.1037/cap0000263

10. Integrating responsive motivational interviewing with cognitive-behavioral therapy for generalized anxiety disorder: Direct and indirect effects on interpersonal outcomes

Muir, H. J., Constantino, M. J., Coyne, A. E., Westra, H. A., Antony, M. M.

This study in the Journal of Psychotherapy Integration (Vol. 31, No. 1) indicates that adding motivational interviewing (MI)—a psychotherapy module that helps people resolve feelings of ambivalence—to cognitive behavioral therapy (CBT) to treat generalized anxiety disorder (GAD) can bring about long-term changes in nonassertiveness and overaccommodation. In other words, the combination treatment helps people better assert themselves and not give in to others’ demands. Researchers randomly assigned 85 Canadian patients with GAD to a brief treatment of CBT or MI-CBT. Patients completed measures of nonassertiveness and overaccommodation throughout the treatment and across a 12-month follow-up. The researchers found that both MI-CBT and CBT reduced nonassertiveness and overaccommodation, but at 12 months, MI-CBT had helped patients more than CBT alone. This effect was explained by MI-CBT therapists’ ability to help patients overcome midtreatment resistance. DOI: 10.1037/int0000194

Contact APA

You may also like.

• Open access
• Published: 29 June 2006

Three options for citation tracking: Google Scholar, Scopus and Web of Science

• Nisa Bakkalbasi 1 ,
• Kathleen Bauer 1 ,
• Janis Glover 2 &
• Lei Wang 2  

Biomedical Digital Libraries volume  3 , Article number:  7 ( 2006 ) Cite this article

122k Accesses

367 Citations

28 Altmetric

Metrics details

Researchers turn to citation tracking to find the most influential articles for a particular topic and to see how often their own published papers are cited. For years researchers looking for this type of information had only one resource to consult: the Web of Science from Thomson Scientific. In 2004 two competitors emerged – Scopus from Elsevier and Google Scholar from Google. The research reported here uses citation analysis in an observational study examining these three databases; comparing citation counts for articles from two disciplines (oncology and condensed matter physics) and two years (1993 and 2003) to test the hypothesis that the different scholarly publication coverage provided by the three search tools will lead to different citation counts from each.

Eleven journal titles with varying impact factors were selected from each discipline (oncology and condensed matter physics) using the Journal Citation Reports (JCR). All articles published in the selected titles were retrieved for the years 1993 and 2003, and a stratified random sample of articles was chosen, resulting in four sets of articles. During the week of November 7–12, 2005, the citation counts for each research article were extracted from the three sources. The actual citing references for a subset of the articles published in 2003 were also gathered from each of the three sources.

For oncology 1993 Web of Science returned the highest average number of citations, 45.3. Scopus returned the highest average number of citations (8.9) for oncology 2003. Web of Science returned the highest number of citations for condensed matter physics 1993 and 2003 (22.5 and 3.9 respectively). The data showed a significant difference in the mean citation rates between all pairs of resources except between Google Scholar and Scopus for condensed matter physics 2003. For articles published in 2003 Google Scholar returned the largest amount of unique citing material for oncology and Web of Science returned the most for condensed matter physics.

This study did not identify any one of these three resources as the answer to all citation tracking needs. Scopus showed strength in providing citing literature for current (2003) oncology articles, while Web of Science produced more citing material for 2003 and 1993 condensed matter physics, and 1993 oncology articles. All three tools returned some unique material. Our data indicate that the question of which tool provides the most complete set of citing literature may depend on the subject and publication year of a given article.

Many researchers have an interest in finding citation information about a given article – both how many times the article is cited and who is citing that article. This may be for the completeness of a literature search, or perhaps to find how often his or her own publications are cited. Eugene Garfield made possible the widespread use of citation analysis in academe through his creation of three citation indices: Science, Humanities and Social Science Citation Indices, which were combined and transformed into an electronic version called the Web of Science. These indices were based on the concept that a carefully selected subset of journals would produce the majority of important citing literature for any given article. Citation analysis has real world implications: for good or bad, citedness is considered in grants, hiring and tenure decisions. For many reasons professors and researchers may want to demonstrate the impact of their work and citation analysis is one way (albeit a controversial one [ 1 – 3 ]) to accomplish this. For many years Web of Science had a virtual monopoly on the provision of citedness tracking. Late in 2004 two competitors to Web of Science emerged – Google Scholar and Scopus.

The Internet search giant Google sponsored the creation of Google Scholar, a tool that attempts to give users a simple way to broadly search the scholarly literature. Google Scholar uses a matching algorithm to look for keyword search terms in the title, abstract or full text of an article from multiple publishers and web sites (Google Scholar does not share the specifics of how this algorithm works). The number of times a journal article, book chapter, or web site is cited also plays an important part in Google Scholar's ranking algorithm. Search results are displayed so that the more cited and highly relevant articles rise to the top of the set. This varies from the more traditional default "reverse chronological" order employed by most scholarly databases. Google Scholar neither lists the journal titles it includes, nor the dates of coverage; although they have indicated that they have agreements with most major publishers (except Elsevier). Another area of difference for Google Scholar is that unlike most scholarly research databases, it looks beyond journal literature to cover other modes of scholarly communication. Other sources covered in Google Scholar include preprint servers such as arXiv (physics) and government and academic Web sites. Google Scholar does not state how a Web site qualifies for inclusion in its searches.

At approximately the same time that Google Scholar was made public, Elsevier introduced Scopus, an indexing and abstracting service that contains its own citation-tracking tool. Scopus indexes a larger number of journals than Web of Science, and includes more international and open access journals. Citation coverage however only dates to 1996 (abstracts, but not citation coverage, are available back to 1966 for some journals.) Scopus includes its own Web search engine, Scirus. Scirus results are presented separately from other Scopus journal results. Also, material from Scirus does not figure into citation counts for Scopus journal records. Table 1 provides a comparison summary of features in Web of Science, Scopus, and Google Scholar.

Citation analysis has been the focus of research and discussion for decades. Much has been written about citation analysis techniques [ 18 – 29 ], application to different disciplines [ 1 , 28 , 30 ], and controversies surrounding the use of citation analysis and journal impact factors to gauge the value and impact of a given journal title or the corpus of a given author [ 1 – 3 ]. With the introduction of Scopus and Google Scholar, there have been many recent articles that include careful analysis of the features of each individual tool as well as comparisons among two or more of these tools, and others (for example, PubMed and Scirus) [ 9 – 17 ]. While these articles discuss the general characteristics and report the results of sample searches the authors have completed, they do not systematically review the citation analysis functions. In a 2005 study analyzing Google Scholar, Noruzi [ 14 ] briefly compared citation counts for two products – Google Scholar and Web of Science – in the field of webometrics. First, the author selected the first article to establish the word "webometrics" [ 18 ], and provided the "times cited" for both the Web of Science and Google Scholar. The author then compared the number of unique and overlapping citations to this one article in each product. Noruzi also looked at the citation counts for the "most-cited" articles in the field by conducting a search on the term "webometrics or webometric" in each product.

There are inherent problems using subject searches as a comparison measure because of the differences in how Web of Science, Google Scholar and Scopus perform searches. For example, Web of Science does not automatically search for common word variations, while Scopus and Google Scholar do. Similar keyword searches in Scopus and Web of Science often return relatively small result sets (less than one hundred records), while the same search in Google Scholar may return hundreds of results. For example, a search for the phrase "complementary medicine" with the word "obesity" returns 9 results in Scopus, 6 in Web of Science and 596 results in Google Scholar.

Citation tracking of known articles as a comparison method avoids the inconsistencies in subject searching. In a preliminary study Bauer and Bakkalbasi [ 31 ] examined the citation counts for these three tools for articles from the Journal of the American Society for Information Science and Technology (JASIST) published in 1985 and 2000. They found that older material appears best covered by the Web of Science, although this was not confirmed statistically due to the small size of the dataset. For the newer material citation counts were higher in Google Scholar than either Web of Science or Scopus, while there was no statistical difference between the citation counts reported by Web of Science and Scopus. The authors recommended a larger, more robust study.

In attempting to provide a more robust study, this paper looks at a known set of articles, and examines the number of citing articles and other material returned by each of the three search tools for that discrete set, thus removing the ambiguity inherent in subject searches. In this way the study produces data sufficient to test the hypothesis that the different scholarly publication coverage provided by the three search tools will lead to different citation counts from each. In selecting a set of articles to work with, we decided that we should also account for the variations in the publication habits for various disciplines [ 4 – 8 ]. Thus we chose two disciplines to investigate that we suspected were following different publication patterns. One, physics, has largely embraced the use of preprint servers for the early dissemination of research literature, while a second discipline, medicine, has not. The subjects were narrowed to condensed matter physics (henceforth referred to as CM physics) and oncology. Sets of known articles from each discipline were selected from both 1993 (before e-publishing dominated scientific disciplines) and 2003 (well into the e-publishing era).

This approach of working with sets of known articles and looking for citing material mirrors the experience of the searcher who is interested in finding citing references to a known article. What can this researcher expect from this new landscape that includes the familiar indices from the Web of Science with the new territory of Scopus and Google Scholar?

Methodology

The sampling process included two steps: first the selection of journal titles from each discipline (oncology and CM physics) and second the selection of articles from those journals. In the first step, we retrieved 123 journals listed in the "Oncology" category and 60 journals listed in the "Physics, Condensed Matter" category, using the 2004 Journal Citation Reports (JCR) database. Eleven journals from each category were selected using systematic sampling technique to ensure the sample contained an even distribution of journals across all levels of impact factors. To draw the sample all titles were ranked from highest to lowest by impact factors. Then title selection began with the first title and was expanded to include every n th subsequent title where n, the sampling interval, was calculated as:

n = Population size/Sample size

Tables 2 and 3 furnish a list of the titles selected for the study.

Articles from years 1993 and 2003 were selected as the population from which to draw the sample. The second step of the sampling process began by retrieving all the articles published in the selected eleven titles for both years using Web of Science, INSPEC, and PubMed. All editorial materials, notes, biographical items, corrections, letters, book reviews, and news items were removed from the dataset before the sampling, as these items were not of primary concern to the study. Using stratified random sampling to allow a proportional representation of each journal, a random sample of articles from each journal was drawn according to the ratio of articles in a given journal to the total number of articles. This resulted in four sets of varying sizes: 234 and 259 for oncology 1993 and 2003, respectively; and 358 and 364 for CM physics 1993 and 2003 respectively. The set for CM physics was larger mainly because of the inclusion in the sample of Physics Review B, which publishes thousands of articles each year. These four sets of articles would be used to gather citation counts from Web of Science, Scopus, and Google Scholar.

To create sets for an examination of the citing references for articles published in 2003, fifty articles from the sets for oncology and CM physics 2003 were tagged for inclusion in two subsets. Between three and five articles from each journal title were included in these subsets.

Data collection

To construct the dataset author names, article title, source, volume number, and issue number were entered in a spreadsheet. Then during the week of November 7–12, 2005, citation counts were extracted for each research article from three sources: Web of Science, Scopus, and Google Scholar. The absence of some articles from any one of the three databases resulted in the elimination of 16 (7%) records from oncology 1993, 6 (2%) records from oncology 2003, and 18 (5%) records from CM physics 2003. Missing data from Scopus for CM physics 1993 resulted in a dataset too small to use for statistical significance. Thus Scopus was excluded from further analysis for this particular subject and year.

To augment the information gathered for citation counts, the actual citing references for the fifty articles in each of the two subsets of articles from 2003 oncology and CM physics were gathered from Web of Science, Scopus, and Google Scholar, resulting in two sets of citing references totaling 296 for CM physics and 614 for oncology.

Table 4 displays the descriptive statistics of the citation counts from each of the three resources. For oncology 1993, Web of Science returned the highest average number of citations, 45.3. Scopus returned the highest average number of citations (8.9) for oncology 2003. Web of Science returned the highest number of citations for CM physics 1993 and 2003 (22.5 and 3.9, respectively).

The hypothesis of the study is that the different scholarly publication coverage provided by the three search tools will lead to different citation counts from each. In addition, scholarly publication varies, encompassing many document dissemination methods depending on the subject discipline, and these differences will further be reflected in different citation counts for the three tools. We began by examining the following hypothesis:

H o : There is no difference among the citation counts extracted from the three resources.

H a : A difference exists among the citation counts extracted from the three resources.

Since the citation counts were highly skewed and the underlying assumptions for a parametric test were not met, a Friedman test, the non-parametric equivalent of the repeated measure ANOVA, was run for each discipline/year. Table 5 displays the summary results. The data showed a significant difference in the mean citation rates for at least one database.

Pairwise post-hoc comparisons using Wilcoxon Signed Ranked tests were performed to compare all possible pairs. Based on the post-hoc investigation there was a statistically significant difference in citation counts between all pairs (p < 0.001) except between Google Scholar and Scopus for CM physics 2003 (p = 0.119).

Overlap and uniqueness of citing references

An examination was done of the citing references returned from 2003 for both oncology and CM physics to further examine the composition of these sets. (Scopus did not return sufficient material for 1993, and so only 2003 articles were examined in this portion of the study.) In particular, we wished to determine the amount of citing references unique to each index, and the amount of citing references occurring in two or all three resources. An automated matching algorithm was developed to identify the overlapping and unique citing references. For each article, the algorithm divided all of its citing references into seven groups:

Overlap of all three resources

Overlap between Web of Science and Google Scholar

Overlap between Web of Science and Scopus

Overlap between Google Scholar and Scopus

Unique references from Web of Science

Unique references from Scopus

Unique references from Google Scholar

A sample of articles was selected to test the accuracy rate of the matching algorithm. The citing references for these articles were gathered, and the resulting set of 320 citing references were checked manually to determine if the algorithm had placed each citing reference in the correct category (of the seven listed above.) If the citing article was not placed in the correct category by the matching algorithm, it was marked as an error. The test demonstrated an accuracy rate of 98% for the matching algorithm. This was an acceptable accuracy rate, and so the matching algorithm was then used to categorize the citing references for all of the 2003 articles. Figure 1 shows the distribution of the unique and overlapping references as returned by the algorithm.

The distribution of the unique and overlapped citing articles as returned by the algorithm. Note: diagram is not to scale.

For oncology articles published in 2003, the greatest number of unique items was found in Google Scholar (78, 13%) followed by Scopus (74, 12%). A search of Web of Science revealed a much smaller number of unique citing articles (41, 7%). A core group of 189 (31%) citing articles was found in all three resources.

In the set of all citing material for the CM physics 2003 articles, the greatest amount of unique material was found in Web of Science (63, 21%), which also produced the highest citation counts. The second greatest concentration of unique material was in Google Scholar (50, 17%). Scopus returned 25 unique citing articles (9%). In contrast to the set of articles for oncology, only 63 (21%) were found in all three resources.

The large amount of unique material returned by Google Scholar led to further examination of the composition of that material. What percentage of that material would correspond to the traditional journal literature, and what percentage might reflect new forms of scholarly communication methods? From each 2003 set, 50 unique citing references from Google Scholar were examined to determine their origin. Citing references were classified as:

1. Journal: any journal, open access or not.

2. Archive: A subject-specific repository. Examples are arXiv (physics), Repec (economics) and ADS (astrophysics).

3. College or university sponsored: An institution-specific resource. May be a repository (such as the DSpace repository at MIT) or simply a departmental Web page.

4. Governmental: white papers and technical reports from .gov sites.

5. Non-governmental Organizations: White papers and technical reports from research institutes.

6. Commercial Entity: A paper published by a for-profit organization, such as a pharmaceutical company.

The largest amount of unique material in Google Scholar came from journal literature for both oncology and CM physics. However in CM physics that percentage (38%) was much lower than for oncology (62%). In CM physics, the next largest contributing factor was material housed on archives (specifically arXiv). This accounted for 12 articles, or 25% of the unique material. In oncology, the next largest group of material came from colleges and universities (9 citing references, or 18%).

This study examined a defined set of articles from two subject disciplines: oncology and condensed matter physics. A search was done to uncover citing material in each of three products, Web of Science, Scopus and Google Scholar. For articles published in 1993, Web of Science returned the greatest number of citing articles in both CM physics and oncology. In oncology 1993, Scopus was next and Google Scholar produced the least number of citing articles. In CM physics 1993, Google Scholar was next and Scopus provided too few articles to study. Given the depth of Web of Science coverage in sciences (back to 1900), and Google Scholar's reliance on digital material (which in general dates back to the mid-1990's) this result is not surprising. For articles published in oncology in 2003, Scopus returned the highest number of citing references, followed by Web of Science and Google Scholar. In CM physics 2003, Web of Science returned the largest number of citing references, and the number returned by Scopus and Google Scholar was not statistically different. This result surprised the authors and contradicted their supposition that changes in scholarly publication, especially in CM physics, would be reflected in the citation counts from Google Scholar.

To look further at the sets of citing material returned, the unique and overlapping material found for both oncology and CM physics in 2003 were examined. In oncology, the largest set of unique material came from Google Scholar, but in CM physics Web of Science returned more unique material. In oncology, a larger percentage of citing material was common to all three resources (31%) than in physics, where only 21% of material was contained in all three resources. When unique citing references were found in a search of Scopus or Web of Science, it was either because of different journal title coverage, or sometimes because one index included articles from a particular publisher faster than the other. In Google Scholar the composition of the unique citing material was more varied, consisting of journals, e-prints, university, governmental and non-governmental material.

The overlap offered by Google Scholar was in some ways as interesting as the unique material. In oncology, 40% of the material it produced overlapped with either Scopus or Web of Science or both indices. The large amount of overlapping material gave some credence to the scholarly nature of the Google Scholar database, and the accuracy of its matching algorithm for detecting correct citing references. The unique material returned by Google Scholar sometimes consisted of journal literature not covered by the other indices (or not yet indexed) but also was comprised of a mix of material published on e-print archives, university, governmental and non-governmental organization web sites.

A possible bias may have been introduced by using JCR (a Thomson Science produced companion product to Web of Science) to select the journal titles for each discipline. One might wonder whether changing the study by selecting the journal titles from another source would have made an impact on the results reported in this study. However, the journals included in this study were indexed by Web of Science and Scopus, and appeared in Google Scholar; this argues against any possible bias created by using JCR.

During this investigation, it became obvious that Google Scholar changed rather dramatically after November 2005. When searches were rerun in Google Scholar in January 2006, some results were much larger. It can safely be assumed that Google Scholar citing result sets for the same articles studied here would now be different and probably larger. Similarly, some searches run in Scopus now give much higher citation counts. It would appear that Scopus has also undergone some improvements.

This study analyzed the numbers of citing references for given sets of articles from journals in oncology and condensed matter physics for two publication years: 1993 and 2003, and further compared the citing references for fifty articles in two disciplines for 2003 only. This study did not identify any one of the three tools studied to be the answer to all citation tracking needs. Scopus showed strength in providing citing literature for more current (2003) oncology articles. However, Web of Science seemed to perform better for current CM physics, and was stronger for both subjects for articles published in 1993. Google Scholar returned a smaller number of citing references, but did provide a large set of unique citing material for 2003. Also, as a resource freely available to anyone with Internet connectivity, Google Scholar deserves consideration as an important adjunct to other research indices. This study however indicated that at this point Google Scholar alone might not replace other scholarly search tools. Scopus and Web of Science remain very important resources, but this study cannot claim one to be the clear winner for all subject matter. Rather, it indicates that the question of which tool is better, or at least which tool is better in terms of providing the most complete set of citing literature, may depend on the subject and publication date of a given article.

This study revealed that a researcher who needs to be comprehensive in a literature search has no simple solution. That is, none of these products covered the entire set of citing articles this study produced. In oncology, a researcher who consulted the index with the largest number of citing references (Scopus) would have found 76% of these citing references, and by adding a search of Google Scholar (which produced the most unique material) would find 94% of citing references. In CM physics a researcher who consulted only Web of Science would find 71% of citing references. By consulting Google Scholar in addition to Web of Science they would find 91%. A researcher using any two of three tools would find the majority of, but not all, citing material found in this study.

We note that Google Scholar and Scopus have both changed, perhaps dramatically since the time this sample was drawn. This would indicate that sampling should be repeated for more up-to-date comparisons, and to most fairly evaluate the utility of Google Scholar and Scopus. In addition, it is clear that Google Scholar provides unique citing material. The exact composition of this citing material should also be more thoroughly examined so that scholars will have a clear idea what is and is not included in Google Scholar searches.

Cheek J, Garnham B, Quan J: What's in a number? issues in providing evidence of impact and quality of research(ers). Qual Health Res. 2006, 16: 423-435. 10.1177/1049732305285701.

Article   PubMed   Google Scholar  

Seglen PO: Why the impact factor of journals should not be used for evaluating research. Br Med J. 1997, 314: 498-502.

Article   CAS   Google Scholar  

Walter G, Bloch S, Hunt G, Fisher K: Counting on citations: A flawed way to measure quality. Med J Aust. 2003, 178: 280-281.

PubMed   Google Scholar  

Kling R, McKim G: Scholarly communication and the continuum of electronic publishing. J Am Soc Inf Sci. 1999, 50: 890-906. 10.1002/(SICI)1097-4571(1999)50:10<890::AID-ASI6>3.0.CO;2-8.

Article   Google Scholar  

Kling R, McKim G: Not just a matter of time: Field differences and the shaping of electronic media in supporting scientific communication. J Am Soc Inf Sci. 2000, 51: 1306-1320. 10.1002/1097-4571(2000)9999:9999<::AID-ASI1047>3.0.CO;2-T.

Modlin IM, Adler G, Alexander K, Arnold RF, Brenner DA, Corazziari E, Floch MH, LaPorte RE, Peterson WL, Quigley EM, Shapiro MD, Spechler SJ, Spiller RC, Tytgat GN, Wiegers WF: Information assimilation and distribution challenges and goals for real and virtual journals. J Clin Gastroenterol. 2005, 39: 181-188. 10.1097/01.mcg.0000152791.45765.32.

Brown CM: Information seeking behavior of scientists in the electronic information age: Astronomers, chemists, mathematicians, and physicists. J Am Soc Inf Sci. 1999, 50: 929-943. 10.1002/(SICI)1097-4571(1999)50:10<929::AID-ASI8>3.0.CO;2-G.

Brown C: The e-volution of preprints in the scholarly communication of physicists and astronomers. J Am Soc Inf Sci Technol. 2001, 52: 187-200. 10.1002/1097-4571(2000)9999:9999<::AID-ASI1586>3.0.CO;2-D.

Deis LF, Goodman D: Web of Science (2004 version) and Scopus. Charleston Advisor. 2005, 6: March 2, 2005, [ http://www.charlestonco.com/comp.cfm?id=43 ]

Google Scholar  

Felter LM: Google Scholar, Scirus, and the scholarly search revolution. Searcher. 2005, 13: 43-48.

Giustani D, Barsky E: A look at Google Scholar, PubMed, and Scirus: Comparisons and recommendations. JCHLA/JABSC. 2005, 26: 85-89.

Henderson J: Google Scholar: A source for clinicians?. Can Med Assoc J. 2005, 172: 1549-1550. 10.1503/cmaj.050404.

Myhill M: Google Scholar. Charleston Advisor. 2005, 6: April 14, 2006, [ http://www.charlestonco.com/review.cfm?id=225 ]

Noruzi A: Google Scholar: The new generation of citation indexes. Libri. 2005, 55: 170-180.

Notess GR: Scholarly web searching: Google Scholar and Scirus. Online. 2005, 29: 39-41.

Roth DL: The emergence of competitors to the science citation index and the Web of Science. Curr Sci. 2005, 89: 1531-1536.

Vine R: Google Scholar. J Med Libr Assoc. 2006, 94: 97-99.

PubMed Central   Google Scholar  

Almind TC, Ingwersen P: Informetric analyses on the World Wide Web: Methodological approaches to webometrics. J Doc. 1997, 53: 404-426. 10.1108/EUM0000000007205.

Borgman CL, Furner J: Scholarly communication and bibliometrics. Annu Rev Inf Sci Technol. 2002, 36: 3-72.

Butler D: Science searches shift up a gear as Google starts scholar engine. Nature. 2004, 432: 423-10.1038/432423a.

Article   CAS   PubMed   Google Scholar  

Garfield E: Citation indexes for science – new dimension in documentation through association of ideas. Science. 1955, 122: 108-111.

Garfield E: Citation Indexing – Its Theory and Application in Science, Technology, and Humanities. 1979, New York: Wiley

Garfield E: The agony and the ecstasy – the history and meaning of the journal impact factor. International Congress on Peer Review and Biomedical Publication. 2005, September 16, 2005; Chicago, IL

Kurtz MJ, Eichhorn G, Accomazzi A, Grant C, Demleitner M, Henneken E, Murray SS: The effect of use and access on citations. Inf Process Manage. 2005, 41: 1395-1402. 10.1016/j.ipm.2005.03.010.

Rahm E, Thor A: Citation analysis of database publications. SIGMOD Rec. 2005, 34: 48-53. 10.1145/1107499.1107505.

Weale AR, Bailey M, Lear PA: The level of non-citation of articles within a journal as a measure of quality: A comparison to the impact factor. BMC Med Res Methodol. 2004, 4: 14-10.1186/1471-2288-4-14.

Article   PubMed   PubMed Central   Google Scholar  

Zhao DZ: Challenges of scholarly publications on the web to the evaluation of science – A comparison of author visibility on the web and in print journals. Inf Process Manage. 2005, 41: 1403-1418. 10.1016/j.ipm.2005.03.013.

Vincent A, Ross D: On evaluation of faculty research: Impact of citation analysis. J Appl Bus Res. 2000, 16: 1-14.

Bollen J, de Sompel HV, Smith JA, Luce R: Toward alternative metrics of journal impact: A comparison of download and citation data. Inf Process Manage. 2005, 41: 1419-1440. 10.1016/j.ipm.2005.03.024.

Holden G, Rosenberg G, Barker K: Tracing thought through time and space: A selective review of bibliometrics in social work. Soc Work Health Care. 2005, 41: 1-34. 10.1300/J010v41n03_01.

Bauer K, Bakkalbasi N: An examination of citation counts in a new scholarly communication environment. D-Lib Mag. 2005, 11: April 14, 2006. Scopus, [ http://dx.doi.org/10.1045/september2005-bauer ]

Download references

Author information

Authors and affiliations.

Yale University Library, 130 Wall St., P.O. Box 208240, New Haven, CT, 06520-8240, USA

Nisa Bakkalbasi & Kathleen Bauer

Cushing/Whitney Medical Library, Yale School of Medicine, 333 Cedar St, P.O. Box 20804, New Haven, CT, 06520-8014, USA

Janis Glover & Lei Wang

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Kathleen Bauer .

Additional information

Competing interests.

The author(s) declare that they have no competing interests.

Authors' contributions

KB conceived the study, coordinated the project, participated in data collection and drafted most parts of the manuscript. NB participated in the data collection, performed the statistical analysis and interpretation of the data and helped with the draft of the manuscript. JG participated in the data collection, conducted literature review, and edited the draft of the manuscript. LW participated in the data collection, developed the Google Scholar citing reference extraction tool and the matching algorithm for grouping citing references. All authors read and approved the final manuscript.

Nisa Bakkalbasi, Kathleen Bauer, Janis Glover and Lei Wang contributed equally to this work.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions.

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Bakkalbasi, N., Bauer, K., Glover, J. et al. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr 3 , 7 (2006). https://doi.org/10.1186/1742-5581-3-7

Download citation

Received : 18 April 2006

Accepted : 29 June 2006

Published : 29 June 2006

DOI : https://doi.org/10.1186/1742-5581-3-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Biomedical Digital Libraries

ISSN: 1742-5581

Clarivate Identifies the One in 1,000 Citation Elite with Annual Highly Cited Researchers List

Mainland China has nearly doubled its share of Highly Cited Researchers in four years

London, U.K., November 16, 2021 – Clarivate Plc (NYSE:CLVT), a global leader in providing trusted information and insights to accelerate the pace of innovation, unveiled its 2021 list of Highly Cited Researchers™ today. The methodology that determines the “who’s who” of influential researchers draws on the data and analysis performed by bibliometric experts and data scientists at the Institute for Scientific Information™ at Clarivate.

The annual list identifies some 6,600 researchers from across the globe who demonstrated significant influence in their chosen field or fields through the publication of multiple highly cited papers during the last decade. The Highly Cited Researchers’ names are drawn from the publications that rank in the top 1% by citations for field and publication year in the Web of Science™ citation index, and the list identifies the research institutions and countries where they are based.

The key findings for 2021 show:

• 6,602 researchers from more than 70 countries and regions have been recognized this year – 3,774 in specific fields and 2,828 for cross-field impact.
• The United States is the institutional home for 2,622 of the Highly Cited Researchers in 2021, which amounts to 39.7%, down from 43.3% in 2018. While there has been a decline in the number of U.S.-based Highly Cited Researchers, there can be no doubt that the U.S. still leads the world in research influence. Of all papers indexed in the Web of Science for 2010 to 2020 the percentage with a U.S.-based author was 24.7%.
• Mainland China is second this year, with 935 Highly Cited Researchers, or 14.2%, up from 7.9% in 2018. In four years, Mainland China has nearly doubled its share of the Highly Cited Researchers population.
• The United Kingdom , with 492 researchers or 7.5%, comes in third. This is a particularly high number of researchers at the very top of their fields in terms of citation impact, given that the United Kingdom has a population 1/5 the size of the United States and 1/20 the size of Mainland China.
• Australia has narrowly overtaken Germany at fourth, with 332 researchers, and the Netherlands is sixth, with 207 researchers – remarkable for countries of 25 million and 17 million, respectively, versus Germany’s 83 million. They also place above Canada, France, Spain and Switzerland in the top 10.
• Harvard University , home to 214 researchers, is once again the institution with the highest concentration of Highly Cited Researchers in the world.
• Hong Kong has increased its number to 79 from 60 last year, an impressive achievement, partly due to a dramatic increase in Highly Cited Researchers from the University of Hong Kong, which more than doubled its number of Highly Cited Researchers from 14 to 33 from 2020 to 2021.
• For the first time, researchers from Bangladesh , Kuwait , Mauritius , Morocco and the Republic of Georgia are included on the list this year.

Naturally, Mainland China’s gain means losses elsewhere. There is a 1.8% loss in Highly Cited Researchers for the United States since last year and 3.6% since 2018. This contrasts with an increase of 6.3% for Mainland China since 2018. The United Kingdom exhibits a decline of .5% since last year and 1.5% since 2018. Germany has lost .9% share since 2018.

David Pendlebury, Senior Citation Analyst at the Institute for Scientific Information, said: “The headline story is one of sizeable gains for Mainland China and a decline for the United States, particularly when you look at the trends over the last four years, which reflect a transformational rebalancing of scientific and scholarly contributions at the top level through the globalization of the research enterprise.”

Nobel Prize recipients and researchers of Nobel quality

This year’s list includes 24 Nobel laureates, including five announced this year : David Julius, University of California San Francisco, San Francisco, CA, United States (Physiology or Medicine); Ardem Patapoutian, Scripps Research, La Jolla, CA, United States (Physiology or Medicine); David W. C. MacMillan, Princeton University, Princeton, NJ, United States (Chemistry); David Card, University of California Berkeley, Berkeley, CA, United States (Economics); and, Guido Imbens, Stanford University, Stanford, CA, United States (Economics). Also included are 77 Citation Laureates ™: individuals recognized by Clarivate, through citation analysis, as ‘of Nobel class’ and potential Nobel Prize recipients.

Exceptional broad performance Of the researchers named as Highly Cited in the 21 Essential Science Indicators (ESI)™ fields, 23 researchers showed exceptionally broad performance, recognized for being highly cited in three or more fields. They are a truly global group – in North America, Europe, Asia and the Middle East. Professor Rob Knight from the University of San Diego California was alone in being named for four ESI fields (Biology and Biochemistry; Environment/Ecology; Microbiology; Molecular Biology and Genetics).

Figure 1: Highly Cited Researchers by country or region

Figure 2: Highly Cited Researchers by research institution or organization

Joel Haspel, SVP Strategy, Science at Clarivate said: “This year’s data reflect a decade’s worth of research publications from the global scientific community. As well as documenting the ‘Eureka!’ moments, our data tell the story of late nights spent filling in grant applications, poring over results in the lab, the unsung work of peer reviewing contemporaries’ manuscripts, and the many small failures that ultimately lead to bigger successes and accelerating innovation.

Our analysts have found continued growth in the highly cited, high-impact research from Mainland China, but the United States remains the scientific powerhouse of the world, and U.S. institutions represent five of the top ten, with Harvard University at the very top of the leader board.”

The full 2021 Highly Cited Researchers list and executive summary can be found here .

Follow us online: on Twitter @ClarivateAG #HighlyCited2021.

Notes to editors:

Methodology: More than 6,600 researchers, in 21 fields of the sciences and social sciences, and cross-field categories were selected based on the number of highly cited papers they produced over an 11-year period from January 2010 to December 2020. The methodology that determines the who’s who of researchers draws on data and analysis performed by bibliometric experts at the Institute for Scientific Information at Clarivate. It uses InCites Benchmarking & Analytics ™, Essential Science Indicators ™ and a unique compilation of science performance metrics and trend data based on scholarly paper publication counts and citation data from the Web of Science ™, the world’s largest publisher-neutral citation index and research intelligence platform. The sharp decline of .7% for Switzerland since 2018 is anomalous and reflects a change in our methodology: Papers with more than 30 institutional addresses were removed from our analysis in past years, but this year we eliminated papers with more than 30 authors or group authorship. The change, which we judged an improvement in reasonably crediting individual authors – the previous use of institutional addresses was a heuristic – happened to impact Switzerland heavily and especially researchers at the Swiss Institute of Bioinformatics, which produces a significant number of highly cited papers with many authors but few institutional addresses.

About Clarivate Clarivate™ is a global leader in providing solutions to accelerate the lifecycle of innovation. Our bold mission is to help customers solve some of the world’s most complex problems by providing actionable information and insights that reduce the time from new ideas to life-changing inventions in the areas of science and intellectual property. We help customers discover, protect and commercialize their inventions using our trusted subscription and technology-based solutions coupled with deep domain expertise. For more information, please visit clarivate.com.

Media Contact: Amy Bourke-Waite Director of External Communications – Science [email protected]

Related news

Clarivate launches research horizon navigator to pinpoint future breakthrough areas.

Clarivate Celebrates 500 Institutions Adopting Rialto Marketplace Solution

Clarivate in the Age of AI: Innovation Rooted in Academia

Subscribe to our news

Let's stay in touch. Subscribe to our latest announcements.

What Are The Most Cited Research Papers Of All Time?

The writers at Nature News recently put together a list of the 100 most highly cited papers of all time. There are a few surprises in here, including the fact that it takes no fewer than 12,119 citations to rank in the top 100.

Suggested Reading

The list, which was created by pulling data from the Science Citation Index (SCI), spans the last 100 years of scholarly publications. The sheer size of the literature — 58 million items — shows that the top 100 papers are true outliers; just three publications have more than 100,000 citations. Many of the world's most famous and influential papers didn't even make the cut.

Related Content

To help put it all into perspective, Nature News put together a video (above) and this infographic (click to embiggen).

If the cumulative stack of all these papers were scaled to the size of Mount Kilimanjaro, then the 100 most-cited papers would represent just one centimeter at the peak. Only 14,499 papers — about a meter and a half's worth — have more than 1,000 citations. Roughly half of the items have only one citation.

So what's the most cited paper of all time? That distinction goes to a 1951 paper by U.S. biochemist Oliver Lowry and colleagues describing an assay to determine the amount of protein in a solution. To date, it has collect more than 305,000 citations. And no one's entirely sure why...

Here are the top five:

Sexy, right?

Shockingly, Watson and Crick's paper on the structure of DNA missed out (just 5,207 citations... what!? ), along with many other historic breakthroughs (like the 1985 discovery of the ozone hole — just 1,871 citations). Instead, papers on methods and software dominate the list.

Here's a breakdown of the citations by category:

You can browse through the entire list here (.xls spreadsheet) or via Nature News 's interactive graphic . And there's much more at the Nature News article .

• Get new issue alerts Get alerts

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

The 100 most cited articles have fewer citations than other bibliometric articles: A pairwise comparison using a temporal bubble graph

Hsieh, Wan-Ting MD a ; Chien, Tsair-Wei MBA b ; Chou, Willy MD c,d,*

a Department of Palliative Medicine, Chi-Mei Medical Center, Tainan, Taiwan

b Medical Research Department, Chi-Mei Medical Center, Tainan, Taiwan

c Department of Physical Medicine and Rehabilitation, Chiali Chi-Mei Hospital, Tainan, Taiwan

d Department of Physical Medicine and Rehabilitation, Chung San Medical University Hospital, Taichung, Taiwan.

Received: 27 October 2022 / Received in final form: 7 November 2022 / Accepted: 8 November 2022

The authors have no funding and conflicts of interest to disclose.

The datasets generated during and/or analyzed during the current study are publicly available.

Supplemental Digital Content is available for this article.

How to cite this article: Hsieh W-T, Chien T-W, Chou W. The 100 most cited articles have fewer citations than other bibliometric articles: A pairwise comparison using a temporal bubble graph. Medicine 2022;101:48(e32101).

* Correspondence: Willy Chou, Department of Physical Medicine and Rehabilitation, Chi-Mei Medical Center, 901 Chung Hwa Road, Yung Kung Dist., Tainan 710, Taiwan (e-mail: [email protected] ).

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial License 4.0 (CCBY-NC) , where it is permissible to download, share, remix, transform, and buildup the work provided it is properly cited. The work cannot be used commercially without permission from the journal.

Background: 

More than 400 articles with the title of 100 top-cited articles (Top100) have been published in PubMed. It is unknown whether their citations are fewer (or more) than those found in other bibliometric studies (Nontop100). After determining article themes using coword analysis, a temporal bubble graph (TBG) was used to verify the hypothesis that the Top100 had fewer citations than the Nontop100.

Methods: 

Using the Web of Science core collection, the top 50 most cited articles were compiled by Top100 and Nontop100, respectively, based on the research area of biomedicine and bibliometrics only. Coword analysis was used to extract themes. The study results were displayed using 6 different visualizations, including charts with bars, pyramids, forests, clusters, chords, and bubbles. Mean citations were compared between Top100 and Nontop100 using the bootstrapping method.

Results: 

There were 18 citations in total for the 2 sets of the 50 most cited articles (range 1–134; 5 and 26.5 for Top100 and Nontop100, respectively). A significant difference in mean citations was observed between the 2 groups of Top100 and Nontop100 based on the bootstrapping method (3, 95% confidence interval: [1.18, 4.82]; 26.5, 95% confidence interval: [23.82, 29.18], P  < .001). The 11 themes were clustered using coword analysis and applied to a TBG, which is composed of 4 dimensions: themes, years, citations and groups of articles. Among the 2 groups, the majority of articles were published in the journal of Medicine (Baltimore ), with 9 and 7, respectively.

Conclusion: 

Eleven themes were identified as a result of this study. In addition, it reveals distinct differences between the 2 groups of Top100 and Nontop100, with the former containing more recently published articles and the latter containing more citations for articles. Clinical and research clinicians and researchers can use bibliometric analysis to appraise published literature and to understand the scientific landmark using TBG in bibliometrics.

• • A coword analysis was carried out to investigate the article themes assigned to each article, which is a novel and modern approach in the literature.
• • The TBG enhances the traditional impact beam plot (dot plot) and the temporal bar graph, with 4 dimensions combined in a view.
• • Six visualizations were provided in this study, and future relevant studies can easily understand the study results with a quick glance

1. Introduction

The number of bibliographic studies published in the field of life sciences and biomedicine has steadily increased over time. There are several reasons behind this rise, including improved accessibility of bibliographic data and software packages that specialize in bibliographic analysis. [ 1 ] Over 400 articles with titles of 100 top-cited articles (Top100) were published in PubMed, [ 2 ] a free search engine that primarily accesses the MEDLINE database of references and abstracts on life sciences and biomedical topics.

A sharp rise in Top100 publications has also been observed [ 2 ] in the past ten years (i.e., [18, 12, 11, 24, 29, 37, 57, 80, 90, 102] by count since 2013). Nonetheless, it is unknown whether their citations are fewer (or more) than their counterparts in other bibliometric studies (Nontop100) based on the research area of biomedicine and bibliometrics. A comparison of article citations between Top100 and Nontop100 is therefore necessary for the study to be conducted.

1.1. Difference in citations between bibliometrics and meta-analysis

The number of meta-analysis studies and systematic reviews has steadily increased, with a total of 5975 and 2119 articles, respectively, [ 3 , 4 ] somewhat lagging behind the number of 7912 for bibliometrics. [ 5 ] The availability of online data and software packages for analyses has led to an increase in bibliographic and meta-analysis (or systematic literature review) studies, [ 2 , 6 ] rather than those relying solely on library literature searches. Compared to the article mean citations, a distinct difference was observed for meta-analysis (7.1:5.2). [ 1 ] The current study is intended to compare the article mean citations between the Top100 and Nontop100 within bibliographical studies, based on the research area of biomedicine and bibliometrics only.

1.2. Challenges encountered in comparison of citations between Top100 and Nontop100

Comparing article citations between Top100 and Nontop100 presents 3 challenges, including: how to conduct citation comparisons when citations are not normally distributed, [ 7 ] classifying article themes in comparison of mean citations (=citations/publications = impact factor [IF]), and displaying citation comparisons on dashboards for a better understanding of detailed features in the 2 groups of Top100 and Nontop100.

It is fortunately possible to obtain the bootstrapping method module from a previous study [ 7 ] if the data are not normally distributed when estimating the mean and standard error. By using cluster names in CiteSpace, [ 8 ] we are able to mimic the method of determining article themes by coword analysis. Each document can be assigned a theme based on a reference to a previous study, [ 9 ] and a timeline clustering map of the articles’ cocitation networks can be constructed using CiteSpace (i.e., themes on the row, years on the column, bubbles sized by citations). [ 10 ]

Furthermore, temporal bar graphs have been used in bibliometric research. [ 11–14 ] It is the disadvantage of temporal bubble graph (TBG) [ 11 ] and timeline clustering maps in CiteSpace [ 8 , 10 ] that they cannot be linked to websites as a dashboard does for readers. This study proposes to enhance TBGs by combining their advantages and adding the function of dashboards to TBG.

1.3. Top100 have fewer citations than Nontop100

Reviewing the top 100 most highly cited articles in PubMed, [ 2 ] most of these articles focus on a specific topic or discipline and thus limit audiences to those working in that area. On the other hand, Nontop100 articles focus on methods and techniques of analysis, as well as the application of bibliometric metrics (e.g., h-/x-/Y-/ht-index [ 15–18 ] ). According to previous studies, [ 19–22 ] a higher IF has been associated with the publication of reviews and original articles rather than case reports. A rigorous systematic review receives a greater number of citations than a narrative review, whereas case reports with low impact factors are characterized by their focus on a specific topic and are then rarely cited. Therefore, it is proposed and required to verify the hypothesis that the Top100 papers have fewer citations than the Nontop100 papers in the current study.

1.4. Study aims

The aim of this study is to verify the hypothesis that the Top100 articles have fewer citations than the Nontop100 articles, as well as to examine which countries, institutes, departments, authors, and journals dominate the articles in the Top100 and Nontop100.

2.1. Data sources

We searched the keywords of (bibliometric [MeSH Major Topic]) since 2013 in PubMed to ensure articles related to the research area of biomedicine and bibliometrics only. A total of 4574 articles with the PubMed identity number were matched to those in the Web of Science core collection for the extraction of corresponding authors because it is difficult to identify which are corresponding authors in PubMed. The top 50 most cited articles in Web of Science (WoS) were compiled each by Top100 and Nontop100; see Supplemental Digital Content 1, https://links.lww.com/MD/I43 .

As this study did not involve the examination or treatment of patients or review of patient records, it was exempt from review and approval by our research ethics committee.

2.2. Four approaches applied to this study

2.2.1. descriptive statistics in publications..

A bar chart was drawn to illustrate the publication trend of articles related to bibliometrics. Pyramid plots in R were used to visualize the top journals with the most publications and mean citations; see Supplemental Digital Content 2, https://links.lww.com/MD/I44 .

2.2.2. Research achievements in Top100 and Nontop100.

A 4-quadrant plot [ 23 ] was applied to present the dominant entities based on the CJAL score [ 23 ] determined by the category, journal impact factor, and authorship (CJA) score [ 24 ] and the L-index [ 25 ] via Equations 1 to 3.

Three factors are considered in the CJA scores for a published article: the Category (C; e.g., review, original article, case report, etc.), the journal “quality” (J; e.g., impact factor, JIF, or ranking of the journal), and the authorship order denoted by A). The CJAL score is calculated by multiplying each of these 3 aspects as well as the L-index (Equation 3). CJA scores original research articles higher than other types of manuscripts; co-first authors (denoted RP and FP to compute the Y-index RP + FP [ 26 , 17 ] ) score higher than other collaborators; for the journal’s quality assessment, they use the JIF or SCI/SSCI journal rankings for SCI/SSCI-indexed papers. [ 24 ] SCI/SSCI journal rankings are based on JIF in each research domain; therefore, domain-specific journal rankings are usually not significantly different from those based on JIF. [ 23 , 24 ]

The top 10 elements in each entity with CJAL scores in Top100 and Nontop100 are shown on a 4-quadrant radar plot, [ 23 ] including countries, institutes, departments, and authors by 2 factors (i.e., RP and FP) on the coordinates. Bubbles were sized by the CJAL score. Accordingly, it is possible to compare the research achievements (Ras) of the top 10 members in each entity with a glance view.

2.2.3. Thematic analysis and comparison of major themes between groups.

2.2.3.1. themes derived from those 100 articles..

By using social network analysis (SNA), [ 27 , 28 ] coword analysis was performed to extract the chief components in clusters as themes (or leaders) in Keywords Plus that were retrieved from the Web of Science core collection. Next, articles were assigned with themes extracted from SNA using equation 4. [ 9 ]

where L is the number of keywords in article i , n is the number of keywords denoted by keyword k belonging to the theme defined in SNA (i.e., the more coexisting keywords are gathered in an identical cluster). Accordingly, the theme is redirected to the maximal number of keywords (= m ) involved in the cluster via Equation 4.

Using a chord diagram, [ 29 ] we can understand which themes dominate these 100 articles and their relationship through the color-coded curves to connections; the way to draw the chord diagram is shown in Supplemental Digital Content 2, https://links.lww.com/MD/I44 .

2.2.3.2. Comparison of major themes between groups.

Comparisons of these themes extracted from the SNA were made for both groups of Top100 and Nontop100 using forest plots. [ 1 , 30 ] Vertical lines represent no effect (e.g., OR = 1). For example, if the confidence intervals (CI) for an individual study (e.g., keyword in this study) overlap with this line, the effect size does not differ from zero (or 1.0) for that study (standard mean difference or odds ratio) at a given level of confidence (e.g., P  < .05). If the points of the diamond touch the line of no effect, the overall effect cannot be said to differ from no effect at a given level of confidence. [ 1 ] With the additional function of zoom-ins and zoom-outs on Google Maps, we incorporated the forest plot on a dashboard to better present the effect on each of these observed studies.

2.2.4. Identification of hypothesis.

To confirm the hypothesis that the Top100 articles have fewer citations than the Nontop100 articles, TBG and forest plots were used.

2.2.4.1. The use of TBG.

Similar to CiteSpace, [ 8 ] the TBG contains 4 dimensions, namely, themes on the row, years on the column, bubbles sized by article citation, and colors by group. The way to draw the TBG is shown in Supplemental Digital Content 2, https://links.lww.com/MD/I44 .

2.2.4.2. The use of bootstrapping method and forest plot.

A forest plot was used to compare the mean citations of each theme between the 2 groups using the standard mean difference. The bootstrapping method was used to compute the mean and standard error of citations for each theme in each group. The standard deviation (SD) was then obtained by using Equation 5.

where n is the sample size for the theme in either Top100 or Nontop100; the way to draw the forest plot is shown in Supplemental Digital Content 2, https://links.lww.com/MD/I44 .

The bootstrapping method [ 15 , 31 , 32 ] was performed to verify the difference in mean citations for each theme between the groups. A total of 1000 medians were retrieved from random samples of 100 repetitions of median values for each theme in each group. Thus, the mean and 95% CI were obtained to compare differences in mean citations for each theme between groups by inspecting whether the two 95% CI bands were not overlaid.

2.3. Creating dashboards on google maps

All graphs were drawn by author-made modules in Excel (Microsoft Corp). We created pages of HTML used for Google Maps. All relevant CJAL scores for each member can be linked to dashboards on Google Maps. The way to draw visualization involved in this study is deposited in Supplemental Digital Content 2, https://links.lww.com/MD/I44 . The bootstrapping method involved with a module is provided in Supplemental Digital Content 3, https://links.lww.com/MD/I45 . The study flowchart is shown in Figure 1 .

3.1. Descriptive statistics

A significant rise in publications regarding bibliometrics in PubMed is evident in Figure 2 . We can expect that articles in 2022 would be higher than those in 2021 based on the exponential trend.

In Figure 3 , we present a list of the top 13 journals in both the Top100 and Nontop100 groups. There are 9 and 7 articles in the journal of Medicine , respectively, which rank top in the 2 Top100 and Nontop100 groups.

When the mean citations are taken into account, the journal World Psychiatry ranks at the bottom of Figure 4 , with 134 mean citations in Nontop100. As shown at the top of Figure 4 , the journal of Medicine has mean citations of 9.8 for Top100 articles and 24.7 for Nontop100 articles.

3.2. Research achievements in Top100 and Nontop100

There were 18 citations in total for the 2 sets of the 50 most cited articles (range 1-134; 5 and 26.5 for Top100 and Nontop100, respectively).

The dominant entities in the Top100 are China, Hallym University (South Korea), the medicine department, and the author Vincenzo Montinaro from Italy, with CJAL scores of 69.86, 12.88, 18.08, and 7.80, respectively, in countries, institutes, departments, and authors.

The dominant entities in Nontop100 are China, Huazhong University Science & Technology (China), the medicine department, and the author Dennis F Tompson from the US, with CJAL scores of 77.74, 18.60, 18.08, and 15.60, respectively, in countries, institutes, departments, and authors.

3.3. Thematic analysis and comparison of major themes between groups

As shown in Figure 5 , eleven themes were extracted from the coword analysis of these 100 articles. According to the chord diagram, the majority of keywords are from the theme of citation-analysis, followed by citation-classics and h-index. Colors are used to identify themes. It is important to note that only the top 3 keywords for each cluster are displayed in the chord diagram.

The majority of major keywords in proportional counts observed in themes are in favor of Nontop100, as shown in Figure 6 . Only 2 of citation analysis and information science are in favor of Top100.

3.4. Comparison of major themes between groups

A significant difference in mean citations was observed between the 2 groups of Top100 and Nontop100 based on the bootstrapping method (3, 95% CI: [1.18, 4.82]; 26.5, 95% CI: [23.82,29.18], P  < .001). Readers are invited to scan the QR code and click on the bubble of interest to examine the details about the article information on PubMed.

The 11 themes were clustered using coword analysis and applied to a TBG, which is composed of 4 dimensions: themes, years, citations and groups of articles, as shown in Figure 7 .

3.5. Online dashboards shown on google maps

All the QR codes in Figures are linked to the dashboards. [ 33–39 ] Readers are suggested to examine the displayed dashboards on Google Maps.

4. Discussion

4.1. principal findings.

We observed that there were 18 citations in total for the 2 sets of 50 most cited articles (range 1–134; 5 and 26.5 for Top100 and Nontop100, respectively). A significant difference in mean citations was observed between the 2 groups of Top100 and Nontop100 based on the bootstrapping method (3, 95% CI: [1.18, 4.82]; 26.5, 95% CI: [23.82, 29.18], P  < .001). The eleven themes were clustered using coword analysis and applied to a TBG, which is composed of 4 dimensions: themes, years, citations, and groups of articles. Among the 2 groups, the majority of articles were published in the journal of Medicine (Baltimore ), with 9 and 7, respectively.

Accordingly, the hypothesis that the Top100 articles have fewer citations than the Nontop100 articles is confirmed.

4.2. Additional information

Articles with a higher IF have usually been associated with the publication of reviews and original articles rather than case reports. [ 19–22 ] The statement that case reports are rarely cited is questionable, since the number of citations that a case report receives are highly dependent on the content of the report, the type of publication, the journal, and even the topic of the article. For instance, an article classified as a case-report type, submitted on March 18, 2020, and titled “A first case of meningitis and encephalitis associated with SARS-Coronavirus-2” received 1138 and 423 citations in WoS and PubMed, respectively, [ 40 ] during the past 2 years and meeting the main strengths noted in accepted manuscripts as the importance “or timeliness” of the problem studies, the quality of the writing, and the soundness of the study design. [ 41 , 42 ]

It is important to note that most Top100 articles using software (e.g., CiteSpace [ 8 ] and VOSviewer [ 43 ] ) with routine reports on article characteristics are case-study types that have low reader interest due to their focus on a particular topic “fishing expeditions in data and conducting systematic reviews that do not provide impactful findings.” [ 44 ] These articles are thus rarely cited.

The best way to increase citations for Top100 articles is to clearly explain or investigate their novel methodology rather than listing only their entity raking. There is a reason for this: readers are interested in articles that provide concise hypothesis and new insight or significant contribution to the field. [ 45 ] For instance, the article would be more interesting if readers were provided with adequate information (e.g., how to conduct this study with visualizations as Supplemental Digital Content 2, https://links.lww.com/MD/I44 in this study) regarding replication of the study.

There have been over 400 publications in PubMed with the titles of 100 top-cited articles. [ 2 ] Most of the articles ranked among the top 100 in citations were published in the Journal of Medicine (Baltimore ), making it the leading journal in this area.

The characteristics of 100 top-cited articles are commonly visualized with 3 categories of information on descriptive statistics, research domain, and research achievement (RA). [ 8 , 44–49 ] Some studies [ 8 , 44–48 , 49 ] have applied citation prediction to predict article citations based on the mean citations of article keywords, but the themes have not been definitely classified and clearly visualized through Equation 4. [ 9 ]

Additionally, many articles include many Tables and graphs in bibliometrics without applying radar plots [ 23 ] and chord diagrams [ 29 ] to condense information of interest for readers, as we did in Figures 5 , 8 , and 9 , particularly with the TBG in Figure 7 and forest plots in Figures 6 and 10 to display all 100 articles on a dashboard and save article spaces when compared to those with 100 and 50 articles listed in their studies [ 50 , 51 ] or with 42 Tables and graphs in an article. [ 52 ]

4.3. Most cited articles in Top100 and Nontop100

The article cited 32 times was authored by Yeung (Hong Kong) et al [ 53 ] and classified it as the theme of VOSviewer in Top100. Based on data provided by WoS in this study, the 100 most-cited articles relevant to neuroscience were identified and characterized. The 100 most-cited articles were mostly research articles published from 1996 to 2000. Stephen M. Smith and Science each had the largest share of these articles. Thirty-seven out of the 100 most-cited articles were interlinked via citations of 1 another, and 41 out of 63 non-interlinked articles could also be categorized under the above 5 topics. It is worth noting that there is no such information regarding VOSviewer that could be directly related to the theme of VOSviewer in the abstract of this study. In contrast, only keywords plus of brain and others were indexed in this article. The article is classified as VOSviewer that could be found in context instead, which is the feature of the current study using coword analysis to extract article theme via Equation 4. [ 9 ]

The article cited 134 times was authored by Fusar-Poli, Paolo (UK) [ 54 ] and classified as the theme of classification in Nontop100. Current psychiatric classification is based on ICD/DSM categorical diagnoses, and a promising alternative has been put forward as the “transdiagnostic” approach. A multistep Web of Science literature search was performed to identify all studies that used the word “transdiagnostic” in their title up to May 5, 2018. A total of 111 studies were included, and the quality of the studies was generally low. The conceptual analysis showed that transdiagnostic approaches are grounded more on rediscoveries than on true innovations and are affected by conceptual biases.

A review of 4 productive authors (Lutz Bornmann, Yuh-Shan Ho, Giovanni Abramo, and Ciriaco Andrea D’angelo) with more than 100 bibliometric articles indexed in WoS, with mean citations of 37.6 and median citations of 19. There were no articles entitled with 100 top-cited found in their publications.

4.4. Implications and changes

Chord diagrams [ 29 ] were used to visualize dynamics related to contraceptive use and to bring data into practice. The dashboards (e.g., in Figure 5 ) [ 36 ] provide an easy way to visualize the relationship between elements in themes. As a result of chord diagrams, we gain a clear understanding of the relationship between 2 or more entities (e.g., the themes and clusters shown at the top of Figure 5 ), something that is rare in bibliometric analysis. Supplemental Digital Content 2, https://links.lww.com/MD/I44 provides the R code for reproducing the chord diagram.

Furthermore, chord diagrams could be used by network diagrams to clearly illustrate their network relationships, with more effective representations than the traditional displays, as illustrated at the top of Figure 5 .

There are 4 factors that contribute to the CJAL score: subject category, journal impact factor, authorship in positions on the article byline, and article citations. The evaluation of individual RAs has traditionally been based on bibliometric metrics (e.g., h-/x-/Y-/ht-index [ 15–18 ] ). These metrics have a number of disadvantages, such as assuming that all coauthors contributed equally to the article, regardless of the type of document or journal impact factor. The CJAL score [ 24 ] bridges the gap between publications and citations when evaluating the RA beyond those bibliometric metrics.

The current study on Top100 and Nontop100 represents the first attempt to compare the difference in mean citations of articles. The dashboard-type 4-quadrant radar plots depicted in Figures 4 and 8 provide a summary of 4 important entities each in the 2 groups rather than Fables and graphs in traditional bibliometrics. A unique and modern approach of the 4-quadrant radar plots has been applied to bibliometrics before. [ 23 ] It is possible to advance bibliometric analysis in this manner in the future.

As seen from the CJAL score, China dominates the bibliometric studies. This study differs from many traditional bibliographical studies in that the publications are computed based on the first and corresponding authors rather than just the first author, as in traditional bibliometric studies. In this study, the dominant entities in the Top100 are China, Hallym University (South Korea), the medicine department, and the author Vincenzo Montinaro from Italy, with CJAL scores of 69.86, 12.88, 18.08, and 7.80, respectively. It is therefore recommended that the CJAL score be used to measure RAs in bibliometric research, particularly when using a radar plot to condense information at a glance.

Traditional bibliographical studies with descriptive statistics, research domain, and RA provided us with a clear understanding of what distinguishes a discipline or field (or topic) from the others and provided insight for physicians and researchers. However, 2 main concerns were frequently overlooked. In such cases, a simplified visualization of all relevant entities is lacking (as in Figures 8 and 9 ), and a method for displaying all 100 top-cited articles at a glance using the TBG is missing, as shown in Figure 5 .

4.5. Limitations and suggestions

A number of issues need to be examined in further research. The first concern is that the Rstudio package used for drawing the chord diagram is not unique and irreplaceable. It is also possible to draw them using several other software packages.

Second, this study uses Google Maps to display dashboards. Since Google Maps requires a paid project key, these installments are not free. It may therefore be difficult for other authors to replicate the usage within a short period of time.

Third, calculating the CJAL score requires considerable computation. The development of this technology will require dedicated software in the future.

Fourth, it has been recommended that the radar plot and CJAL score be combined to simplify article spaces in comparison to other traditional bibliographical studies with many Tables and graphs. However, the RAs are determined by other factors that must also be considered when drawing radar plots in the future.

Fifth, to present the study results, 6 typical visualizations were used, including charts with bars, pyramids, forests, clusters, chords, and bubbles. It is common for bibliometric analysis to be represented visually in a variety of ways. For future studies, it is recommended that more appropriate visual displays be used to facilitate readers’ understanding of the study features.

Sixth, article citations should not be solely determined by IF as we compared themes in citations between Top100 and Nontop100 in this study. According to some studies, [ 55 , 56 ] IF should not be used alone as a criterion for evaluating journals. A better assessment of their significance and importance in particular disciplines can be achieved by using Eigenfactor Score, Article Influence Score, and Cited Half-life.

Finally, even though 100 top-cited articles were extracted primarily from WoS and PubMed, the results were different for articles retrieved from other databases (such as Google Scholar and Scopus), while other types of research fields (e.g., engineering and agriculture) were also considered. Future studies should examine whether the Top100 articles have fewer citations than Nontop10 0 articles, as found in this study.

5. Conclusion

A breakthrough was achieved by comparing mean citations in articles of Top100 and Nontop100, which included chord diagrams and the TBG with a demonstration of theme classification. It is possible to match article themes with author-collaboration clusters (e.g., countries, institutes, and authors) as cluster names labeled in CiteSpace. In future studies, a TBG with 4 dimensions should be applied to 100 top-cited articles in bibliometrics.

Acknowledgments

We thank Enago ( www.enago.tw ) for the English language review of this manuscript.

Author contributions

WT and TWC initiated the research, collected data, conducted the analysis, and wrote the manuscript. WC contributed to the design of the study and provided critical reviews of the manuscript, and WC and TWC contributed to the interpretation of the results.

Conceptualization: Wan-Ting Hsieh.

Investigation: Willy Chou.

Methodology: Tsair-Wei Chien.

Abbreviations:

• Cited Here |
• Google Scholar

100 top-cited articles; bibliometrics; coword analysis; PubMed; temporal bubble graph; Web of Science

Supplemental Digital Content

• MD_1_1_2022_11_12_CHIEN_MD-D-22-06777_SDC1.xlsx; [Excel] (175 KB)
• MD_1_1_2022_11_12_CHIEN_MD-D-22-06777_SDC2.pdf; [PDF] (1.24 MB)
• MD_1_1_2022_11_12_CHIEN_MD-D-22-06777_SDC3.xlsm; [Other] (1.41 MB)
• + Favorites
• View in Gallery

Readers Of this Article Also Read

Correlation between body mass index and periprosthetic joint infection..., cemented versus uncemented hemiarthroplasty for displaced femoral neck..., total en bloc spondylectomy for primary tumors of the lumbar spine, a study on dynamic monitoring, components, and risk factors of embolism during....

• Search Menu
• Sign in through your institution
• Advance Articles
• Editor's Choice
• Supplements
• Author Guidelines
• Submission Site
• Why Publish With Us?
• Open Access
• About Neuro-Oncology
• About the Society for Neuro-Oncology
• About the Japan Society for Neuro-Oncology
• About the European Association of Neuro-Oncology
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Terms and Conditions
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

• < Previous

OTHR-23. PATIENT DESIRE TO SHARE DATA AND PARTICIPATE IN RESEARCH IN THE DIPG/DMG COMMUNITY

• Article contents
• Figures & tables
• Supplementary Data

Nicole Lyons, Lisa Ward, OTHR-23. PATIENT DESIRE TO SHARE DATA AND PARTICIPATE IN RESEARCH IN THE DIPG/DMG COMMUNITY, Neuro-Oncology , Volume 26, Issue Supplement_4, June 2024, Page 0, https://doi.org/10.1093/neuonc/noae064.693

• Permissions Icon Permissions

Direct and reciprocal communication between patient communities and their research and care teams is essential to building trust and ensuring that all essential information is shared. Patient participation in data collection of any sort is often challenging, as patients may be hesitant or unmotivated to share their information. The DIPG/DMG patient community faces the opposite challenge. Because of the frustration of the lack of treatment options and the established trust that exists between patient families and research professionals, DIPG/DMG patient families are both willing and eager to share any information they can. Patient families have seen how their knowledge has helped other families navigate impossible situations and care teams understand how to better support their patients. The information families eagerly provide is crucial to advancing treatment and research. Patient families have repeatedly expressed willingness to participate in quality of life assessments, repeated surveys during and after treatment, and provide their medical and biological information in the hopes that it can aid research. There is a need for a centralized manner for all of this data to be shared by the patient and accessed by care teams and researchers. Patient-led initiatives are addressing this by creating a network of resources. This network helps to guide other patient families through the data-sharing process, address concerns, and find ways to provide what is needed to push research and treatments ahead. There are many benefits to having a centralized data ecosystem for researchers and care teams, such as standardizing data collection and ease of analysis. In the future, this centralized hub could inform personalized care, making options such as n-of-1 trials and matching patients to optimal clinical trials possible in a manner that is effective and efficient.

Email alerts

Citing articles via, looking for your next opportunity.

• Recommend to your Library
• Society for Neuro-Oncology
• SNO Twitter
• European Association of Neuro-Oncology
• The Japan Society for Neuro-Oncology

Affiliations

• Online ISSN 1523-5866
• Print ISSN 1522-8517
• Copyright © 2024 Society for Neuro-Oncology
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Critically Thinking About “Citing Up”

Considering credibility, familiarity, and patience when citing research..

Posted June 12, 2024 | Reviewed by Davia Sills

• What Is a Career
• Find a career counselor near me
• Newer studies tend not to be referenced as frequently as more established scientific research.
• Credibility, familiarity, and implicit bias all play a role in which research gets cited more often.
• With patience and hard work, researchers can build their reputations as worthy of citation.

I came across an interesting social media post recently in preparation for a professional skills development workshop that I was presenting. The post discussed how academics tend to “cite up” in terms of referencing older, more famous scholars relative to more junior researchers. I thought about this proposition in light of my own citation strategies and knowledge of bibliometrics and concluded that this statement is likely true, but probably not for any explicit bias against junior researchers, as some might posit.

First and foremost, we must consider the purpose of citing research—to represent a source of evidence and indicate that someone didn’t just make up what they’re saying. It’s been established in previous work, and we pay that research kudos to further our argument in context. References are also useful for “ cutting a long story short”—one can cite another’s work that can more fully explain a concept without having to reiterate the whole thing. When I use a reference in my arguments, given that I’m trying to convince the reader of my point, I want to use the most credible source(s) that I can find.

Possible reasons for “citing up”

If Author A is at the apex of credible sources in the field, I’m going to cite them where appropriate. Indeed, if I was reviewing a relevant paper and didn’t see Author A cited, I might be concerned. Of course, one can include multiple citations, but perhaps the reason why more junior or early career researchers are not cited (relative to the Author As out there) is that other researchers may not be as familiar with the early career researchers”—Author Es’—research.

Maybe the citing researcher remembers the research but not the name of the author. Obviously, Author E’s work hasn’t seemed to “stick” yet, maybe because they’re yet to make a bigger impact in the field. Sure, that’s largely the citing researcher’s issue for not having better organized their reading and note-taking, but simply, it’s also an issue of accessibility. If a researcher can’t remember Author E’s name in this context, the credibility of Author A will more than suffice. “Citing up” is not a slight here; it’s just that Author E’s contribution might not be that impactful, accessible, or memorable to a more established researcher. Moreover, I must admit there might be a level of laziness here.

For example, the scenario above is context-dependent. If I can’t remember Author E, that’s fine; I have Author A to cite. However, if Author E is the only appropriate citation, the citing strategy will change. If I know a claim is fundamental to my rationale but I can’t remember where it came from, despite knowing I’ve seen solid evidence for it in the past, I will search for Author E’s paper until I find it (because I have to if there’s no Author A to rely on). This might take time and effort.

I can imagine that some researchers will be reading this and thinking, “Surely, others are reading the new literature and taking notes as they go along or maybe even writing the rationale as they engage the new literature.” Ideally, this should be the case; indeed, it’s a handy way of keeping up-to-date with the literature. However, this does not always happen.

I imagine more established researchers in a field are “familiar enough” with it to write a rationale without having to look up papers every few lines and, instead, are more likely to write what they know. Such is human nature. When they eventually get some free time, they might dedicate a few hours to reading recently published papers. I’m also aware that some researchers are better at this than others. Obviously, this is worrying in the realm of research—perhaps more worrying altogether than the issue of “citing-up.”

With that, what are the chances that a researcher has read every paper in their field? Slim-to-none. Given the exponential increase in the amount of information available to people in the past 25 years and, likewise, the increase in the amount of Ph.D. degrees awarded and research being conducted, being up-to-date with all work in a field just isn’t feasible.

So, maybe “lazy” is unfair in context. Maybe these researchers are indeed reading as much as they can, but because the amount that’s feasible is finite relative to the seemingly endless new research that’s coming out, they might be “pickier” in what they read; for example, prioritizing known and credible researchers in their field. So, there’s a good chance that when only Author A is cited regarding a particular finding, it’s quite possible that it’s because the citing researcher has never even heard of Author E’s paper, let alone read it.

Takeaways for early-career researchers

“New” papers—regardless of when and by whom they’re read, need “sticking power,” and by that, I mean that the research is well-conducted: It is well-written, and interesting food for thought is provided. I compile and read new papers every month—maybe one per session has any sticking power—and that’s not because I’m some kind of research snob; rather, it’s the case that much of it failed some of the criteria above. With that, if the paper had well-conducted research, was well-written, and provided either something novel or some food for thought, then regardless of familiarity, this paper (and its author) would be on my radar for the future. So, just as much as older researchers may be set in their bibliographies or “lazy” referencing, it is most definitely up to younger researchers to publish impactful work.

I completely understand how this is frustrating for early career researchers. I was there once, too. Even though it’s been well over 10 years since I received my Ph.D., I still find myself trying to make the aforementioned impact necessary to be considered one of those “A” researchers in the field. Of course, I get annoyed when I see missed opportunities for other researchers to cite my work. But I’m realistic enough to recognize that maybe they have not come across my work, I have not made a large enough impact for it to be noticed, or the research they did cite was sufficient to make their point. I don’t take it personally, and neither should young researchers. Their time will come, but they must be patient.

Consider the research by Morris, Wooding, and Grant (2011), where it was suggested that it takes approximately 17 years on average for health research implementation from “bench to bedside.” That’s a long time for “research to be realized.” I know citations are different and should be more visible quicker in the land of research, but the same logic applies. Patience—and continued hard work (i.e., to advance one’s research acumen)—are necessary for citation success.

Again, I don’t think that “citing up” is consciously done to slight early career academics; researchers are not conspiring against their junior colleagues—at least, not in my field. If anything, they want to see them and their field flourish. Instead, I think it’s more likely that this issue boils down to an implicit bias (which we all face on a day-to-day basis) toward what we know as familiar, accessible, and credible.

Morris, Z. S., Wooding, S., & Grant, J. (2011). The answer is 17 years, what is the question: understanding time lags in translational research. Journal of the royal society of medicine , 104 (12), 510-520.

Christopher Dwyer, Ph.D., is a lecturer at the Technological University of the Shannon in Athlone, Ireland.

• Find a Therapist
• Find a Treatment Center
• Find a Psychiatrist
• Find a Support Group
• Find Online Therapy
• United States
• Brooklyn, NY
• Chicago, IL
• Houston, TX
• Los Angeles, CA
• New York, NY
• Portland, OR
• San Diego, CA
• San Francisco, CA
• Seattle, WA
• Washington, DC
• Asperger's
• Bipolar Disorder
• Chronic Pain
• Eating Disorders
• Passive Aggression
• Personality
• Goal Setting
• Positive Psychology
• Stopping Smoking
• Low Sexual Desire
• Relationships
• Child Development
• Self Tests NEW
• Therapy Center
• Diagnosis Dictionary
• Types of Therapy

At any moment, someone’s aggravating behavior or our own bad luck can set us off on an emotional spiral that threatens to derail our entire day. Here’s how we can face our triggers with less reactivity so that we can get on with our lives.

• Emotional Intelligence
• Gaslighting
• Affective Forecasting
• Neuroscience

Twenty-eight Purdue HHS researchers named in 2023 list of world’s most-cited scientists

Written by: Denise Buhrmester, [email protected]

Purdue HHS data provided by: Jordan Gooch, senior data analyst, External Data Analytics, [email protected]

In a database compiled by Stanford University in partnership with Elsevier , 28 scientists and scholars in the Purdue University College of Health and Human Sciences (HHS) appear among the top 2% of the most-cited researchers across seven disciplines. Inclusion in the database means the individual met a variety of metrics that place them among the top 2% of researchers based on citations for 2022.

In addition to appearing on Elsevier’s list for 2022 citations, one HHS faculty member — Louis Tay , the William C. Byham Professor in Psychological Sciences — also appeared on Clarivate’s 2023 list of Highly Cited Researchers , which identifies scholars whose work has been cited most often in papers over the past decade. Those chosen for the 2023 list authored studies that rank in the top 1% of scholarly citations worldwide within their disciplines.

The following HHS researchers (current, in alphabetical order) appeared on Elsevier’s list of the top 2% of the world’s most-cited researchers:

• Thomas Berndt , professor of psychological sciences
• Liping Cai , professor of hospitality and tourism management
• Wayne Campbell , professor of nutrition science
• Jason Cannon , professor of health sciences
• Heather Eicher-Miller , professor of nutrition science
• Laura Elenbaas , assistant professor of human development and family science
• Dan Foti , associate professor of psychological sciences
• Jennifer Freeman , professor of health sciences
• Elliot Friedman , William and Sally Berner Hanley Professor of Gerontology
• SooCheong (Shawn) Jang , professor of hospitality and tourism management
• Qing Jiang , professor of nutrition science
• Jeffrey Karpicke , James V. Bradley Professor of Psychological Sciences
• Ceridwyn King , professor of hospitality and tourism management
• Ananthanarayan Krishnan , professor of speech, language, and hearing sciences
• Xinran Lehto , professor of hospitality and tourism management
• Laurence Leonard , Rachel E. Stark Distinguished Professor of Speech, Language, and Hearing Sciences
• Shuang Liu , professor of health sciences
• Donald Lynam , distinguished professor of psychological sciences
• Richard Mattes , distinguished professor of nutrition science
• Catherine (Cammie) McBride , professor of human development and family science
• Laura Murray-Kolb , professor of nutrition science
• David Purpura , professor of human development and family science
• Thomas Redick , associate professor of psychological sciences
• Douglas Samuel , professor of psychological sciences
• Jonathan Shannahan , associate professor of health sciences
• Louis Tay , William C. Byham Professor of Industrial-Organizational Psychology
• Kipling Williams , distinguished professor of psychological sciences
• Wei Zheng , professor of health sciences

The database, updated October 2023, is available through the  Elsevier Data Repository .

About the Standford-Elsevier list Elsevier’s single-year data, from which this list was derived, pertains to citations received during the 2022 calendar year. Scientists are classified into 22 scientific fields and 174 subfields according to the standard Science-Metrix classification. The selection is based on the top 100,000 scientists by c-score (with and without self-citations) or a percentile rank of 2% or above in the subfield. Calculations were performed using all Scopus author profiles as of October 1, 2023. If an author is not on the list, the composite indicator value was not high enough to appear. Using the most recent single-year data provides an up-to-date snapshot of productivity and expertise across 22 disciplines. This list does not include emeritus researchers as well as those who have left Purdue University or are officially on leave. Some Purdue researchers may not be on the list because their Scopus account still has them affiliated with a former institution. Researchers can request  Scopus  update their ID to Purdue University.  Learn more .

Americans are embracing flexible work—and they want more of it

When the COVID-19 pandemic shuttered workplaces nationwide, society was plunged into an unplanned experiment in work from home. Nearly two-and-a-half years on, organizations worldwide have created new working norms  that acknowledge that flexible work is no longer a temporary pandemic response but an enduring feature of the modern working world.

About the survey

This article is based on a 25-minute, online-only Ipsos poll conducted on behalf of McKinsey between March 15 and April 18, 2022. A sample of 25,062 adults aged 18 and older from the continental United States, Alaska, and Hawaii was interviewed online in English and Spanish. To better reflect the population of the United States as a whole, post hoc weights were made to the population characteristics on gender, age, race/ethnicity, education, region, and metropolitan status. Given the limitations of online surveys, 1 “Internet surveys,” Pew Research Center. it is possible that biases were introduced because of undercoverage or nonresponse. People with lower incomes, less education, people living in rural areas, or people aged 65 and older are underrepresented among internet users and those with high-speed internet access.

The third edition of McKinsey’s American Opportunity Survey  provides us with data on how flexible work fits into the lives of a representative cross section of workers in the United States. McKinsey worked alongside the market-research firm Ipsos to query 25,000 Americans in spring 2022 (see sidebar, “About the survey”).

The most striking figure to emerge from this research is 58 percent. That’s the number of Americans who reported having the opportunity to work from home at least one day a week. 1 Many of the survey questions asked respondents about their ability or desire to “work from home.” “Work from home” is sometimes called “remote work,” while arrangements that allow for both remote and in-office work are often interchangeably labeled “hybrid” or “flexible” arrangements. We prefer the term flexible, which acknowledges that home is only one of the places where work can be accomplished and because it encompasses a variety of arrangements, whereas hybrid implies an even split between office and remote work. Thirty-five percent of respondents report having the option to work from home five days a week. What makes these numbers particularly notable is that respondents work in all kinds of jobs, in every part of the country and sector of the economy, including traditionally labeled “blue collar” jobs that might be expected to demand on-site labor as well as “white collar” professions.

About the authors

This article is a collaborative effort by André Dua , Kweilin Ellingrud , Phil Kirschner , Adrian Kwok, Ryan Luby, Rob Palter , and Sarah Pemberton as part of ongoing McKinsey research to understand the perceptions of and barriers to economic opportunity in America. The following represents the perspectives of McKinsey’s Real Estate and People & Organizational Performance Practices.

Another of the survey’s revelations: when people have the chance to work flexibly, 87 percent of them take it. This dynamic is widespread across demographics, occupations, and geographies. The flexible working world was born of a frenzied reaction to a sudden crisis but has remained as a desirable job feature for millions. This represents a tectonic shift in where, when, and how Americans want to work and are working.

The following six charts examine the following:

• the number of people offered flexible working arrangements either part- or full-time
• how many days a week employed people are offered and do work from home
• the gender, age, ethnicity, education level, and income of people working or desiring to work flexibly
• which occupations have the greatest number of remote workers and how many days a week they work remotely
• how highly employees rank flexible working arrangements as a reason to seek a new job
• impediments to working effectively for people who work remotely all the time, part of the time, or not at all

Flexible work’s implications for employees and employers—as well as for real estate, transit, and technology, to name a few sectors—are vast and nuanced and demand contemplation.

1. Thirty-five percent of job holders can work from home full-time, and 23 percent can do so part-time

A remarkable 58 percent of employed respondents—which, extrapolated from the representative sample, is equivalent to 92 million people from a cross section of jobs and employment types—report having the option to work from home for all or part of the week. After more than two years of observing remote work and predicting that flexible working would endure  after the acute phases of the COVID-19 pandemic, we view these data as a confirmation that there has been a major shift in the working world and in society itself.

We did not ask about flexible work in our American Opportunity Survey in past years, but an array of other studies indicate that flexible working has grown by anywhere from a third to tenfold since 2019. 1 Rachel Minkin et al., “How the coronavirus outbreak has—and hasn’t—changed the way Americans work,” Pew Research Center, December 9, 2020; “Telework during the COVID-19 pandemic: Estimates using the 2021 Business Response Survey,” US Bureau of Labor Statistics, Monthly Labor Review, March 2022.

Thirty-five percent of respondents say they can work from home full-time. Another 23 percent can work from home from one to four days a week. A mere 13 percent of employed respondents say they could work remotely at least some of the time but opt not to.

Forty-one percent of employed respondents don’t have the choice. This may be because not all work can be done remotely  or because employers simply demand on-site work. Given workers’ desire for flexibility, employers may have to explore ways to offer the flexibility employees want  to compete for talent effectively.

2. When offered, almost everyone takes the opportunity to work flexibly

The results of the survey showed that not only is flexible work popular, with 80 million Americans engaging in it (when the survey results are extrapolated to the wider population), but many want to work remotely for much of the week when given the choice.

Eighty-seven percent of workers offered at least some remote work embrace the opportunity and spend an average of three days a week working from home. People offered full-time flexible work spent a bit more time working remotely, on average, at 3.3 days a week. Interestingly, 12 percent of respondents whose employers only offer part-time or occasional remote work say that even they worked from home for five days a week. This contradiction appears indicative of a tension between how much flexibility employers offer and what employees demand .

3. Most employees want flexibility, but the averages hide the critical differences

There’s remarkable consistency among people of different genders, ethnicities, ages, and educational and income levels: the vast majority of those who can work from home do so. In fact, they just want more flexibility: although 58 percent of employed respondents say they can work from home at least part of the time, 65 percent of employed respondents say they would be willing to do so all the time.

However, the opportunity is not uniform: there was a large difference in the number of employed men who say they were offered remote-working opportunities (61 percent) and women (52 percent). At every income level, younger workers were more likely than older workers to report having work-from-home opportunities.

People who could but don’t work flexibly tend to be older (19 percent of 55- to 64-year-olds offered remote work didn’t take it, compared with 12 to 13 percent of younger workers) or have lower incomes (17 percent of those earning $25,000 to $74,999 per year who were offered remote work didn’t take it, compared with 10 percent of those earning over $75,000 a year). While some workers may choose to work on-site because they prefer the environment, others may feel compelled to because their home environments are not suitable, because they lack the skills and tools to work remotely productively, or because they believe there is an advantage to being on-site. Employers should be aware that different groups perceive and experience remote work differently and consider how flexible working fits with their diversity, equity, and inclusion strategies .

4. Most industries support some flexibility, but digital innovators demand it

The opportunity to work flexibly differs by industry and role within industries and has implications for companies competing for talent. For example, the vast majority of employed people in computer and mathematical occupations report having remote-work options, and 77 percent report being willing to work fully remotely. Because of rapid digital transformations across industries , even those with lower overall work-from-home patterns may find that the technologists they employ demand it.

A surprisingly broad array of professions offer remote-work arrangements. Half of respondents working in educational instruction and library occupations and 45 percent of healthcare practitioners and workers in technical occupations say they do some remote work, perhaps reflecting the rise of online education and telemedicine. Even food preparation and transportation professionals said they do some work from home.

5. Job seekers highly value having autonomy over where and when they work

The survey asked people if they had hunted for a job recently or were planning to hunt for one. Unsurprisingly, the most common rationale for a job hunt was a desire for greater pay or more hours, followed by a search for better career opportunities. The third-most-popular reason was looking for a flexible working arrangement.

Prior McKinsey research has shown that for those that left the workforce during the early phases of the COVID-19 pandemic, workplace flexibility was a top reason that they accepted new jobs . Employers should be aware that when a candidate is deciding between job offers with similar compensation, the opportunity to work flexibly can become the deciding factor.

6. Employees working flexibly report obstacles to peak performance

The survey asked respondents to identify what made it hard to perform their jobs effectively. Those working in a flexible model were most likely to report multiple obstacles, followed by those working fully remotely, and then by those working in the office. Our research doesn’t illuminate the cause and effect here: it could be that people who face barriers are more likely to spend some time working from home. It could also be that workers who experience both on-site and at-home work are exposed to the challenges of each and the costs of regularly switching contexts.

Some obstacles were reported at much higher rates by specific groups: for example, about 55 percent of 18- to 34-year-olds offered the option to work fully remotely say mental-health issues  impacted their ability to perform effectively, though only 17 percent of people aged 55 to 64 said the same. Workers with children at home  who were offered full-time remote-work options were far more likely than their peers without children to report that problems with physical health or a hostile work environment had a moderate or major impact on their job.

The results of the American Opportunity Survey reflect sweeping changes in the US workforce, including the equivalent of 92 million workers offered flexible work, 80 million workers engaged in flexible work, and a large number of respondents citing a search for flexible work as a major motivator to find a new job.

Competition for top performers and digital innovators demands that employers understand how much flexibility their talent pool is accustomed to and expects. Employers are wise to invest in technology, adapt policies, and train employees to create workplaces that integrate people working remotely and on-site (without overcompensating by requiring that workers spend too much time in video meetings ). The survey results identify obstacles to optimal performance that underscore a need for employers to support workers with issues that interfere with effective work. Companies will want to be thoughtful about which roles can be done partly or fully remotely—and be open to the idea that there could be more of these than is immediately apparent. Employers can define the right metrics and track them to make sure the new flexible model is working.

At a more macro level, a world in which millions of people no longer routinely commute has meaningful implications for the commercial core in big urban centers and for commercial real estate overall. Likewise, such a world implies a different calculus for where Americans will live and what types of homes they will occupy. As technology emerges that eliminates the residual barriers to more distributed and asynchronous work, it could become possible to move more types of jobs overseas, with potentially significant consequences.

In time, the full impact of flexible working will be revealed. Meanwhile, these data give us early insight into how the working world is evolving.

For more on the imperative for flexible work and how organizations can respond, please see McKinsey.com/featured-insights/ Future-of-the-workplace .

André Dua is a senior partner in McKinsey’s Miami office;  Kweilin Ellingrud is a senior partner in the Minneapolis office;  Phil Kirschner is a senior expert in the New York office, where Adrian Kwok is an associate partner and Ryan Luby is a senior expert; Rob Palter is a senior partner in the Toronto office; and Sarah Pemberton is a manager in the Hong Kong office.

Explore a career with us

Related articles.

Hybrid work: Making it fit with your diversity, equity, and inclusion strategy

Workplace real estate in the COVID-19 era: From cost center to competitive advantage

This paper is in the following e-collection/theme issue:

Published on 18.6.2024 in Vol 26 (2024)

Identification of Ethical Issues and Practice Recommendations Regarding the Use of Robotic Coaching Solutions for Older Adults: Narrative Review

Authors of this article:

• Cécilia Palmier 1, 2 * , MSc   ; 
• Anne-Sophie Rigaud 1, 2 * , Prof Dr Med   ; 
• Toshimi Ogawa 3 , PhD   ; 
• Rainer Wieching 4 , Prof Dr   ; 
• Sébastien Dacunha 1, 2 * , MSc   ; 
• Federico Barbarossa 5 , MEng   ; 
• Vera Stara 5 , PhD   ; 
• Roberta Bevilacqua 5 , MSc   ; 
• Maribel Pino 1, 2 * , PhD  

1 Maladie d’Alzheimer, Université de Paris, Paris, France

2 Service de Gériatrie 1 & 2, Hôpital Broca, Assistance Publique - Hôpitaux de Paris, Paris, France

3 Smart-Aging Research Center, Tohoku University, Sendai, Japan

4 Institute for New Media & Information Systems, University of Siegen, Siegen, Germany

5 Scientific Direction, Istituto Nazionale di Ricovero e Cura per Anziani, Ancona, Italy

*these authors contributed equally

Corresponding Author:

Anne-Sophie Rigaud, Prof Dr Med

Service de Gériatrie 1 & 2

Hôpital Broca

Assistance Publique - Hôpitaux de Paris

54 rue Pascal

Paris, 75013

Phone: 33 144083503

Fax:33 144083510

Email: [email protected]

Background: Technological advances in robotics, artificial intelligence, cognitive algorithms, and internet-based coaches have contributed to the development of devices capable of responding to some of the challenges resulting from demographic aging. Numerous studies have explored the use of robotic coaching solutions (RCSs) for supporting healthy behaviors in older adults and have shown their benefits regarding the quality of life and functional independence of older adults at home. However, the use of RCSs by individuals who are potentially vulnerable raises many ethical questions. Establishing an ethical framework to guide the development, use, and evaluation practices regarding RCSs for older adults seems highly pertinent.

Objective: The objective of this paper was to highlight the ethical issues related to the use of RCSs for health care purposes among older adults and draft recommendations for researchers and health care professionals interested in using RCSs for older adults.

Methods: We conducted a narrative review of the literature to identify publications including an analysis of the ethical dimension and recommendations regarding the use of RCSs for older adults. We used a qualitative analysis methodology inspired by a Health Technology Assessment model. We included all article types such as theoretical papers, research studies, and reviews dealing with ethical issues or recommendations for the implementation of these RCSs in a general population, particularly among older adults, in the health care sector and published after 2011 in either English or French. The review was performed between August and December 2021 using the PubMed, CINAHL, Embase, Scopus, Web of Science, IEEE Explore, SpringerLink, and PsycINFO databases. Selected publications were analyzed using the European Network of Health Technology Assessment Core Model (version 3.0) around 5 ethical topics: benefit-harm balance, autonomy, privacy, justice and equity, and legislation.

Results: In the 25 publications analyzed, the most cited ethical concerns were the risk of accidents, lack of reliability, loss of control, risk of deception, risk of social isolation, data confidentiality, and liability in case of safety problems. Recommendations included collecting the opinion of target users, collecting their consent, and training professionals in the use of RCSs. Proper data management, anonymization, and encryption appeared to be essential to protect RCS users’ personal data.

Conclusions: Our analysis supports the interest in using RCSs for older adults because of their potential contribution to individuals’ quality of life and well-being. This analysis highlights many ethical issues linked to the use of RCSs for health-related goals. Future studies should consider the organizational consequences of the implementation of RCSs and the influence of cultural and socioeconomic specificities of the context of experimentation. We suggest implementing a scalable ethical and regulatory framework to accompany the development and implementation of RCSs for various aspects related to the technology, individual, or legal aspects.

Introduction

Challenges associated to population aging.

Technological and medical advances have led to a demographic shift in the population, with the number of older adults constantly increasing. According to the United Nations [ 1 ], older adults (aged 60-65 years) will represent 16% of the world’s population in 2050. In addition, life expectancy is increasing, from 64.2 years in 1990 to 72.6 years in 2019, and is expected to reach 77.1 years in 2050 [ 1 ]. However, there is a wide diversity of health conditions among older adults. The health status of older adults is dependent on multiple factors, including nonmodifiable genetic factors and environmental factors, such as lifestyle [ 2 ]. Thus, older adults represent a very heterogeneous population with multiple and diverse needs and desires. With advancing age, the loss of functional independence; frailty; and other health diseases such as cardiovascular problems, cancers, osteoarthritis, osteoporosis, or major neurocognitive disorders may appear [ 3 - 5 ]. Among age-related conditions, major neurocognitive disorders (eg, Alzheimer disease) receive particular attention due to the increasing prevalence of these diseases [ 6 ].

The aging population is not only a public health issue but also a socioeconomic one. To face this challenge, it is important to develop preventive measures to support active and healthy aging and to preserve the independent functioning and quality of life of older adults. The adoption of healthy behaviors can help prevent or delay the onset of pathologies or treat them if detected early [ 7 ].

The Use of Technologies for Older Adults

Preventive health measures can be supported through new technologies, such as robotic coaching solutions (RCSs) that promote healthy aging among older adults [ 8 , 9 ]. RCSs have been defined as personalized systems that continuously monitor the activities and environment of the user and provide them with timely health-related advice and interventions [ 10 - 12 ]. These systems can help users define and achieve different health-oriented goals [ 12 ].

RCSs may encompass artificial intelligence (AI) technologies that can analyze user data, personalize coaching programs, and adapt recommendations based on each individual’s needs [ 1 , 13 - 19 ]. RCSs can involve robots equipped with sensors such as cameras, microphones, or motion sensors to collect real-time data about the user, AI, and programming that enables their interaction with users [ 20 , 21 ]. These technologies are often equipped with voice and visual recognition and learning capabilities [ 20 , 21 ]. They can benefit from advanced natural language processing techniques, which allow for understanding of the user’s input, facilitating natural and effective communication [ 22 ]. RCSs can offer guidance, support, and feedback based on preprogrammed information or real-time data analysis. These data can inform coaching strategies and allow RCSs to provide users with relevant feedback [ 8 ].

RCSs can also encompass a virtual agent, which refers to a computer program or an AI system that interacts with users in a manner that simulates human conversation [ 14 , 18 , 23 ]. A virtual agent is an animated character capable of adopting a social behavior mimicking that of humans to encourage the users to make changes in their habits [ 14 ]. Virtual agents might take the form of a chatbot, voice assistant, or other AI-driven communication system [ 14 ]. Biometric monitoring devices to track physiological data such as heart rate, sleep patterns, or stress levels can also be included in RCSs [ 8 , 20 , 21 ]. These data can contribute to the configuration of personalized coaching plans. RCSs can also encompass advanced data analytics that can process large data sets generated by users’ interactions and behaviors. This functionality helps in identifying patterns, trends, and areas for improvement in coaching strategies [ 24 ]. Integrating Internet-of-Things devices in RCSs can provide additional data points about a user’s environment, lifestyle, or habits, thus contributing to a personalized coaching approach [ 25 ].

Health-oriented RCSs could enable users to lead a healthy lifestyle, by identifying needs and goals and providing appropriate risk predictions and individualized recommendations [ 12 , 26 - 28 ]. There are RCSs dedicated to a particular domain, such as physical activity or motor rehabilitation [ 9 , 16 ]. Others may have the objective of promoting independent and healthy aging [ 29 ].

Promoting active and healthy aging can allow older adults to maintain their independence and continue to live at home [ 4 , 30 ], which is a wish of many [ 3 ]. This intervention could also help to reduce the need for assistance, usually provided by informal caregivers and health professionals [ 4 , 19 , 30 - 33 ]. Furthermore, RCSs could lead to a reduction in individual and collective health care expenses [ 4 , 32 , 34 ] by easing access to health and social care interventions to a wide population, including hard-to-reach (eg, geographically isolated) individuals. However, although the use of health-related RCSs could have many benefits, several ethical issues arise with their development and implementation in human environments [ 3 , 35 - 38 ].

An Ethical Framework for the Use of Technologies for Older Adults

For RCSs to contribute to active and healthy aging, it is important that all the stakeholders (engineers, geriatricians, psychologists, etc) involved in their design and implementation refer to an ethical framework [ 3 , 38 ]. It is also important to inform society (politicians and legal experts) about such an extension of technology in people’s lives (private, professional, medicosocial, and commercial context), so that we can create a legal framework for the use of these technologies. An analysis of the way in which ethical and legal dimensions have been addressed by studies, in the field of RCSs for health care, seems useful to support the key actors in their development and implementation. The growing interest in the ethical questions associated with the use of social and assistive robots is evidenced by the volume of literature reviews [ 3 , 12 , 18 , 31 , 32 , 37 , 39 - 51 ] on the topic.

Now, it appears appropriate to systematically examine this body of work, focusing on the ethical analysis, and provide an overview of the literature. Therefore, we performed a review of the literature on RCSs for older adults using the European Network of Health Technology Assessment (EUnetHTA Core Model; version 3.0) model [ 52 ] for analysis. This Health Technology Assessment (HTA) model makes it possible to assess the intended and unintended consequences of the use of a specific technology regarding multiple domains (eg, technological, ethical, clinical, and organizational), providing methods and concepts for this analysis [ 53 ]. Therefore, HTA is a process that informs decision-making about the introduction of new technologies such as RCSs in health care. It also seems necessary to issue guidelines for the development and implementation of health-oriented RCSs [ 54 ].

The objective of this study was to highlight the main ethical questions and corresponding recommendations linked to the use of RCSs for older adults for engineers, researchers, and health professionals in this field. For this purpose, we conducted a narrative literature review using the ethical dimension of the EUnetHTA Core Model to guide the analysis. To the best of our knowledge, such a study has not been conducted so far.

A thematic analysis of the literature was performed to identify publications that describe RCSs for supporting older adults in health care and prevention and those that address ethical issues and recommendations regarding their development and implementation. The methodology used for the narrative review was inspired by the study by Green et al [ 55 ].

Inclusion and Exclusion Criteria

The review encompassed papers focusing on all populations, with particular attention to older adults. It focused on the concept of RCSs for health, while also incorporating publications discussing other health technologies for older adults if the authors have delved into relevant ethical considerations for their development or implementation.

The context of the review revolved around the use of RCSs (or related technologies), especially for older adults, across diverse living environments such as homes, hospitals, and nursing homes. Publications addressing RCSs and related ethical issues within the health care domain were considered, whereas those focusing solely on technical aspects (eg, AI and deep learning) or those outside the health care domain were excluded.

Various types of publications, including theoretical papers, research studies, and reviews, were included if they offered ethical reflections or recommendations for RCS use in health care. These reflections and recommendations were expected to align with the topics and issues of the ethical dimension of the EUnetHTA Core Model.

All publications, regardless of language (English or French), were eligible if published after 2011. This time frame was chosen considering the technological advancements over the past decade, which may have influenced the evolution of ethical issues and recommendations in the field of remote care systems and related technologies. Textbox 1 summarizes the inclusion and exclusion criteria adopted for the selection of papers in this review.

Inclusion criteria

• Types of participants: all populations
• Interventions or phenomena of interest: RCSs or other technologies used in health care, if ethical issues are discussed
• Context: the use of RCSs in the health care sector
• Paper type: all paper types (theoretical papers, research studies, and reviews) that discuss ethical issues
• Language: English or French
• Date of publication: after 2011

Exclusion criteria

• Types of participants: not applicable
• Interventions or phenomena of interest: RCSs or all other types of technology outside the health care sector
• Context: the use of RCSs in non–health care sectors
• Paper type: papers about RCSs and other technologies that are not dealing with ethical issues
• Language: all other languages
• Date of publication: before 2011

Search Strategy and Study Selection

The review was conducted using the following keywords: “seniors,” “older adults,” “social robots,” “assistive robots,” “assistive technology,” “robots,” “virtual coach,” “e-coaching,” “coaching system,” “coaching device,” “ethics,” and “recommendations.”

The review was performed between August 2021 and December 2021 using the PubMed, CINAHL, Embase, Scopus, Web of Science, IEEE Explore, SpringerLink, and PsycINFO databases.

This search allowed us to find 4928 initial publications. Then, secondary research using references from other articles and the same inclusion criteria was conducted. This search allowed us to find 13 additional papers.

In total, 4943 papers were analyzed. The selection of the final publications was performed after reading the title and abstract first and, then, the full article. This selection process helped us to exclude irrelevant papers and duplicates ( Figure 1 ). In total, 0.51% (25/4943) of the papers were included in our review.

Data Analysis Criteria

The selected papers were analyzed using the ethical domain of the EUnetHTA Core Model [ 52 ]. Proper registration of the use of EUnetHTA Core Model for the purpose of this review was made on the HTA Core Model website [ 52 ].

The model was developed for the production and sharing of HTA information, allowing for the support of evidence-based decision-making in health care, but it can also be customized to other research needs. The EUnetHTA Core Model is composed of 9 domains, each including several topics. Each topic also includes different issues (ie, questions that should be considered for the evaluation of health technologies). Thus, the model is structured into 3 levels: domain (level 1), topic (level 2), and issue (level 3). The combination of a domain, topic, and issue is linked to an assessment element ID, which can be identified using a specific code for standardization purposes (B0001, B0002, etc).

The main EUnetHTA model domains include the following: (1) health and current use of the technology, (2) description and technical characteristics of the technology, (3) safety, (4) clinical effectiveness, (5) costs and economic evaluation, (6) ethical aspects, (7) organizational aspects, (8) patient and social aspects, and (9) legal aspects.

The ethical domain (level 1) in the EUnetHTA Core Model [ 52 ] includes 5 topics (level 2): “benefit-harm balance,” “autonomy,” “respect for people,” “justice and equity,” and “legislation.” Each of these topics includes several issues (level 3) [ 52 ].

In this study, 2 authors (CP and ASR) independently analyzed the 25 selected articles. First, they read the articles several times to improve familiarity with the ideas addressing the ethical aspects of RCSs. Then, in each publication (methods, results, and discussion sections), they identified segments of data that were relevant or captured an idea linked to the “ethical” domain of the model. A subsequent exploration of the coded data (sentences or set of statements) was performed to get a more precise classification at the topic level (level 2) and at the issue level (level 3). Then, the coding was performed using the HTA nomenclature. The 2 experts (CP and ASR) compared their results. In a few cases, the coding results showed a lack of consensus between the 2 coding authors, which was resolved through a subsequent discussion between them. Interrater correlation was not calculated.

A thematic analysis using the EUnetHTA framework for conducting a literature review has been described in other studies [ 56 , 57 ]. Furthermore, the use of EUnetHTA to perform an ethical analysis of health technologies has already been proposed [ 58 ]. The 25 selected articles were all coded using this methodology. Some authors have previously emphasized the possibility of overlapping issues between topics in the HTA analysis. They have suggested to assess the overlapping issues in the most relevant topic section [ 59 ].

This review was not registered, and a protocol for the review was not prepared.

Selected articles are presented in Multimedia Appendix 1 [ 3 , 12 , 18 , 31 , 32 , 37 - 51 , 60 - 64 ]. For each topic, we have presented our findings in terms of questions and recommendations according to the EUnetHTA Core Model, wherever possible.

Ethical Issues and Recommendations for the Use of New Technologies

This section aims to summarize the ethical analysis performed regarding the use of RCSs with older adults and to provide recommendations for ethical use of these devices. Table 1 presents a synthetic summary of the elements presented in this section.

Topic 1: Benefit-Harm Balance

RCSs should be developed according to the principles of beneficence (ie, to promote the interest of users) and nonmaleficence (ie, to avoid inflicting harm) [ 39 , 60 , 64 ].

What Are the Known and Estimated Benefits and Harms for Patients When Implementing or Not Implementing the Technology?

Risk of social isolation.

According to Sharkey and Sharkey [ 50 ], technological devices, when used appropriately, could benefit older adults by promoting social interaction and connection with their loved ones [ 4 , 31 , 40 ]. Broadbent et al [ 19 ] have discussed the potential of robots to reduce older adults’ social isolation. However, other authors reported the negative influence of the use of robotic devices on human contact [ 31 , 32 , 65 ]. The use of robots (eg, telepresence robots) to make some cost savings (eg, reducing travel costs and time spent on trips for family and professionals to visit older adults) would reduce face-to-face interactions [ 3 , 36 , 39 , 40 ]. Moreover, according to Körtner [ 47 ], the more people become accustomed to communicating with robots, the less they will be used to communicating with humans. The use of social robots could lead to a reduction of interactions with humans and thus to social isolation and emotional dependence [ 39 ]. However, the influence of technological devices, such as RCSs, on social isolation is still under debate, and the impact of technology would depend on the manner in which it is used.

To avoid exacerbating the users’ social isolation, Portacolone et al [ 38 ] advocate that social robots and similar technologies should be designed with the objective of fostering interactions with other humans, for instance, keeping users informed about the entertainment and socializing activities near their home, connecting them with their loved ones, and so on.

Risk of Deception

Another major risk for users is deception [ 39 , 64 , 66 ]. Portacolone et al [ 38 ] described 3 types of deception that people with neurocognitive disorders may face when interacting with social robotic systems but which may also apply to all users. The first type involves the user’s misconception of what is driving the technological device [ 51 ]. Users may be misled if they think that behind a medical chatbot, there is a real physician who communicates and reads their messages [ 44 ] or, alternatively, if they are not aware that, at some point, there are real humans guiding the technological device [ 38 ]. The second type refers to robotic devices programmed to express feelings or other types of affective communication, which may lead the user to believe that the system’s emotions are authentic. Related to this issue, Körtner [ 47 ] discussed how some older adults may fear that their social robot will forget them during their absence from home. The resemblance with the living in terms of affective behavior (eg, crying, laughing, or expressing concern) can make the user believe that there is a reciprocity between human and robot feelings [ 43 ]. The last type of deception is related to the inadequate interpretations that older adults may have regarding the nature of the robot, for example, thinking that an animal-shaped robot is a real animal or a pet [ 38 ]. Some current developments of social robots tend to make them resemble a living being, in terms of their verbal and nonverbal behaviors [ 34 , 60 ] or by highly anthropomorphizing their design [ 47 ], which may blur the boundary between the real and the artificial [ 45 , 60 ]. These design choices can also impact users’ dignity by infantilizing them as they are led to believe in something that is false [ 50 ].

However, according to some researchers [ 51 , 63 , 64 ], the notion of deception should be considered in terms of the gradation between what is morally acceptable and what is not. Deception would be morally acceptable when it aims to improve a person’s health or quality of life, for example, the use companion robots to calm a person experiencing behavioral disorders linked to dementia [ 51 ].

According to Danaher [ 43 ] and Vandemeulebroucke et al [ 40 ], to avoid deception, it is essential to be transparent to users about the design and operation of devices. As the information given to the participants is the basis for obtaining consent to use the technology, it is essential to offer them documents explaining how the device is built and its advantages and limitations in a clear manner adapted to the user’s knowledge and experience. It is also important to inform users on how to behave with technology [ 12 ]. Researchers should also answer users’ questions, pay attention to their feedback, and use it to improve the device and its documentation [ 60 ]. During experiments with RCSs, it is also important that researchers regularly remind participants of the nature of the technological device to reduce the risk of misinterpretation and to ensure that they still consent to participate in the study [ 38 ].

Biases of Algorithms

An autonomous device does not work without AI or algorithms that allow it to make decisions. However, these technologies are created by humans, and programming biases can be incorporated into them and lead to failures [ 44 ]. A technological device can, for instance, misread a situation and react accordingly, leading to a safety risk for the user [ 18 ]. Thus, it is essential that the researcher scrutinizes the algorithms used in RCSs before their implementation [ 44 ]. Fiske et al [ 44 ] also suggest providing the users with detailed explanations about the algorithms present in the technological device they are using.

What Are the Benefits and Harms of the Technology for Relatives, Other Patients, Organizations, Commercial Entities, Society, Etc?

At the society level, Boada et al [ 39 ] mentioned an ethical consideration related to the ecological impact of robotic devices in the current context of climate crisis and the lack of natural resources. The construction of RCSs requires raw materials, high energy consumption, and the management of their waste. Therefore, it is important for developers to design technologies that consume less energy and can be recycled.

Are There Any Unintended Consequences of the Technology and Its Application for Patients?

Technologies evolving very quickly.

For some older adults, technologies evolve very quickly, which makes it difficult for them to keep up with [ 62 ]. Denning et al [ 67 ] encourage designers to develop products that are intuitive to use or to offer users a simplified training. However, although some technologies are progressing quickly, technological limitations are still present, especially regarding social robotic systems, impacting their performance [ 68 ] and generating frustration among some users [ 69 ].

Unsuitability of Technology

The lack of experience with the technologies and the fact that the systems are not suitable to everyone can reduce the usability and acceptability of RCSs among older adults [ 3 , 60 , 62 ]. Frennert and Östlund [ 62 ] highlighted that some older adults were not confident in their ability to handle a robot because of previous complicated experience with technology. Peek et al [ 70 ] also reported that users had doubts about their ability to use the technology and feared that they would easily forget how to use it. They may also fear false alarms generated by monitoring technologies. For example, a person may decide to sit on the floor, but this behavior can be considered as a fall by the technology, and it could call for an ambulance to be sent to the person’s home in vain [ 70 ].

To promote acceptability and usability of RCSs, it is essential to develop them considering the capabilities, needs, and wishes of various users [ 31 , 47 ]. “User-centered design” approaches should be used for this purpose [ 71 ]. This methodology must be performed in a continuous manner to consider the development, new preferences, and experiences of the users. Technology assessment should also be conducted before deployment in ecological environments to improve the predictability of RCSs and decrease the risk of confusion and accidents [ 40 , 47 ].

Topic 2: Autonomy

According to Anderson and Kamphorst [ 42 ], the notion of autonomy implies the recognition of people, for instance, users of RCSs, as thinking individuals who have their own perspective on matters and are able to judge what is best for them.

Is the Technology Used for Individuals Who Are Especially Vulnerable?

Free and informed consent is a prerequisite for the involvement of an individual in research, regardless of the domain. This aspect is mentioned in numerous codes and declarations such as the Declaration of Helsinki (1964-2008) [ 72 ]. In the context of studies of the use of RCSs, this principle ensures that the person has freely chosen to use a device. However, some older adults, particularly those with cognitive disorders, may have difficulties in understanding and evaluating information related to RCSs and therefore in making appropriate choices [ 3 ]. Moreover, the person may not remember that the RCS is in their environment or how it works [ 38 , 44 ]. The question of how to ensure that the older adult has understood the purpose of RCS and that their choice of using the technology is based solely on their own decision and not that of a relative, caregiver, or institution has also been discussed [ 46 ].

Researchers in the field of RCS should adapt to the cognitive abilities of the populations they work with to facilitate communication and decision-making [ 46 ]. Thus, the observation of the person’s behavior is necessary to identify potential reservations regarding the use of RCSs. When the person is very vulnerable to respond, informed consent could be sought by proxy such as from children, spouse, or partner [ 46 , 64 ]. However, according to Diaz-Orueta et al [ 37 ], the final decision of using RCSs lies with the user. To prevent loss of capacity and to guard against any risk of inducement to participate, advance directives [ 46 , 64 ] or implementation of an advance power of attorney [ 46 ] can be proposed.

Does the Implementation or Use of the Technology Affect the Patient’s Capability and Possibility to Exercise Autonomy?

Dependence on the technology.

Although the main interest of RCSs for older adults is the maintenance of functional independence, it has been claimed that these devices could make people dependent on them. By replacing users in tasks that they can still perform, the use of RCSs could create new forms of vulnerability [ 3 , 31 , 39 , 41 , 51 ].

People could rely entirely on autonomous technological devices, such as RCSs, to guide their behaviors, goals, and actions [ 12 , 73 ]. A questioning of the authenticity of users’ actions has been mentioned by Anderson and Kamphorst [ 42 ]. Users might not feel responsible for the success of their actions if they feel they are completely driven by the guidance of the RCS. People could also develop emotional and psychological feelings toward the technology. This may have negative consequences for the individuals [ 38 , 49 ] and lead to new vulnerabilities [ 39 ].

Loss of Freedom

Another ethical issue relates to the conflict between the user’s safety, encouraged by the technology guidance, and a loss of freedom. The RCS could impose constraints on the user under the pretext that the user’s actions are not good for them [ 39 , 40 , 74 ]. Sharkey and Sharkey [ 50 ] explained that to promote home care, RCS could act as a supervisor (ie, programmed to ensure that no danger is present and, if there is a danger, to implement procedures to stop it and avoid it in the future). For instance, the RCS could prevent the person from eating fatty and high-caloric food because it is harmful to them. To protect users and ensure that they live in good health, individuals using RCSs could end up being deprived of certain actions or being under some type of “house arrest” [ 50 ].

One of the goals of using such RCSs is to support older adults’ independence; therefore, it is essential that developers and researchers in the field take measures to preserve the person’s autonomy [ 75 ]. Furthermore, RCS users must have the opportunity to evaluate and re-evaluate the role given to the device, to assess whether the system is reliable and whether it is serving their interests [ 12 , 42 ].

Creating a New Source of Authority

The use of RCSs could alter human relationships, for example, by creating tensions between older adults and their informal caregivers. Their use could also create some tensions with health care professionals by creating a new source of authority [ 12 ]. Monitoring older adults through RCSs can generate anger in the user, for example, when the device insists that the older adult should take a medication that they do not want to take [ 41 , 75 ].

Topic 3: Respect for Persons

Does the implementation or use of the technology affect human dignity.

Human dignity may be affected by the use of RCSs as these technologies may be perceived as “problem evocators” [ 41 ]. Some RCSs are used to compensate for impaired capacities. However, according to Körtner [ 47 ], their use can make older adults aware of their limitations and lead to negative feelings, anxiety, or exhaustion. RCS use can also lead to a form of stigmatization by making one’s own inabilities visible to others [ 3 , 70 ]. It is important to have positive communication regarding RCSs, to provide a less stigmatizing view of their use.

Does the Technology Invade the Sphere of Privacy of the Patient or User?

To continue living at home, users are increasingly willing to tolerate intrusion in their privacy [ 70 ], but they are not always aware of when and how they are being monitored by RCSs [ 61 ]. Portacolone et al [ 38 ] provided the example of an animal-shaped companion robot, for which the older adults can signal that they no longer wish to interact with it by putting the robot to sleep. However, the animal-shaped robot can record data even when it is sleeping, but users are not always aware of this information. Forgetfulness and the lack of understanding of the device can lead to the risk of manipulation and coercion [ 44 ]. The person who is vulnerable may forget that they are being monitored and reveal personal information [ 50 ].

Technological devices, such as RCSs, must remain under the control of the users [ 47 ]. Users should have the ability to define when and where the device is used—when it collects data—to maintain their privacy, especially in intimate or private care settings.

Security of Data

According to Portacolone et al [ 38 ], remote monitoring technologies are usually controlled by third parties, sometimes even operating in another country, which can lead to cultural biases during the interaction between the older adult and the RCS. This context involves the risk that the person controlling the device (third party) takes advantage of the older adult’s vulnerability to steal their personal information or exposes the user to financial abuse [ 38 ]. Older adults are not always aware or vigilant about the sharing and use of data, which may be personal and sensitive [ 73 ]. Furthermore, RCSs can be connected to internet services that collect, store, and transfer these sensitive data [ 47 ] for commercial use [ 49 , 61 ].

In addition, the use of technologies connected to digital networks involves the risk of hacking and unauthorized surveillance [ 34 , 51 ], which can make people vulnerable [ 62 ]. Denning et al [ 67 ] found that home robots could not only be remotely located and identified but also hacked and controlled. First, users may have either preconceived and erroneous ideas about the capabilities of the device or a lack of knowledge to evaluate the safety, especially regarding data protection [ 3 ]. Second, users do not always configure their technological device correctly or update them [ 67 ].

Encryption or security systems must be put in place to protect users’ personal data captured by the devices at every stage: during collection, storage, transmission, and processing [ 3 ]. Researchers must also give particular attention to data security. In Europe, for instance, researchers and technology providers are required to comply with the General Data Protection Regulation [ 40 , 76 ]. Data collection must be performed legally or approved by the local relevant ethical committees.

To address data security challenges, 3 principles are recommended by Ienca et al [ 46 ] when developing technological devices: transparency, legitimate purpose, and proportionality. Transparency refers to the fact that the user knows that the system is collecting data and has consented to it. The user must also have precise information about when and what type of data are recorded and who has access to them [ 47 ]. Legitimate purpose refers to the notion that the monitoring and collection of data is performed for a specific purpose, (ie, in the best interest of the user or, if applicable, a relative who has consented to it). Finally, the principle of proportionality refers to the fact that the data collected are not disproportionate to the user’s needs.

Topic 4: Justice and Equity

The consequences of the technology implementation on the distribution of health care resources was discussed in the literature.

How Does Implementation or Withdrawal of the Technology Affect the Distribution of Health Care Resources?

Societal pressure.

Socioeconomic issues are also linked to the development and use of RCSs can also be raised. Individual freedom may be hindered by the “incentive” of certain stakeholders or authorities to enforce the use of RCSs [ 37 ]. The use of RCSs and similar systems may also lead to a lesser involvement of relatives, caregivers, and institutions that provide care to older adults and to the reduction of care costs; these perceived economic benefits may pressurize older adults to consent to use these devices [ 40 , 46 ]. It is also possible that older adults may have to agree to use the technological device to receive other health care benefits (eg, aids and subsidies) [ 42 ].

Digital Divide

Different opportunities to access RCSs can result in digital divide, defined by the Organisation for Economic Cooperation and Development [ 77 ] as a gap between those who have access to information and communication technologies and those who do not. This difference can create educational, economic, social, and even health-related disparities among citizens. Some citizens would be able to use these devices and, therefore, could benefit from their advantages, whereas others will not be able to use them and will not enjoy their benefits. The use of technologies in the health care context, through public or private institutions, should be subject to previous authorization by independent ethical committees to ensure that the use of these devices will not harm users in any way.

Inequalities in Resources

Questions about justice, equity, and equality among all citizens also arise [ 12 , 40 , 46 ]. RCSs have relatively high costs [ 64 ] and can generate additional expenses such as an internet subscription [ 3 ] that only a part of the population can afford, and this may be owing to the lack of research allowing to measure the cost-to-benefit ratio of these technologies on health [ 32 ]. It is important to ensure the access to RCSs among different living areas (ie, urban and rural). Therefore, involving municipalities and neighborhood associations seems an interesting way of raising awareness about the opportunities offered by RCSs for older adults and reaching a wider range of people.

To promote justice, equity, and fair distribution, Ienca et al [ 46 ] and Wangmo et al [ 64 ] recommend reducing the development costs of RCSs by promoting an open dissemination of source codes. In addition, RCSs should be distributed in priority to those in greatest need; therefore, measures to ensure access to RCSs under fair conditions should be established [ 51 ]. Joachim [ 78 ] also suggests to cover some of the costs of these health care–oriented technologies through health insurance.

Recommendations have been published by researchers to improve equality of access to technologies, such as using open-source software, providing priority access for individuals with low income, or relying on certain collective financing systems such as retirement or health insurance [ 46 , 51 , 78 ]. Discussions must be conducted among developers, legislators, and private and public organizations to identify viable financing solutions that allow for fair distribution of RCSs.

Replacement of Professionals

Researchers have also reported fears expressed by older adults and caregivers about how the use of technological devices could eliminate care-related jobs or replace humans [ 17 , 34 , 48 , 61 ]. There are also concerns about the use of these technological tools to reduce health care costs by decreasing the number of available health care resources and services, thereby exacerbating social inequalities [ 44 ]. The introduction of health-oriented RCSs requires adapting the contexts of care practices, which may threaten their quality [ 39 ]. Their incorporation into the care work environment can be difficult because the devices are automated and some care situations are unpredictable [ 17 , 62 ]. Furthermore, the gestion of certain tasks by technological devices requires a restructuring of the roles and responsibilities of caregivers [ 39 ]. Fiske et al [ 44 ] highlight that there are currently no recommendations or training to enable health care professionals to adopt RCSs, even though these professionals are increasingly confronted with technological devices in their practice.

The incorporation of RCSs must always be accompanied by a discussion with concerned care professionals regarding the advantages and limits of the technology. Professionals must also be supported in the use of these devices through effective training. Structured training and supervision will contribute to the development of a controlled framework of practice around the use of RCSs and thus avoid potential abuse [ 44 ]. Moreover, to encourage their use among professionals, it is essential to clearly define the role of RCSs as an additional resource for professionals and not a replacement of human care services [ 44 ].

Topic 5: Legislation

The ethical challenges linked to the lack of existing legislations and regulations dedicated to the use of the technology were discussed in the literature.

Can the Use of the Technology Pose Ethical Challenges That Have Not Been Considered in the Existing Legislations and Regulations?

Safety of devices.

The use of RCSs by older adults can result in damage and harm to their environment [ 79 ], especially when the device is still at the prototype stage [ 47 ]. Safety risks linked to the use of RCSs (eg, malfunctioning of the technology and incorrect decisions made by the coaching system) arise when they share a common space with humans and interact with them [ 39 ]. The following questions must be considered: Who is responsible in case of an accident, and who pays for the damages [ 39 , 40 , 48 , 62 , 80 ]? Is it the designer, the device, or the user himself? Currently, the civil code favors the cascade system (ie, first, the liability falls on the designer of the product; then, on the developer; and finally, on the user who has not followed the rules of use) [ 74 ]. However, the more the machine becomes autonomous, the less the existing legal frameworks can answer these questions [ 80 ]. This is a key legal issue regarding the implementation of RCSs in real settings because the person responsible for damage to the user or the environment may incur legal or even penal proceedings.

Damage and prejudice can also be caused by a failure to share authority [ 45 , 49 , 60 ]. Who between the human and the technological device holds the power to make decisions and control a functionality [ 81 ]? According to Grinbaum et al [ 45 ], it is important to specify the circumstances in which the human must take control over the technological device (RCS) and those in which the device should decide autonomously. According to Riek and Howard [ 49 ], it is preferable that in certain cases, the technological device, although autonomous, requires a human validation of its actions to keep the user in control of the device. In addition, Bensoussan and Puigmal [ 80 ] suggested the idea that technological devices must have an emergency stop button, so that the human can switch off the technology at any time.

Regulation of Technology

Currently, there is a gray area between the capabilities of RCSs, the reality of the field, and the regulations in force [ 38 ]. To accompany the researcher during the whole process of development and diffusion of RCSs, an ethical framework should be established [ 18 , 60 ]. Specifically, this can be in the form of an ethical code of conduct illustrating the expectations to all the employees of a company [ 18 ]. The researcher must regularly inform themselves about the ethics to be consistent with the evolution of the regulatory framework [ 60 ]. However, according to Nevejans [ 82 ], these ethical recommendations have no legal value and cannot protect humans from the damage caused by new technologies. Thus, it is necessary to think about a new legal framework to protect the users of RCSs [ 37 ].

The use of technologies, such as RCSs, in the health care field has grown significantly in recent years [ 17 , 18 ]. RCSs are increasingly being used for older adults with the aim of promoting healthy behaviors, quality of life, and well-being. However, the use of RCSs also raises several ethical challenges regarding the cost-to-benefit balance of these new care practices, respect for the autonomy of users, respect for privacy, justice and equity linked to their access, or need for a suitable legal framework. Such challenges could be addressed by establishing relevant recommendations for the development and use of RCSs. Some guidelines regarding the use of robotic systems have been published [ 49 , 83 ]. Moreover, in April 2021, the European Commission unveiled the first legal framework about AI [ 84 ]. However, to the best of our knowledge, no recommendations have been proposed in this field directly linked to an analysis of the literature dealing specifically with these ethical issues and potential solutions to address them.

This narrative review identified 25 articles in which authors highlighted ethical issues and recommendations related to the use of RCSs and similar technologies. The use of the EUnetHTA Core Model for the analysis of these articles made it possible to classify the information retrieved in the publications according to 5 main ethical topics—“benefit-harm balance,” “autonomy,” “respect for persons,” “justice and equity,” and “legislation”—and to provide a detailed analysis of RCS-related ethical issues. Our review also aimed to identify recommendations for better development, diffusion, and use of RCSs by a population of older adults.

Technology devices, such as RCSs, are used with older adults to enable them to live independently; to enhance their quality of life and well-being; and, therefore, to cope with the increasing care demands for older populations. RCSs may be used to encourage a range of health-related goals: physical, cognitive, nutritional, social, and emotional domains. To be effective, RCSs must be able to motivate the user by providing highly personalized care programs [ 85 , 86 ]. However, studies have shown that not all potential target users are included in the development of these devices [ 37 , 87 , 88 ]. Therefore, RCSs design might fail to meet a wide range of users’ needs, capabilities, and wishes. Thus, it is essential to apply “user-centered design” approaches and involve target users with various sociodemographic characteristics and technology experience throughout the development process. A strong involvement of the intended users of these systems in their design process would also improve the quality of the information provided to potential users of RCSs regarding their operation, type of data collected, and potential benefits of the technology. In this way, the involvement of the users would improve the quality of the process of obtaining the consent required from older adults to use the technology.

Another ethical challenge related to the use of RCSs is the fact that their wide implementation for older adults’ care may affect the distribution of health care resources. For instance, it has been found that for some older adults and informal and formal caregivers, the use of RCSs could replace humans in many caregiving tasks, eventually leading to a suppression of jobs or to a degradation of the quality of health care services [ 17 , 34 , 48 , 61 ]. In this regard, the participation of a third person (professional, volunteer, or family member) as a “human coach” could be considered when implementing RCSs in the older adults’ environment. This “human coach” could help build a “chain of trust” by being an intermediary between the RCS and the user. On the one hand, the involvement of a real person in the use of the RCS could reduce the risk of replacement of human assistance by technological assistance. On the other hand, the “human coach” could help enhance the acceptability and usability of the device, while at the same time, reassuring the user and providing recommendations to the developers, so that the RCS is consistent with users’ needs and desires. However, the benefits of involving a “human coach” in the RCS service provision has yet to be evaluated by scientific studies.

According to some studies [ 3 , 39 , 41 , 51 , 65 ], the use of RCSs can have an impact on social relationships, reducing human contact and even altering social relationships by creating tension between older adults and their caregivers. Thus, it would be interesting to identify the repercussions and implications of these devices in older adults’ daily life and in the life of the members of their social environment through new studies. It also seems necessary to evaluate the organizational impact of the implementation of RCSs and to identify potential obstacles to their use in the care professionals’ work context.

Our analysis also confirmed that for RCSs to provide personalized health-related recommendations, the collection of sensitive data is necessary. Data collection in this context also raises several ethical issues. For instance, personal data can be exposed to hacking and misuse. Proper data management, anonymization, and encryption are essential to protect the personal data of RCS users [ 86 ]. In addition, researchers and developers in this field must evaluate RCSs before implementation to ensure that they do not cause physical or moral harm to users. Thus, it has been suggested that stakeholders refer to local and regional regulatory and safety standards to guide their development and use.

Finally, our analysis also discussed how legal and ethical frameworks regarding the use of RCSs need to be adapted to cope with the constant development of new technologies. So far, existing legal frameworks are not yet adequate to respond effectively to the question of liability in case of damage caused by RCSs, particularly because these devices are becoming increasingly autonomous [ 80 ]. The establishment of “operational ethics committees in digital sciences and technologies” could help in the development and conduct of projects in this area [ 60 ]. Guidelines should be established to identify the types of applications and technological devices that require regulatory review and approval [ 44 ]. Research projects and working groups involving users, researchers, and lawyers should be set up to further investigate the legal and ethical issues related to the use of RCSs.

Some countries and regions, such as Europe and Japan have initiated the work of structuring relevant legal and ethical frameworks; however, their orientations and measures may differ culturally [ 78 ]. Future studies in the area of RCSs could consider the influence of cultural and socioeconomic specificities of the contexts of experimentation (countries and regions) regarding the acceptance and use of RCSs by older adults and formal and informal caregivers and regarding the definition of ethical and legal frameworks governing their uses. Therefore, the use of validated and widely applied analysis frameworks, for example, the Western, Educated, Industrialized, Rich and Democratic framework [ 89 ], formulated to measure countries’ commonalities in their approaches to the interpretation of behavioral research findings (eg, regarding technology adoption) could be interesting. The Western, Educated, Industrialized, Rich and Democratic framework [ 89 ] could help not only to explore the differences among countries regarding the validation and adoption of new technologies for older adult care but also to seek greater cultural and demographic diversity in technology research.

This dimension of cross-cultural comparison has received particular attention in the framework of a current international research partnership between Europe and Japan, such as the EU-Japan Virtual Coach for Smart Ageing (e-VITA) project. This project aims to develop a cross-cultural RCS that can be tailored to the needs of healthy older adults to promote aging well. The e-VITA RCS will be made available to older adults in their homes, which raises many of the ethical questions discussed in this paper. Therefore, the study will require the researchers to set up procedures adapted not only to the users but also to the 2 cultures (European and Japanese), respecting the corresponding ethical and legal regulations. Thus, it would be interesting to perform an analysis of the ethical issues raised by users from different countries and cultures within the framework of the e-VITA project.

Limitations

A narrative review of the literature was conducted to provide a nonexhaustive synthesis of the various ethical concerns and recommendations when using RCSs for older adults. This review has some limitations. Only articles in French and English were included. Some articles indicating ethical concerns or recommendations may not have been included when this information was not mentioned in the keywords or abstract.

Conclusions

The use of RCSs in the context of health care, particularly with an older adult population, tends to show many benefits. RCSs have the potential to improve the quality of life of older adults and their independence. When used in an ethical and appropriate manner, RCSs can help improve older adults’ emotions and cognitive and physical abilities and promote social relationships. By helping older adults to continue living at home for as long as possible, the use of health-oriented RCSs could help to address some of the challenges resulting from demographic aging. However, the use of these new health care technologies involves some ethical concerns, with the most cited issues being not only the risk of accidents, lack of reliability, loss of control, risk of deception, and risk of social isolation but also the confidentiality of data and liability in case of safety problems.

Some recommendations have been made in the past regarding the use of social and assistive robotic technologies for older adults, such as considering the opinion of target users; collecting their consent; training the care professionals to use them; and ensuring proper data management, anonymization, and encryption. However, the integration of RCSs in current health practices and, particularly, in the private homes of older adults can be disruptive. It requires the establishment of scalable and adapted ethical and regulatory frameworks that follow the technology progress and the social and digital change of society Thus, studies are needed to identify new ethical concerns arising from the organizational impact of the implementation of RCSs in different contexts, especially in the homes of older adults. The influence of cultural and socioeconomic specificities of the contexts of experimentation (countries and regions) regarding the acceptance and use of RCSs by older adults and formal and informal caregivers is also an area of interest for future studies.

Acknowledgments

This paper is a part of the EU-Japan Virtual Coach for Smart Ageing (e-VITA) project, which aims to develop a robotic coaching system for older adults [ 90 ]. The authors thank the collaborators who made this project possible: European Commission and Assistance Publique–Hôpitaux de Paris (Délégation à la Recherche Clinique et à l’Innovation). This review was based on data collected within the e-Vita project, funded by the European Union H2020 Program (grant 101016453) and the Japanese Ministry of Internal Affairs and Communication (Ministry of Internal Affairs and Communication; grant JPJ000595).

Data Availability

Data sharing is not applicable to this paper as no data sets were generated or analyzed during this review.

Conflicts of Interest

None declared.

The selected relevant articles on ethical issues regarding the use of robotic coaching solutions for older adults.

• United Nations. Croissant à un rythme plus lent, la population mondiale devrait atteindre 9,7 milliards d'habitants en 2050 et pourrait atteindre près de 11 milliards vers 2100 : Rapport de l'ONU. United Nations, Population Division. United Nations; 2019. URL: https://population.un.org/wpp/Publications/Files/WPP2019_PressRelease_FR.pdf [accessed 2022-03-04]
• Rapport mondial sur le vieillissement et la santé. Organisation Mondiale de la Santé. 2016. URL: https://apps.who.int/iris/handle/10665/206556 [accessed 2022-03-04]
• Chung J, Demiris G, Thompson HJ. Ethical considerations regarding the use of smart home technologies for older adults: an integrative review. Annu Rev Nurs Res. 2016;34:155-181. [ CrossRef ] [ Medline ]
• Ganesan B, Gowda T, Al-Jumaily A, Fong KN, Meena SK, Tong RK. Ambient assisted living technologies for older adults with cognitive and physical impairments: a review. Eur Rev Med Pharmacol Sci. Dec 2019;23(23):10470-10481. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Palazzolo J, Baudu C, Quaderi A. Psychothérapies du Sujet Âgé: Prise en Charge des Pathologies du Vieillissement. 2nd edition. Amsterdam, The Netherlands. Elsevier Masson; 2016.
• Principaux repères de l'OMS sur la démence. World Health Organization. URL: https://www.who.int/fr/news-room/fact-sheets/detail/dementia [accessed 2022-03-04]
• Guilbaud A, Mailliez A, Boulanger É. Vieillissement: une approche globale, multidimensionnelle et préventive. Med Sci (Paris). Dec 09, 2020;36(12):1173-1180. [ FREE Full text ] [ CrossRef ] [ Medline ]
• op den Akker H, Klaassen R, op den Akker R, Jones VM, Hermens HJ. Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems. 2013. Presented at: CBMS '13; June 20-22, 2013:546-547; Porto, Portugal. URL: https://ieeexplore.ieee.org/document/6627870 [ CrossRef ]
• Görer B, Salah AA, Akın HL. A robotic fitness coach for the elderly. In: Augusto JC, Wichert R, Collier R, Keyson D, Salah AA, Tan AH, editors. Ambient Intelligence: Lecture Notes in Computer Science. Volume 8309. Cham, Switzerland. Springer; 2013.
• Lete N, Beristain A, García-Alonso A. Survey on virtual coaching for older adults. Health Informatics J. Dec 01, 2020;26(4):3231-3249. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Siewiorek D, Smailagic A, Dey A. Architecture and applications of virtual coaches. Proc IEEE. Aug 2012;100(8):2472-2488. [ CrossRef ]
• Kamphorst BA. E-coaching systems: what they are, and what they aren't. Pers Ubiquit Comput. May 23, 2017;21(4):625-632. [ CrossRef ]
• Kyriazakos S, Schlieter H, Gand K, Caprino M, Corbo M, Tropea P, et al. A novel virtual coaching system based on personalized clinical pathways for rehabilitation of older adults-requirements and implementation plan of the vCare project. Front Digit Health. Oct 5, 2020;2:546562. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wrobel J, Pino M, Wargnier P, Rigaud AS. Robots et agents virtuels au service des personnes âgées : une revue de l’actualité en gérontechnologie. NPG Neurol Psychiatr Geriatr. Aug 2014;14(82):184-193. [ CrossRef ]
• Pérez PJ, Garcia-Zapirain B, Mendez-Zorrilla A. Caregiver and social assistant robot for rehabilitation and coaching for the elderly. Technol Health Care. 2015;23(3):351-357. [ CrossRef ] [ Medline ]
• Fasola J, Mataric MJ. A socially assistive robot exercise coach for the elderly. J Hum-Robot Interact. 2013;2(2):3-32. [ CrossRef ]
• Cresswell K, Cunningham-Burley S, Sheikh A. Health care robotics: qualitative exploration of key challenges and future directions. J Med Internet Res. Jul 04, 2018;20(7):e10410. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Winfield AF, Jirotka M. Ethical governance is essential to building trust in robotics and artificial intelligence systems. Philos Trans A Math Phys Eng Sci. Oct 15, 2018;376(2133):20180085. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Broadbent E, Stafford R, MacDonald B. Acceptance of healthcare robots for the older population: review and future directions. Int J of Soc Robotics. Oct 3, 2009;1(4):319-330. [ CrossRef ]
• Abou Allaban A, Wang M, Padır T. A systematic review of robotics research in support of in-home care for older adults. Information. Jan 30, 2020;11(2):75. [ CrossRef ]
• Cortellessa G, De Benedictis R, Fracasso F, Orlandini A, Umbrico A, Cesta A. AI and robotics to help older adults: revisiting projects in search of lessons learned. Paladyn J Behav Robot. 2021;12(1):356-378. [ CrossRef ]
• Lima MR, Wairagkar M, Gupta M, Rodriguez y Baena F, Barnaghi P, Sharp DJ, et al. Conversational affective social robots for ageing and dementia support. IEEE Trans Cogn Dev Syst. Dec 2022;14(4):1378-1397. [ CrossRef ]
• Sidner CL, Bickmore T, Nooraie B, Rich C, Ring L, Shayganfar M, et al. Creating new technologies for companionable agents to support isolated older adults. ACM Trans Interact Intell Syst. Jul 24, 2018;8(3):1-27. [ CrossRef ]
• Pepito JA, Locsin RC, Constantino RE. Caring for older persons in a technologically advanced nursing future. Health. 2019;11(05):439-463. [ CrossRef ]
• Pinheiro PR, Pinheiro PG, Filho RH, Barrozo JP, Rodrigues JJ, Pinheiro LI, et al. Integration of the mobile robot and internet of things to monitor older people. IEEE Access. 2020;8:138922-138933. [ CrossRef ]
• Chatterjee A, Gerdes MW, Martinez S. eHealth initiatives for the promotion of healthy lifestyle and allied implementation difficulties. In: Proceedings of the 2019 International Conference on Wireless and Mobile Computing, Networking and Communications. 2019. Presented at: WiMOB '19; October 21-23, 2019:1-8; Barcelona, Spain. URL: https://ieeexplore.ieee.org/document/8923324 [ CrossRef ]
• Yousuf H, Reintjens R, Slipszenko E, Blok S, Somsen GA, Tulevski II, et al. Effectiveness of web-based personalised e‑Coaching lifestyle interventions. Neth Heart J. Jan 28, 2019;27(1):24-29. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bevilacqua R, Casaccia S, Cortellessa G, Astell A, Lattanzio F, Corsonello A, et al. Coaching through technology: a systematic review into efficacy and effectiveness for the ageing population. Int J Environ Res Public Health. Aug 15, 2020;17(16):5930. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Justo R, Ben Letaifa L, Palmero C, Gonzalez-Fraile E, Torp Johansen A, Vázquez A, et al. Analysis of the interaction between elderly people and a simulated virtual coach. J Ambient Intell Human Comput. May 22, 2020;11(12):6125-6140. [ CrossRef ]
• Fiorini L, De Mul M, Fabbricotti I, Limosani R, Vitanza A, D'Onofrio G, et al. Assistive robots to improve the independent living of older persons: results from a needs study. Disabil Rehabil Assist Technol. Jan 22, 2021;16(1):92-102. [ CrossRef ] [ Medline ]
• Sriram V, Jenkinson C, Peters M. Informal carers' experience of assistive technology use in dementia care at home: a systematic review. BMC Geriatr. Jun 14, 2019;19(1):160. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pilotto A, Boi R, Petermans J. Technology in geriatrics. Age Ageing. Nov 01, 2018;47(6):771-774. [ CrossRef ] [ Medline ]
• Koceska N, Koceski S, Beomonte Zobel P, Trajkovik V, Garcia N. A telemedicine robot system for assisted and independent living. Sensors (Basel). Feb 18, 2019;19(4):834. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Broadbent E. Interactions with robots: the truths we reveal about ourselves. Annu Rev Psychol. Jan 03, 2017;68(1):627-652. [ CrossRef ] [ Medline ]
• Suwa S, Tsujimura M, Ide H, Kodate N, Ishimaru M, Shimamura A, et al. Home-care professionals’ ethical perceptions of the development and use of home-care robots for older adults in Japan. Int J Hum Comput Interact. Mar 13, 2020;36(14):1295-1303. [ CrossRef ]
• Nordgren A. How to respond to resistiveness towards assistive technologies among persons with dementia. Med Health Care Philos. Sep 6, 2018;21(3):411-421. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Diaz-Orueta U, Hopper L, Konstantinidis E. Shaping technologies for older adults with and without dementia: reflections on ethics and preferences. Health Informatics J. Dec 2020;26(4):3215-3230. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Portacolone E, Halpern J, Luxenberg J, Harrison KL, Covinsky KE. Ethical issues raised by the introduction of artificial companions to older adults with cognitive impairment: a call for interdisciplinary collaborations. J Alzheimers Dis. 2020;76(2):445-455. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Boada JP, Maestre BR, Genís CT. The ethical issues of social assistive robotics: a critical literature review. Technol Soc. Nov 2021;67:101726. [ CrossRef ]
• Vandemeulebroucke T, Dierckx de Casterlé B, Gastmans C. The use of care robots in aged care: a systematic review of argument-based ethics literature. Arch Gerontol Geriatr. Jan 2018;74:15-25. [ CrossRef ] [ Medline ]
• Zafrani O, Nimrod G. Towards a holistic approach to studying human-robot interaction in later life. Gerontologist. Jan 09, 2019;59(1):e26-e36. [ CrossRef ] [ Medline ]
• Anderson J, Kamphorst B. Ethics of e-coaching: implications of employing pervasive computing to promote healthy and sustainable lifestyles. In: Proceedings of the 2014 IEEE International Conference on Pervasive Computing and Communication Workshops. 2014. Presented at: PerComW '14; March 24-28, 2014:351-356; Budapest, Hungary. [ CrossRef ]
• Danaher J. Robot betrayal: a guide to the ethics of robotic deception. Ethics Inf Technol. Jan 04, 2020;22(2):117-128. [ CrossRef ]
• Fiske A, Henningsen P, Buyx A. Your robot therapist will see you now: ethical implications of embodied artificial intelligence in psychiatry, psychology, and psychotherapy. J Med Internet Res. May 09, 2019;21(5):e13216. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grinbaum A, Chatila R, Devillers L, Ganascia JG, Tessier C, Dauchet M. Ethics in robotics research: CERNA mission and context. IEEE Robot Automat Mag. Sep 2017;24(3):139-145. [ CrossRef ]
• Ienca M, Jotterand F, Vică C, Elger B. Social and assistive robotics in dementia care: ethical recommendations for research and practice. Int J of Soc Robotics. Jun 22, 2016;8(4):565-573. [ CrossRef ]
• Körtner T. Ethical challenges in the use of social service robots for elderly people. Z Gerontol Geriatr. Jun 25, 2016;49(4):303-307. [ CrossRef ] [ Medline ]
• Operto F. Ethics in advanced robotics. IEEE Robot Automat Mag. Mar 2011;18(1):72-78. [ CrossRef ]
• Riek LD, Howard DA. A Code of Ethics for the Human-Robot Interaction Profession. In: Proceedings of 2014 Conference on We Robot. 2014. Presented at: WeRobot '14; April 4-5, 2014:1; Coral Gables, FL. URL: https:/​/robots.​law.miami.edu/​2014/​wp-content/​uploads/​2014/​03/​a-code-of-ethics-for-the-human-robot-interaction-profession-riek-howard.​pdf
• Sharkey A, Sharkey N. Granny and the robots: ethical issues in robot care for the elderly. Ethics Inf Technol. Jul 3, 2010;14(1):27-40. [ CrossRef ]
• Yew GC. Trust in and ethical design of carebots: the case for ethics of care. Int J Soc Robot. May 23, 2021;13(4):629-645. [ FREE Full text ] [ CrossRef ] [ Medline ]
• HTA Core Model ® version 3. EUnetHTA. URL: http://www.corehta.info/model/HTACoreModel3.0.pdf [accessed 2022-03-04]
• Health technology assessment process: fundamentals. EUPATI Toolbox. 2015. URL: https://toolbox.eupati.eu/resources/health-technology-assessment-process-fundamentals/ [accessed 2022-03-04]
• Tuli TB, Terefe TO, Rashid MM. Telepresence mobile robots design and control for social interaction. Int J Soc Robot. Jul 13, 2021;13(5):877-886. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med. Sep 2006;5(3):101-117. [ CrossRef ]
• Isabet B, Pino M, Lewis M, Benveniste S, Rigaud AS. Social telepresence robots: a narrative review of experiments involving older adults before and during the COVID-19 pandemic. Int J Environ Res Public Health. Mar 30, 2021;18(7):3597. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Naudé B, Rigaud AS, Pino M. Video calls for older adults: a narrative review of experiments involving older adults in elderly care institutions. Front Public Health. Jan 14, 2021;9:751150. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Saarni S, Hofmann B, Lampe K, Lühmann D, Mäkelä M, Velasco-Garrido M, et al. Ethical analysis to improve decision-making on health technologies. Bull World Health Organ. Aug 01, 2008;86(8):617-623. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lampe K, Mäkelä M, Garrido MV, Anttila H, Autti-Rämö I, Hicks NJ, et al. The HTA core model: a novel method for producing and reporting health technology assessments. Int J Technol Assess Health Care. Dec 23, 2009;25(S2):9-20. [ CrossRef ]
• Collectif C. Éthique de la recherche en robotique. ALLISTENE. 2014. URL: http://cerna-ethics-allistene.org/digitalAssets/38/38704_Avis_robotique_livret.pdf [accessed 2024-04-02]
• Tisseron S. Introduction. In: Tisseron S, Tordo F, editors. Robots, de Nouveaux Partenaires de Soins Psychiques. Fairfield CA. Érès Publication; 2018.
• Frennert S, Östlund B. How do older people think and feel about robots in health- and elderly care? In: Proceedings of the 2018 INBOTS Conference on Inclusive Robotics for a Better Society. 2018. Presented at: INBOTS '18; October 16-18, 2018:167-174; Pisa, Italy. URL: https://link.springer.com/chapter/10.1007/978-3-030-24074-5_28 [ CrossRef ]
• van Maris A, Zook N, Caleb-Solly P, Studley M, Winfield A, Dogramadzi S. Designing ethical social robots-a longitudinal field study with older adults. Front Robot AI. Jan 24, 2020;7:1. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wangmo T, Lipps M, Kressig RW, Ienca M. Ethical concerns with the use of intelligent assistive technology: findings from a qualitative study with professional stakeholders. BMC Med Ethics. Dec 19, 2019;20(1):98. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Portet F, Vacher M, Rossato S. Les technologies de la parole et du TALN pour l’assistance à domicile des personnes âgées : un rapide tour d’horizon (quick tour of NLP and speech technologies for ambient assisted living). In: Proceedings of the 2012 Workshop on Interactions Langagières pour personnes Agées Dans les habitats Intelligents. 2012. Presented at: ILADI '12; June 4-8, 2012; Grenoble, France. URL: https://aclanthology.org/W12-1402.pdf
• Bradwell HL, Winnington R, Thill S, Jones RB. Ethical perceptions towards real-world use of companion robots with older people and people with dementia: survey opinions among younger adults. BMC Geriatr. Jul 14, 2020;20(1):244. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Denning T, Matuszek C, Koscher K, Smith JR, Kohno T. A spotlight on security and privacy risks with future household robots: attacks and lessons. In: Proceedings of the 11th international conference on Ubiquitous computing. 2009. Presented at: UbiComp '09; September 30-October 3, 2009:105-114; Orlando, FL. URL: https://dl.acm.org/doi/10.1145/1620545.1620564 [ CrossRef ]
• Papadopoulos I, Koulouglioti C, Lazzarino R, Ali S. Enablers and barriers to the implementation of socially assistive humanoid robots in health and social care: a systematic review. BMJ Open. Jan 09, 2020;10(1):e033096. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pripfl J, Kortner T, Batko-Klein D. Results of a real world trial with a mobile social service robot for older adults. In: Proceedings of the 11th ACM/IEEE International Conference on Human-Robot Interaction. 2016. Presented at: HRI '16; March 7-10, 2016:497-498; Christchurch, New Zealand. URL: https://ieeexplore.ieee.org/document/7451824 [ CrossRef ]
• Peek ST, Wouters EJ, van Hoof J, Luijkx KG, Boeije HR, Vrijhoef HJ. Factors influencing acceptance of technology for aging in place: a systematic review. Int J Med Inform. Apr 2014;83(4):235-248. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lespinet-Najib V, Roche A, Chibaudel Q. Santé et handicap: d'une conception centrée «utilisateur» à la conception universelle. Ann Mines Réal Ind. 2017;2:25-27. [ CrossRef ]
• Déclaration d’Helsinki de l’AMM – Principes éthiques applicables à la recherche médicale impliquant des êtres humains. L'Association Médicale Mondiale. URL: https:/​/www.​wma.net/​fr/​policies-post/​declaration-dhelsinki-de-lamm-principes-ethiques-applicables-a-la-recherche-medicale-impliquant-des-etres-humains/​ [accessed 2022-02-03]
• Adams S, Niezen M. Digital ‘solutions’ to unhealthy lifestyle ‘problems’: the construction of social and personal risks in the development of eCoaches. Health Risk Soc. Feb 06, 2016;17(7-8):530-546. [ CrossRef ]
• Saerens P. Le droit des robots, un droit de l’homme en devenir ? Commun Technol Dév. Jun 30, 2020;(8). [ CrossRef ]
• Jenkins S, Draper H. Care, monitoring, and companionship: views on care robots from older people and their carers. Int J of Soc Robotics. Sep 15, 2015;7(5):673-683. [ CrossRef ]
• Règlement (UE) 2016/679 du parlement européen et du Conseil du 27 avril 2016 relatif à la protection des personnes physiques à l'égard du traitement des données à caractère personnel et à la libre circulation de ces données, et abrogeant la directive 95/46/CE (règlement général sur la protection des données) (Texte présentant de l'intérêt pour l'EEE). Le Parlement Européen Et Le Conseil De L'union Européenne. 2016. URL: http://data.europa.eu/eli/reg/2016/679/oj/fra [accessed 2022-02-03]
• Understanding the digital divide. OECD Digital Economy Papers. 2001. URL: https://econpapers.repec.org/paper/oecstiaab/49-en.htm [accessed 2024-04-02]
• Joachim C. Silver economy, robotique et droit - Comparaison franco-japonaise. Centre d’études et de Coopération Juridique Interdisciplinaire – Université de Poitier. Jan 2020. URL: https://hal.science/hal-03147937/document [accessed 2024-04-05]
• Kernaghan K. The rights and wrongs of robotics: ethics and robots in public organizations. Can Public Adm. Dec 12, 2014;57(4):485-506. [ CrossRef ]
• Bensoussan A, Puigmal L. Le droit des robots ? Quelle est l'autonomie de décision d'une machine ? Quelle protection mérite-t-elle ? Arch Philos Droit. 2017;59(1):165-174. [ CrossRef ]
• Tessier C. Robots autonomy: some technical issues. In: Lawless WF, Mittu R, Sofge D, Russell S, editors. Autonomy and Artificial Intelligence: A Threat or Savior? Cham, Switzerland. Springer; 2017:179-194.
• Nevejans N. Comment protéger l'homme face aux robots ? Arch Phil Droit. 2017;59(1):131-163. [ CrossRef ]
• Tambornino L, Lanzerath D, Rodrigues R, Wright D. SIENNA D4.3: survey of REC approaches and codes for artificial intelligence and robotics. Zenodo. Aug 19, 2019. URL: https://zenodo.org/records/4067990 [accessed 2024-04-04]
• Ethics guidelines for trustworthy AI. Shaping Europe's Digital Future. URL: https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai [accessed 2022-07-15]
• Kamali ME, Angelini L, Caon M, Carrino F, Rocke C, Guye S, et al. Virtual coaches for older adults’ wellbeing: a systematic review. IEEE Access. 2020;8:101884-101902. [ CrossRef ]
• Banos O, Nugent C. E-coaching for health. Computer. Mar 2018;51(3):12-15. [ CrossRef ]
• Gustafson DH, McTavish F, Gustafson DH, Mahoney JE, Johnson RA, Lee JD, et al. The effect of an information and communication technology (ICT) on older adults’ quality of life: study protocol for a randomized control trial. Trials. Apr 25, 2015;16(1):3-8. [ CrossRef ]
• Sakaguchi-Tang DK, Cunningham JL, Roldan W, Yip J, Kientz JA. Co-design with older adults: examining and reflecting on collaboration with aging communities. Proc ACM Hum Comput Interact. Oct 18, 2021;5(CSCW2):1-28. [ CrossRef ]
• Henrich J, Heine SJ, Norenzayan A. The weirdest people in the world? Behav Brain Sci. Jun 15, 2010;33(2-3):61-83. [ CrossRef ]
• EU-Japan virtual coach for smart ageing. e-VITA Consortium. URL: https://www.e-vita.coach/ [accessed 2024-04-05]

Abbreviations

Edited by A Mavragani; submitted 12.04.23; peer-reviewed by J Sedlakova, S Liu; comments to author 20.08.23; revised version received 22.12.23; accepted 12.03.24; published 18.06.24.

©Cécilia Palmier, Anne-Sophie Rigaud, Toshimi Ogawa, Rainer Wieching, Sébastien Dacunha, Federico Barbarossa, Vera Stara, Roberta Bevilacqua, Maribel Pino. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 18.06.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 15 April 2024
• Correction 22 April 2024

Revealed: the ten research papers that policy documents cite most

• Dalmeet Singh Chawla 0

Dalmeet Singh Chawla is a freelance science journalist based in London.

You can also search for this author in PubMed   Google Scholar

When David Autor co-wrote a paper on how computerization affects job skill demands more than 20 years ago, a journal took 18 months to consider it — only to reject it after review. He went on to submit it to The Quarterly Journal of Economics , which eventually published the work 1 in November 2003.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-024-00660-1

Updates & Corrections

Correction 22 April 2024 : The original version of this story credited Sage, rather than Overton, as the source of the policy papers’ citation data. Sage’s location has also been updated.

Autor, D. H., Levy, F. & Murnane, R. J. Q. J. Econ. 118 , 1279–1333 (2003).

Article   Google Scholar  

Costanza, R. et al. Nature 387 , 253–260 (1997).

Willett, W. et al. Lancet 393 , 447–492 (2019).

Article   PubMed   Google Scholar  

Steffen, W. et al. Science 347 , 1259855 (2015).

Rockström, J. et al. Nature 461 , 472–475 (2009).

Download references

Reprints and permissions

Related Articles

I was prevented from attending my own conference: visa processes need urgent reform

Correspondence 18 JUN 24

The global refugee crisis is above all a human tragedy — but it affects wildlife, too

We can make the UK a science superpower — with a radical political manifesto

World View 18 JUN 24

Extending the Sustainable Development Goals to 2050 — a road map

Comment 17 JUN 24

Securing your science: the researcher’s guide to financial management

Career Feature 14 JUN 24

Hybrid working from home improves retention without damaging performance

Article 12 JUN 24

Department of Health and Human Services (DHHS), is seeking exceptional candidates for the position of Director

Bethesda, Maryland (US)

National Institutes of Health

Principal Investigator Positions at the Institute for Regenerative Biology and Medicine, CIMR

Regenerative Biology and Medicine, including but not limited to disease immunology, ageing, biochemistry of extracellular matrix...

Beijing, China

The Chinese Institutes for Medical Research (CIMR), Beijing

Principal Investigator Positions at the Institute for Molecular and Cellular Therapy, CIMR, Beijing

We're looking for outstanding scientists at all ranks interested in developing novel therapeutics in all disease areas.

Post-Doctoral Fellowship in Regenerative Biology and Medicine (Lab of. Dr. Yuval Rinkevich)

Discovery of cellular and molecular mechanisms of tissue repair and regeneration.

Institute for Regenerative Biology and Medicine, Chinese Institutes for Medical Research (CIMR)

Career Opportunities at the Yazhouwan National Laboratory, Hainan, China

YNL recruits leading scientists in agriculture: crop/animal genetics, biotech, photosynthesis, disease resistance, data analysis, and more.

Sanya, Hainan, China

Yazhouwan National Laboratory

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Reference Examples

More than 100 reference examples and their corresponding in-text citations are presented in the seventh edition Publication Manual . Examples of the most common works that writers cite are provided on this page; additional examples are available in the Publication Manual .

To find the reference example you need, first select a category (e.g., periodicals) and then choose the appropriate type of work (e.g., journal article ) and follow the relevant example.

When selecting a category, use the webpages and websites category only when a work does not fit better within another category. For example, a report from a government website would use the reports category, whereas a page on a government website that is not a report or other work would use the webpages and websites category.

Also note that print and electronic references are largely the same. For example, to cite both print books and ebooks, use the books and reference works category and then choose the appropriate type of work (i.e., book ) and follow the relevant example (e.g., whole authored book ).

Examples on these pages illustrate the details of reference formats. We make every attempt to show examples that are in keeping with APA Style’s guiding principles of inclusivity and bias-free language. These examples are presented out of context only to demonstrate formatting issues (e.g., which elements to italicize, where punctuation is needed, placement of parentheses). References, including these examples, are not inherently endorsements for the ideas or content of the works themselves. An author may cite a work to support a statement or an idea, to critique that work, or for many other reasons. For more examples, see our sample papers .

Reference examples are covered in the seventh edition APA Style manuals in the Publication Manual Chapter 10 and the Concise Guide Chapter 10

Related handouts

• Common Reference Examples Guide (PDF, 147KB)
• Reference Quick Guide (PDF, 225KB)

Textual Works

Textual works are covered in Sections 10.1–10.8 of the Publication Manual . The most common categories and examples are presented here. For the reviews of other works category, see Section 10.7.

• Journal Article References
• Magazine Article References
• Newspaper Article References
• Blog Post and Blog Comment References
• UpToDate Article References
• Book/Ebook References
• Diagnostic Manual References
• Children’s Book or Other Illustrated Book References
• Classroom Course Pack Material References
• Religious Work References
• Chapter in an Edited Book/Ebook References
• Dictionary Entry References
• Wikipedia Entry References
• Report by a Government Agency References
• Report with Individual Authors References
• Brochure References
• Ethics Code References
• Fact Sheet References
• ISO Standard References
• Press Release References
• White Paper References
• Conference Presentation References
• Conference Proceeding References
• Published Dissertation or Thesis References
• Unpublished Dissertation or Thesis References
• ERIC Database References
• Preprint Article References

Data and Assessments

Data sets are covered in Section 10.9 of the Publication Manual . For the software and tests categories, see Sections 10.10 and 10.11.

• Data Set References
• Toolbox References

Audiovisual Media

Audiovisual media are covered in Sections 10.12–10.14 of the Publication Manual . The most common examples are presented together here. In the manual, these examples and more are separated into categories for audiovisual, audio, and visual media.

• Artwork References
• Clip Art or Stock Image References
• Film and Television References
• Musical Score References
• Online Course or MOOC References
• Podcast References
• PowerPoint Slide or Lecture Note References
• Radio Broadcast References
• TED Talk References
• Transcript of an Audiovisual Work References
• YouTube Video References

Online Media

Online media are covered in Sections 10.15 and 10.16 of the Publication Manual . Please note that blog posts are part of the periodicals category.

• Facebook References
• Instagram References
• LinkedIn References
• Online Forum (e.g., Reddit) References
• TikTok References
• X References
• Webpage on a Website References
• Clinical Practice References
• Open Educational Resource References
• Whole Website References