Dissemination of academic information currently relies on a system of independently operated journals to which researchers submit the results of their labours. Various disciplines and medical subspecialties have their own journals, with many subspecialties having multiple journals, leaving researchers to decide on how to prioritise their submissions. Selection of the journal for submission can be based on a variety of factors, including the perceived interest of readership, previous publication experience, the presumed potential for citation, the perceived ease of submission, and the impact factor.Reference Wong, Srikrishnan, Hadka and Keller1
Paediatric cardiology is a field in which, for several years, there were no specific journals. Because of this, much of the initial research pertaining to paediatric cardiology was published in journals focusing on general cardiology, which showed a bias towards adult cardiology. Although subspeciality journals in paediatric cardiology are now well established, there are still many involved in the field who believe that publication of their findings in the journals focusing on adult cardiology is more beneficial in terms of the potential for subsequent citation. Some of these beliefs emerge, in part, from the notion that these journals focusing on adult cardiology have a higher impact factor. This, however, is to be expected, as there are many more adult cardiologists, and published manuscripts devoted to adult cardiology, when compared to paediatric cardiologists and works appearing in journals devoted to paediatric cardiology.Reference Loomba and Anderson2 Whether or not journal type impacts subsequent citations, and the performance potential of the manuscripts related to paediatric cardiology has not been evaluated. With this in mind, we compared the performance of manuscripts related to paediatric cardiology and CHD published during the same year in journals devoted primarily to paediatric as opposed to adult cardiology.
Methods
Since the study used publicly available data pertaining to publication performance, and did not use any data relating to individual patients, institutional review board review or approval was not necessary.
Objective
Our primary objective was to compare the citation performance of manuscripts relating to paediatric cardiology and CHD published in the year 2014 in journals devoted to paediatric cardiology as opposed to adult cardiology.
Methods to calculate the impact of an article, journal, or author
Several metrics are used to estimate the impact of an article, journal, or author, including the number of citations, the impact factor, and the author h-index (https://libguides.usc.edu/evaluate/impact). The number of citations can be used at the level of the article, journal, or author.
The impact factor is used only at the level of the journal and is defined as the number of citations to all items published in that journal in the past 2 years divided by the number of articles and reviews published over those past 2 years referencing those citations. For example, if a journal has an impact factor of 3.5, then for that given time interval, each article published was cited on average 3.5 times in the previous 2 years in that journal.
The h-index was originally used at the level of the author, but is now also used at the level of the journal. H is the number of articles published by an author which have each been cited at least h times. For example, if an author published four papers that are cited 12, 7, 4, and 2 times, respectively, then the h-index of that author is 3. The h-index has also more recently been applied to journals. For journals, h is the number of articles published that have been cited h times over a given time period (e.g., H5 = 5 years).
Selection of journals and year
For the journals focusing on paediatric cardiology journals, we chose Cardiology in the Young, Paediatric Cardiology, and Annals of Paediatric Cardiology. These journals were all publishing actively throughout 2014, and were indexed in PubMed and Google Scholar.
The journals selected to represent adult cardiology were Journal of the American College of Cardiology and Circulation. We chose these journals on the basis that, anecdotally, they are most popular journals focusing primarily on adult cardiology amongst both adult and paediatric cardiologists. These journals are also indexed in both PubMed and Google Scholar.
Time period
We chose the year 2014 because it offered a complete 5-year period for which we could gather evidence regarding the performance in terms of citations of published manuscripts. Even those manuscripts published at the end of 2014 would still have completed a period of 5-years for potential citation. We limited our selection to 2014 on the basis that this would minimise any confounding from the lifespan of the manuscripts, and would have offered ample time for retractions should these have been necessary. We collected the number of citations in August, 2020.
Study identification
As all our chosen journals were indexed in PubMed, we identified the manuscripts by searching PubMed using journal title and year. Manuscripts were deemed appropriate for inclusion if they pertained to those with congenital malformations of the heart, acquired paediatric heart disease, paediatric electrophysiology, or care of adults with CHD. Manuscripts published in 2014 from each journal were then compiled into a list. The date of publication and journal were then confirmed. The manuscripts were identified by two authors separately (RL and SF), with the lists then reviewed by the third author (EV), who highlighted any discrepancies. Identified discrepancies were then reviewed by these three authors (RL, SF, EV), thus providing a group consensus.
Data collection
Data were collected by two authors (DS, AD) separately using an electronic data collection tool. Manuscript title, name of the first author, journal, issue, volume, and citation number were collected for each manuscript. The number of citations was manually collected using data from Google Scholar. The data collected relative to citations was then reviewed by the third author (RL) to identify discrepancies. Discrepancies were reviewed by these three authors (DS, AD, RL), again permitting group consensus.
Statistical analyses
All statistical analyses were conducted using SPSS Version 23.0. A p-value of less than 0.05 was considered statistically significant.
Statistical analyses – number of citations
The performance of all manuscripts relating to CHD published in 2014 was first analysed in terms of subsequent citation, irrespective of the journal in which they were published. We determined the median (50th percentile) and mean number of citations for all published manuscripts, as well as the number of citations corresponding with the 25th, 75th, 90th, and 95th percentiles.
We then analysed the performance of the manuscripts in terms of citations after dividing the manuscripts into two groups: manuscripts published in journals focusing specifically on paediatric cardiology and manuscripts published in journals focusing on adult cardiology. The median (50th percentile) and mean number of citations as well as the number of citations corresponding with the 25th, 75th, 90th, and 95th percentiles were calculated for the manuscripts falling into these two groups. A Mann–Whitney-U test was then conducted to compare the number of citations between the groups.
Statistical analyses – potential for citation
We identified 45 eligible manuscripts published in journals devoted to adult cardiology. To compare the performance in terms of the potential for citation of these manuscripts to a similar number of manuscripts published in specialised paediatric cardiology journals, we identified the 45 most cited eligible manuscripts published in the latter journals. We then determined the median (50th percentile) and mean number of citations as well as the number of citations corresponding with the 25th, 75th, 90th, and 95th percentiles for these manuscripts. We again used a Mann–Whitney-U test to compare the number of citations between the groups.
Statistical analyses – h-index
We also calculated the h-index for all publications. The h-index is a metric which is equal to the number h, such that h is the number of manuscripts that have received at least h number of citations.Reference Hirsch3 Thus, an h-index of 5 indicates that five manuscripts have been cited at least five times.
Qualitative analyses – top 50 most cited manuscripts
Next, we simply arranged the manuscripts from most cited to least cited and identified the 50 most highly cited manuscripts to find the proportion of these manuscripts that were published in adult cardiology journals and paediatric cardiology journals.
Informal social media survey
We also carried out an informal, social-media-based poll, approaching researchers in paediatric cardiology with the following prompt: “Do you believe congenital heart disease manuscripts have lower citation potential in paediatric (Cardiology in the Young, Paediatric Cardiology, etc.) versus adult cardiology (JACC, Circulation, etc.) journals?”. This was done on Twitter using a poll with the options of “yes” or “no” as responses. No other response choice was present. No free-text option was present. The survey was targeted at paediatric cardiologists and paediatric intensivists.
Results
All manuscripts
We identified a total of 828 manuscripts for analysis. Of these, 783 (95%) were published in a journal focusing specifically on paediatric cardiology, while 45 (5%) were published in journals devoted to adult cardiology. The total number of citations for these 828 manuscripts was 11,936. Of these, 8419 (71%) were for manuscripts published in journals concerned specifically with paediatric cardiology, while 3517 (29%) were for manuscripts published in journals focusing on adult cardiology. The median number of citations for all these manuscripts was 8. This resulted in a median annual citation rate of 1.6. The number of citations corresponding to various percentiles is outlined in Table 1. Of the 828 manuscripts, 53 (6%) remained uncited within the study period of approximately 5 years. The h-index over the study period of 5 years for all the manuscripts was 46.
Citation number corresponding with various percentiles
Comparison between manuscripts published in paediatric and adult cardiology journals
The median number of citations for manuscripts published in journals specific to paediatric cardiology during the study period was 7, representing 1.4 per year, compared to 41, at 8.2 per year, for manuscripts published in adult cardiology journals (p < 0.01). The number of citations corresponding to the various percentiles is outlined in Table 1.
The h-index over the 5-year study period was 36 for manuscripts published in journals devoted to paediatric cardiology, compared to 27 for those published in the adult cardiology journals.
Citation performance potential: comparison of top-performing manuscripts in paediatric cardiology journals to manuscripts published in adult cardiology journals
A total of 90 manuscripts was included in these analyses. This included all 45 manuscripts published in adult cardiology journals, along with the 45 manuscripts published in paediatric cardiology journals with the greatest number of citations. The median number of citations for the manuscripts published in specific journals during the study period was 46, representing 9.2 citations per year. This was compared with 41, at 8.2 citations per year, for manuscripts published in adult cardiology journals (p = 0.64).
Fifty most cited manuscripts
Of the 50 most cited papers, 28 of these were published in paediatric cardiology journals while the remaining 22 were published adult cardiology journals.
Informal survey
A total of 153 invitees responded to the social media survey. Of these, 81 (53%) responded “yes” to the prompt.
Discussion
Our findings reveal that the median number of citations for all manuscripts in a period of approximately 5 years, irrespective of the journals in which they were published, was 7, equating to 1.4 citations per year. The median number of citations proved to be greater for manuscripts published in journals devoted to adult cardiology, although the calculated performance in terms of the potential for citation was not different between the journals. The h-index, furthermore, was greater for those manuscripts published in the specialised paediatric cardiology journals.
The findings highlight several key points. Perhaps most importantly, the performance in terms of the potential for citation is no different whether the manuscripts are published in specialised paediatric cardiology as opposed to general adult cardiology journals. This means that, when an equal number of the most cited manuscripts published are compared, the number of citations does not differ. A second key finding was that the h-index for manuscripts published in the specialised paediatric cardiology journals was, in fact, greater than for those published in the general cardiology journals. This emphasises that the potential for citation, when assessed in terms of performance, is not diminished when a publication takes place in journals specific for paediatric cardiology.
This finding is in contrast to the beliefs of those researching in the field of paediatric cardiology. When we conducted our informal, social-media-based poll, asking researchers if they anticipated their works would have a lower potential for citation if published in the specialised paediatric cardiology journals, just over half replied in the affirmative. The results of our study now serve to reassure researchers that this is not the case.
We need to reconcile, however, our finding in terms of similar performance potential with the other finding that the median number of citations when work was published in the specialised paediatric cardiology journals was significantly lower than for manuscripts published in the general cardiology literature. This reconciliation is mediated by the proportion of manuscripts published by the specialised as opposed to the generalised journals. Those journals focusing on adult cardiology published only 5% of the total manuscripts. It is well known that the journals focusing on adult cardiology are more selective with the manuscripts chosen for publication, and that achieving publication in these adult cardiac journals is itself significantly harder. The works devoted to paediatric cardiology and CHD also represent a small proportion of the overall number of manuscripts published in the general journals. That the h-index is greater for manuscripts published in the specialised paediatric cardiology journals further highlights this phenomenon.
Compared to the mean, or even median number, of citations, the h-index is less sensitive to outliers and provides a better sense of consistency. As the h-index is defined as the value h whereby there are h number of publications with x citations, the h-index cannot increase with single outliers and can only increase with an increasing number of publications with certain performance. Thus, the h-index is a marker of consistency and has, in fact, been shown to correlate better with future performance of the individual author or journal than mean or median citation number.Reference Hirsch4
Take, for example, a journal which has published six papers with the following number of citations: 10, 10, 2, 1, 1, 1. The mean number of citations for this author would be 4.2, the median number of citations 1.5, and the h-index 1. Clearly, the mean value is greatly skewed by the higher performing publications, the median less so. However, the h-index reflects the consistency of the publications of this journal, half of which have only had a single citation. Assuming no new publications, as the publications of this journal accrue citations of each of these publications, it will take more citations to increase the h-index than the other metrics. If the citations for each of these publications were to double: 20, 20, 4, 2, 2, 2, the mean would increase to 8.3, the median to 3, and the h-index to 2. Thus, the h-index seems to indicate consistency and, as such has been demonstrated to be a more discriminating predictor of future publication performance.Reference Hirsch4 Additionally, the h-index “credits” journals for prioritising publication of a large number of studies in specific disciplines in the sense that disciplines for which publications tend to get a larger number of citations may be favoured by journals to help improve journal performance metrics. It is important to note that the h-index of an individual or journal is limited by the number of publications. A greater number of publications is required for a higher h-index. By publishing more paediatric cardiology manuscripts, the paediatric cardiology journals are better poised to achieve a higher h-index.
The shortcoming of mean or median citation number is highlighted by not only outliers on the high end but also the low end. Journals devoted to paediatric cardiology not only publish a larger number of manuscripts related to the field; they also publish a larger number of case reports. Case reports tend to be cited less frequently than either original studies or systematic reviews. The journals concerned with adult cardiology, which do not publish such case reports, are likely to have an improved number of citations because of this factor. Case reports and series can be particularly helpful in assisting with clinical management, even though they may never accrue a large number of citations. Case reports are also the first entry into the world of academic publication for many young clinicians and scientists. Journals, therefore, should consider the publication of these items as part of their academic mission, despite their potentially negative effect with regard to the impact factor itself.
Researchers, journals, reviewers, editors, and readers are all equal stakeholders in the scientific publishing endeavour. They possess complementary roles when it comes to disseminating data. Once researchers generate, analyse, and document their findings, it becomes imperative for them to publish their findings so that it can then be shared with others. The selection of the journal for publication becomes an important consideration. The most appropriate journal should possess the appropriate target audience, and possess the means by which to get the published material into the hands of the largest number of those in the target audience.
For those publishing in the fields of paediatric cardiology and CHD, there has been anecdotal thought that manuscripts published in adult cardiology journals tend to perform better than those published in the journals concentrating specifically on paediatric cardiology. In this light, performance can be considered in terms of impact on clinical practice, number of reads, and number of citations. While a discussion of assessment of the impact of research on clinical practice is beyond our scope, we emphasise that the impact of a given manuscript on clinical practice cannot be quantified with any current objective metric.Reference Osterloh and Frey5,Reference Seglen6 While some have attempted quantification by using the number of reads or citations, these metrics have demonstrated to be largely independent of impact on clinical practice. Hence, they should be acknowledged as separate metrics of performance, rather than surrogate markers.Reference van Eck, Waltman, van Raan, Klautz and Peul7–Reference Hanney, Home, Frame, Grant, Green and Buxton11 The number of reads is often not quantified. Indeed, the focus in modern medical publishing has tended to be placed on the number of citations. Previous analyses have demonstrated that factors influencing citation include the topic of the manuscript, the number of authors, their recognition, evidence of international collaborative efforts, and other factors. The number of citations, furthermore, has shown to lack correlation even with adherence to standardised data reporting recommendations or the appropriateness of statistical analyses. Despite these limitations on the value of “the number of citations as a metric pf performance”, a perceived correlation exists between the number of citations of a given work and its clinical impact.Reference Nieminen, Carpenter, Rucker and Schumacher12
With these concepts and challenges in mind, we recommend that those publishing manuscripts related to paediatric cardiology and CHD should divorce themselves from the notion that their publication will have less potential for subsequent citation if published in a journal concentrating on adult cardiology rather than one specific for paediatric cardiology. High-quality, novel manuscripts published in specialised paediatric cardiology journals can obtain the same number of citations as those published in the general cardiology literature. As the specialised paediatric cardiology journals are more intimately linked to the field, and more are focused on the targeted audience, support, and preservation of these journals by those working in the field is of utmost importance. As papers published in the specialised paediatric cardiology journals accrue more citations, these journals will improve in terms of both reputation and publishing metrics. This improvement will then lead to a greater number of citations for the works they publish, creating a perpetual and self-contained cycle within the speciality. With over nine-tenths of the manuscripts related to the field being published in the journals specific to the field, reliance on work published in adult cardiology journals to be a reliable and efficient academic hub for the speciality is ill-founded. Those submitting manuscripts related to paediatric cardiology should seriously consider submitting to paediatric cardiology journals as there isn’t an inherent limitation to citation potential when compared to adult cardiology journals.
A logical argument to the above is that publishing in an adult cardiology journal essentially “guarantees” a higher median number of citations. From our data, it is impossible to tell if the same manuscript is published in an adult cardiology and paediatric cardiology journal if there is a true advantage as the citation potential does not differ between the two. Unfortunately, as such, a rebuttal to this argument is beyond the scope of the data here.
Our analyses, we believe, has uncovered new data, characterising the performance of manuscripts relating to the field in terms of their potential for citation, and quantifying the influence of the type of journal in which the works are published. Despite these strengths, we recognise that our analyses also have limitations. We limited the analyses only to the type of journal, not analysing their other characteristics. We also confined ourselves to a set of manuscripts published at least 5 years ago. We accept that the absolute and relative performance of manuscripts published more recently may have changed. The current analyses are also unable to account for some element of “chance” that is most certainly a factor in citations. For instance, in the current era of COVID-19 and the related increase in Kawasaki disease with macrophage activation syndrome, Kawasaki disease papers have now seen a recent increase in their citations. If one type of journal published more such papers, they would have an increase in their metrics. Additional analyses of the characteristics of publication for those working in the field of paediatric cardiology are both needed and warranted.
Conclusion
We are able to conclude that around one-twentieth of the work relating to paediatric cardiology and CHD is published in journals that focus predominantly on adult cardiology. While such publications in journals that focus on adult cardiology tend to generate a greater number of citations than those achieved for works published in specialised paediatric cardiology journals, the potential for citation is no different between the journals. Of more importance, the h-index is higher for the manuscripts published in the specialised paediatric cardiology journals. This finding means that those contemplating publication of work in our field of paediatric and congenital cardiac care should continue to focus on submission to journals devoted to paediatric cardiology and CHD.
Acknowledgements
None.
Financial support
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Conflicts of interest
None.
