Ultrasound is widely recognized as method of choice for visualizing organs in children. M-mode and cross-sectional echocardiography is a noninvasive, cost-effective and efficient method. Many epidemiological studies use echocardiographic measurements to examine the associations of various determinants with left ventricular structures in children.Reference Schieken1 This information may be relevant for identifying critical periods in life for left ventricular growth and development. To compare echocardiographic measurements as an outcome of different studies, it is necessary to minimize intraobserver and interobserver variability.Reference Wallerson and Devereux2 Although the clinical relevance and research importance of echocardiographic measurements in children is well known, little has been published about the reliability of these measurements. One group has assessed the reliability of echocardiographic measurements in a group children of mothers infected with the human immunodeficiency virus, demonstrating that measurements of left ventricular mural thickness and fractional shortening made in a local facility differed from those made in a central echocardiography laboratory.Reference Lipshultz, Easley and Orav3 In this study, however, intraobserver and interobserver reliability were not assessed directly. Results from Project Heartbeat! showed that echocardiographic measurements in healthy children aged 8 years and older can be performed accurately with acceptable reproducibility.Reference Dai, Ayres, Harrist, Bricker and Labarthe4 Less is known about the reliability of echocardiographic measurements in younger children. With this deficiency in mind, our aim was to quantify intraobserver and interobserver reliability of echocardiographic measurements in young children.
Materials and methods
The children were selected from the outpatient clinic of the Erasmus Medical Center – Sophia Children’s Hospital, a university hospital in Rotterdam, the Netherlands. In total, 28 subjects, resulting in 56 paired measurements, participated in this study to validate echocardiographic measurements. None of them had congenitally malformed hearts, or other cardiovascular disease influencing the circulation. Written informed consent was obtained for enrollment. All children in this study were participants in two ongoing follow-up studies, specifically the Intra-Uterine Growth Retardation follow-up study and the Generation R Study,Reference Jaddoe, Bakker and van Duijn5, Reference Jaddoe, van Duijn and van der Heijden6 and had given informed consent for echocardiographic measurements.
All examinations were performed by 2 experienced sonographers (SPE Snijders, VE Kleyburg) at the same visit. Every single cardiac ultrasound measurement was measured twice by the two sonographers. The first observer scanned the ultrasound measurements in no specific order. Subsequently, the other observer did the same, after which the first examiner repeated the process. The sonographers left the ultrasound room during the assessment by the other observer. The time interval for the first observer to rescan the patient depended on the time the second observer took for the measurements, being about 10 minutes. Measurement results were blinded to both operators; the caliper read out on the screen was hidden. Printouts of the measurements were not seen by the observers and were saved on hard disc for later analyses.
The examination was carried out in a quiet room, with the child quietly awake in a supine position. To measure left cardiac structures, M-mode cross-sectional echocardiography was performed in all the children. Left ventricular measurements, including left ventricular end diastolic diameter, ventricular end-diastolic septal thickness, and left ventricular end-diastolic posterior mural thickness, were measured according to the recommendations of the American Society of Echocardiography.Reference Schiller, Shah and Crawford7 Other assessments of left cardiac structures were the diameters of the aortic root and the left atrium. All examinations were performed using an ATL-Philips Model HDI 5000 (Seattle, Washington, USA) equipped with a 5.0 megahertz, high frequency, curved array transducer.
We used the methods to compare observers in detail as described by Bland and Altman.Reference Bland and Altman8, Reference Bland and Altman9 The first step was to plot the data and draw the line of equality. This visualizes the degree of agreement.Reference Bland and Altman8 Second, the consensus between and among observers was analyzed using the intraclass correlation coefficient for all cardiac measurements. The intraclass correlation coefficient is defined as the ratio of the variance between subjects to total variance. The intraclass correlation coefficient measures the strength of the agreement of the variables, independent of the dimension of the variable considered. Additionally, the 95% confidence interval was calculated for all cardiac measurements.
Agreement was then tested to investigate intra- and interobserver differences and reproducibility.Reference Bland and Altman8 For assessing intraobserver agreement, main interest was in the mean difference between repeated measures of one observer. For assessing the interobserver agreement, the difference between the first measurements of each separate observer was used in the analyses. We created Bland and Altman plots by plotting the differences of all the measurements against their mean with the 2 standard deviations of the mean to see the distribution and to find any possible differences from the mean within or between the observers.Reference Bland and Altman8 The average differences between duplicate measures were tested using the paired-sample t-test to see if there was systemic bias. If the differences are Gaussian distributed, 95% of the differences will lie between the mean plus or minus 2 standard deviation limits. These are the limits of agreement, and the measures between and among observers can be assumed to be interchangeable within these limits. An advantage of the Bland and Altman plots over the statistical testing is that one can visualize between which boundaries a measurement is interchangeable and how large the measurement differences between operators are in proportions. How small the limits of agreement should be is a clinical, not a statistical, decision that should be made in advance of the analysis. We decided that we could confidently say that cardiac measurements in children would be reliable and valid in case of intraclass correlation coefficient over 0.80 and mean difference within 10% from the mean of two different measurements. Statistical analysis was performed using Statistical Package of Social Sciences version 11.0 for Windows (SPSS Inc, Chicago, IL, USA).
Results
The median age of the children was 7.5 years, with an interquartile range from 3 to 11 years. Of the cohort, 17 (61%) were male. Half of the children was younger than 8 years of age. In Table 1, we present the descriptive statistics of the echocardiographic measurements of the left cardiac structures assessed in this study. A comparison of the means and standard deviation of the different echocardiographic measurements among and between observers is shown. None of the differences between the means on repeated observation and between observers were statistically significant.
Descriptive statistics for echocardiographic measurements in children.
Analyses were based on a group of 28 children.
In Figure 1, we show plots of the measurements between observers against the line of equality. All points seem to lie randomly around this line, meaning that there seems to be no bias. The variables are close to the line of equality as well, indicating good agreement and suggesting small differences between observers.
Echocardiographic measurements of observers with the line of equality (a). Aortic root diameter measurements (b). Left atrial diameter measurements (c). Ventricular septal thickness measurements (d). Left ventricular diameter measurements (e). Left ventricular posterior mural thickness measurements.
In Table 2, we present the intraclass correlation coefficients with their 95% confidence interval. Intraobserver intraclass correlation coefficient ranged from 0.91, for the ventricular septal thickness, to 0.99 for the diameter of the aortic root. Interobserver intraclass correlation coefficient ranged from 0.78 for the left ventricular posterior mural thickness to 0.96 for the left ventricular diameter.
Intra- and interobserver intraclass correlation coefficient and 95% confidence interval for left cardiac structures in childhood.
In Figure 2, we show Bland and Altman plots for the differences in left cardiac measurements among or between observers against the mean. The limits of agreemen, plus or minus 2 standard deviations, are plotted in the figures. Most differences lie between the limits of agreement.
Intra- and interobserver Bland and Altman plots of variation in echocardiographic measurements among and between observers (a). Intraobserver agreement aortic root diameter (b). Intraobserver agreement left atrial diameter (c). Intraobserver agreement ventricular septal thickness (d). Intraobserver agreement left ventricular diameter (e). Intraobserver agreement left ventricular posterior mural thickness (f). Interobserver agreement aortic root diameter (g). Interobserver agreement left atrial diameter (h). Interobserver agreement ventricular septal thickness (i). Interobserver agreement left ventricular diameter (j). Interobserver agreement left ventricular posterior mural thickness.
In Table 3, we show the mean differences for the measurements of the left cardiac structures among and between observers, and the 95% limits of agreement, shown as the mean difference plus or minus 2 standard deviations of the difference. The observed difference from the mean for left cardiac structures ranged from zero millimetres for the left ventricular end-diastolic posterior mural thickness, to 1.60 millimeter (6.3%) for the left atrial diameter. The mean difference between duplicate measures was not significantly different from zero. This suggest that no systematic difference exists between the pairs of results. The limits of agreement ranged from −0.96 to 0.96 millimeter for the left ventricular end-diastolic posterior mural thickness to the widest limit from −5.55 to 7.55 millimeters for the left atrial diameter.
Intra- and interobserver measurement variation in absolute number and in proportions of the mean with 95% limits of agreement.
t-tests were performed to compare the difference from the mean, none of the measurements was significantly different from zero.
Discussion
Echocardiography is widely used to visualize organs in children. Left cardiac structures can be measured in children using M-mode cross-sectional echocardiography. The aim of our study was to assess the reliability of measurements made of left cardiac structures in young children with structurally normal hearts. Both reproducibility, whether two observers use the same measurement to obtain the same result, and repeatability, whether a single observer obtains the same results when he/she takes the repeated measurement, are types of agreement or reliability. Any process of measurement should be both accurate and reproducible. The mean difference between duplicate measurements is an indicator of the amount of systematic difference, or relative bias, between the pairs of results. The standard deviation of the differences between duplicated measurements represents the reproducibility of the process. The larger the mean difference, the larger the systematic bias, and the lower the accuracy of the process. Similarly, the larger the standard deviation, the larger random errors, and the lower the reproducibility.
By using various statistical methods to assess the reproducibility and repeatability of our measurements, the simple plot of the results of one observer against the other showed that the data points are clustered near the line of equality, indicating good agreement and little differences between observers.
Beforehand, we decided that we could consider cardiac measurements in children as good reliability and valid in case of intraclass correlation coefficient over 0.80. Our results showed high intraclass correlation coefficients, over 0.80, for almost all echocardiographic measurements, with low 95% confidence intervals, indicating a high degree of similarity among and between observers, and thus good agreement. Intraobserver intraclass correlation coefficients were higher, ranging from 0.99 to 0.91, than interobserver intraclass correlation coefficients, which ranged from 0.96 to 0.78 for all left cardiac structures. This phenomenon, so called interobserver variability, could be expected, since two observers measure differently.
The Bland and Altman plots showed good agreement among and between observers. We found for almost all echocardiographic measurements differences within 10% from the mean, so the differences are unlikely to be of clinical significance. Only for the measurement of left ventricular posterior mural thickness was the difference from the mean more than 10%. The differences were similar regardless of the mean size of the measurement, meaning that differences did not vary in any systemic way across the range of the measurement.
Others have measured intraobserver and interobserver variations of echocardiographic measurements in children, for example the reliability of echocardiographic measurements as assessed in a group children of mothers infected with the human immunodeficiency virus.Reference Lipshultz, Easley and Orav3 This study showed substantial variability between measurements made in a central echocardiography laboratory as opposed to a local facility, although left ventricular dimension was reliably measured. Intraclass correlation coefficients ranged from 0.64 for fractional shortening to 0.97 for left ventricular dimension.
The Project Heartbeat!, in contrast, showed that echocardiographic measurements in healthy children aged 8, 11, and 14 years can be performed accurately with acceptable reproducibility.Reference Dai, Ayres, Harrist, Bricker and Labarthe4 They presented mean differences in left ventricular mass for intraobserver variability of −1.82, with standard deviation of 18.79 grams, and for interobserver variability of 4.50, with standard deviation of 24.16 grams. Although the children in the Project Heartbeat were older than in our study, the mean differences and standard deviations for the left cardiac structures reported were similar to ours.
The major strength of our study is the young age at which we studied the intraobserver and interobserver variability of echocardiographic measurements. To our knowledge, previous studies of reliability of echocradiographic measurements have only been performed in older childrenReference Lipshultz, Easley and Orav3, Reference Dai, Ayres, Harrist, Bricker and Labarthe4 and adultsReference Sahn, DeMaria, Kisslo and Weyman10, Reference Valdez, Motta and London11, Reference Ladipo, Dunn, Pringle, Bastian and Lawrie12. Second, two well trained sonographers performed all the echocardiographic measurements in our study, so our results are probably to a large extend dependend on the experience of these two sonographers. Currently, these sonographers work together, but they were trained in different echocardiographic laboratories according to current recommendations.Reference Schiller, Shah and Crawford7 We expect that the reproducibility would have been lower if the ultrasound measurements were performed by sonographers working in different research centers. This difference should be small, since all measurements were performed using strict protocols, which are used by many centers to maximize uniformity of performance. Furthermore, cardiac measurements could be measured in all of the participating children. None of the children started crying or wanted to quit before the measurements were finished. The sonographers put major efforts to get high quality measurements. All echocardiograms we performed, therefore, including septal and free wall left ventricular endocardium measurements, had sufficient quality for tracing. Another strength of our study is that we used various statistical methods that give a useful indication about the reliability of echocardiographic measurements in children.
A limitation could be that we only used one probe to measure the left cardiac structures at all different ages. Furthermore, the number of children in our study is relatively low. Other studies assessing reliability of cardiac and other ultrasound measurements, however, were performed in groups containing similar numbers, showing that a total of 30 children is sufficient to evaluate reliability,Reference Valdez, Motta and London11–Reference Schlesinger, Hernandez, Zerin, Marks and Kelsch14 especially in studies that are designed to assess whether ultrasound measurements are reproducible and repeatable as an outcome measure for other studies.
In conclusion, therefore, we have demonstrated good repeatability and reproducibility of most left cardiac structures in children measured by ultrasound. This information is important for clinical and epidemiological research projects focused on left cardiac structures in young children.
Acknowledgements
We gratefully acknowledge the contribution of parents and children.
