Phantom accuracy study

A phantom accuracy study was conducted, similar to the work of Pisani et al.Reference Pisani, Lockman, Jaffray, Yan, Martinez and Wong³ An anthropomorphic head phantom (Alderson RANDO, Salem, NY) was computed tomography (CT) scanned and five sets of digitally reconstructed radiographs (DRRs) were generated, for the five different isocentre positions as shown in Table 1.

Table 1.

Isocentre positions for phantom digitally reconstructed radiograph (DRR) matching sets

All measurements in centimetres.

For each type of imaging, the phantom was positioned on a treatment couch with the isocentre matching the first set of images. Images were acquired and reviewed to verify the correct positioning of the phantom.

Without moving the phantom, four further sets of images were taken and associated with the DRRs from the four other isocentre position to produce sets of reference and verification images with known offsets.

Six radiation therapists who were blinded to these offsets volunteered to participate in this section of the study. All therapists were experienced in performing online corrections with both imaging systems. Three assessments were made by each reviewer for each dummy image set – one in each axial dimension [superior–inferior (S-I), anterior–posterior and left–right], resulting in a total of 12 assessments being made by each of the six reviewers. This process was followed using the two imaging technologies to be compared–namely MV EPI and kV OBI, yielding a total of 144 assessments being made (72 on each imaging technology). For each of the assessments, the residual error was recorded. The same reviewers and images were used for each of the imaging technologies, making this a matched pairs experiment. The deliberate offsets of the second set of images (MV EPI) were changed slightly (all moved 3 mm in the superior direction) from the first set (kV OBI) to prevent the reviewers from relying on memory when assessing the MV EPI images. In addition, a gap of 1 month was placed between the testing on kV OBI images and the testing on the MV EPI images to further reduce the risk of memory playing a role in reviewer accuracy.

Retrospective analysis of imaging session duration

The imaging records of 20 H&N IMRT patients were analysed. All patients were treated with IMRT to a total dose of 70 Gy in 35 fractions for either nasopharyngeal (18 patients) or oropharyngeal (2 patients) cancer.

Pretreatment position verification imaging was performed at every treatment fraction.

Tolerances for corrective action were patient specific, based on dosimetric implications of a deviation in a particular direction. No tolerance in any direction was greater than 3 mm. All patients were immobilised with custom-made neck supports and masks that encompassed the head and shoulders.

Ten patients underwent MV EPI position verification imaging between January 2002 and February 2004.

Ten patients underwent kV OBI position verification imaging between April 2006 and May 2007.

MV position verification

Orthogonal images of open 6 MV ‘set-up’ fields were captured with Varian aS-500 panels, 40 cm × 30 cm (512 pixels × 384 pixels) providing images of the patient with a resolution of 0.5 mm. Set-up field apertures were restricted to the minimum that would display sufficient anatomy for the clinical matching process.

Reference images were either digitised simulator films or DRRs generated with the treatment planning computer. Reference images were enhanced by the addition of lines and curves to highlight specific bony anatomy. Image review was performed with Vision (Version 7.3.1., Varian Medical Systems, Palo Alto, CA) software, allowing image overlay.

The two orthogonal treatment images were individually reviewed by the treatment staff, that is, one image reviewed at a time. As such, each image could indicate a different value for a set-up deviation in the S-I axis. Where discrepancies between these values occurred, the value from the lateral image was used as it was considered more reliable for detecting S-I set-up deviations in the H&N setting.

Corrections were performed manually by treatment staff re-entering the bunker and adjusting the treatment couch position. After a correction, additional images were acquired and reviewed prior to treatment to ensure the adjustment was performed correctly.

kV position verification

Orthogonal images of open-field kV X-ray beams were captured with a flat panel detector, 40 cm × 30 cm (2048 pixels × 1536 pixels) capable of capturing images of the patient with a resolution of 0.13 mm, and optimised for lower energy X-rays. Image review was performed by treatment staff using OBI (Version 1.2 and 1.3, Varian Medical Systems) software. During the review, the two orthogonal images were linked such that a correction in the S-I direction on one image was automatically translated to the other image.

Corrections were performed remotely, using software to control motorised patient support couch movements. After a correction, additional images were acquired and reviewed prior to treatment to ensure the system performed the adjustment correctly.

For both types of imaging, offline review of images was performed at least weekly by the treating radiation oncologist.

Imaging session duration

A timing study was conducted, similar to the work of Fox et al.Reference Fox, Elder, Crocker, Davis, Landry and Johnstone⁵ For each studied patient, the following data were extracted from the IMPAC^TM Record and Verify database for each fraction

• • the time of the first verification image (not including tube warm up in the case of kV imaging) [T1]

• • the time that the first treatment field delivery is completed [T2]

T2–T1 = approximated duration of verification imaging session

In the case of kV imaging, T1 is reported by the kV imaging computer and T2 is reported by the treatment monitoring computer. The synchronicity of the individual computer clocks was checked regularly throughout the data recording period.

In addition, the proportion of fractions where a correction occurred and the number of directions (i.e., posterior, left, etc.) with a tolerance for correction of less than 3 mm was recorded for each patient.

Phantom accuracy study

For each of the 72 combinations of image, axis of assessment and reviewer, the difference between the absolute value of the residual error under OBI following assessment and the absolute value of the residual error under EPI following assessment was computed. Given the expectation of improved accuracy on the OBI, the resulting 72 values were termed ‘improvement’ values–the amount by which the OBI technique improves upon the EPI technique in terms of residual error after assessment. Note that if, for a given pair of assessments, the EPI residual error was smaller than that using OBI, the improvement value was negative for that pair.

Tests for intra-class correlation (ICC) were conducted, considering both the possibility of correlation within reviewers and the correlation within images. In both cases, the ICC was very low, indicating that the assessment measurements can be regarded as independent. We can hence treat our data as 72 independent matched pairs.

Test for difference between imaging techniques

The resolution of the residual error measurements (1 mm) was roughly in the same order as the size of the residual errors themselves. In other words, most of the residual error measurements were ≤ 2 mm, and all were recorded only to the nearest millimetre. This is highly granular data which cannot reasonably be treated as continuous. Hence, the non-parametric Wilcoxon test was used on the paired residual errors to test for differences between the imaging techniques. The resulting p-value was smaller than 0.001, indicating that from a statistical perspective, the reduction in residual error as a result of moving to the OBI technology was highly significant.

A simple comparison of the number of errors by magnitude for each imaging technology is given in Figure 1. Note that none of the errors in the OBI set exceed 1 mm, whereas some 3 mm errors were made when EPI was used.

Figure 1.

Comparison of residual errors in phantom position for kilovoltage on-board imaging (kV OBI) and megavoltage electronic portal imaging (MV EPI) systems. Note that no residual error on kV OBI is larger than 1 mm.

Point estimate and confidence limits for imaging technique difference

The data were highly granular with correction measurements all in the order of 0–3 mm, and rounded to the nearest millimetre. Many conventional methods measuring confidence intervals break down when applied to such data. Therefore, confidence limits were generated using Monte–Carlo simulation. The estimated improvement in OBI residual error over EPI residual error was 0.57 mm, with the approximate 95% confidence interval on this improvement being 0.33 mm, 0.81 mm.

Imaging session duration

Descriptive statistics

Table 2 shows the mean and standard deviation of the execution times for each imaging system. All times are in minutes.

Table 2.

The mean and standard deviation of the execution times for head and neck intensity modulated radiation therapy (H&N IMRT) verification imaging on each system

All times are in minutes.

Statistical techniques

The observational unit was taken to be the patient. A unifactor linear regression model was then built with the average treatment time across the 35 fractions taken to be the response variable, and the imaging method taken to be the (dichotomous) explanatory variable. The p-value of 0.664 for inclusion of imaging method in the model shows that there is no evidence of a difference between the treatment times for the two imaging techniques.

For 20.8% of the fractions on EPI and 31.8% of fractions on the OBI, a set-up error larger than the allowed tolerance was detected, resulting in a positioning correction followed by a second set of images. This process results in longer imaging session durations for those fractions. To control for this in the modelling exercise, a multifactor model with two explanatory variables was built. The first variable is the imaging method, as for the unifactor case, and the second, newly introduced explanatory variable, is the proportion of fractions for which a second set of images was necessary. The modelling exercise demonstrated that even though the need to take a second set of images increased execution time by a statistically significant amount (p = 0.002), there was no evidence that the effect of the imaging technique is significant (p = 0.33), even when controlling for the number of image sets.

Figure 2 shows the proportion of fractions where a correction occurred for each patient against the number of directions where the patient’s individual tolerance for correction was less than 3 mm. Note that patients on the OBI had a greater prevalence of smaller tolerances for correction. Although it would be expected that smaller tolerances would result in more frequent correction, in this small sample no such trend was observed.

Figure 2.

Scatter plot of the proportion of fractions where a correction occurred for each patient against the number of directions where the patient’s individual tolerance for correction was less than 3 mm.

Phantom accuracy study

Similar to the study of Pisani et al., treatment staff were able to detect translational errors on both systems, mostly within 1 mm, but with larger variation in MV imaging. Pisani et al. conducted their experiments on an Elekta^TM treatment unit with different imaging specifications.Reference Pisani, Lockman, Jaffray, Yan, Martinez and Wong³ They also physically shifted their phantom to create the mismatched datasets. The mismatch is this study was created digitally (in the planning system) requiring no phantom movement and thereby eliminating a potential source of error from the experiment.

Because of resource limitations in our centre, H&N IMRT has been restricted to use in patients where other radiation therapy options would result in unacceptably high doses to critical normal structures. The majority of H&N IMRT patients are treated for nasopharyngeal disease, where the tumour volume and critical normal structures such as the brainstem and spinal cord are in close proximity. The detected improvement in accuracy in this study appears small. However, the presence of sharp dose gradients in close proximity to critical structures, as typically observed in H&N IMRT, amplifies the clinical importance of such an improvement.Reference Manning, Wu, Cardinale, Mohan, Lauve, Kavanagh, Morris and Schmidt-Ullrich²

Although it is tempting to use this improvement to justify a reduction in the CTV to PTV margin, it is only one of multiple uncertainties in H&N IMRT delivery. These uncertainties include target volume delineation, rotation and deformation, all of which can be significant and are likely to play a large part in clinical decisions regarding margins.Reference Dosoretz, Court and Chen^6–Reference Hong, Tomé, Chappell, Chinnaiyan, Mehta and Harari¹⁰

Imaging session duration

This study has found no evidence to support the hypothesis that the patient position verification process on the kV OBI system is faster than on the traditional MV EPI system in the setting of H&N IMRT at our centre.

The study by Fox et al., on which our method was based, studied kV imaging timing data for a range of anatomical sites shortly after the clinical acceptance of an OBI unit. They did not compare kV OBI with MV EPI, but observed general reductions in verification imaging duration over the first few months of OBI operation and predicted that as staff became more familiar with the OBI system, times would decrease further.Reference Fox, Elder, Crocker, Davis, Landry and Johnstone⁵ The comparison of MV and kV imaging by Pisani et al. focussed on comparative accuracy and did not compare imaging session duration.Reference Pisani, Lockman, Jaffray, Yan, Martinez and Wong³ In this study, the imaging times for individual patients appear to reduce over their course of treatment, but in the 13 months of data collection on the OBI system no trend in the average imaging times was observed. It is possible that a trend is occurring, but other factors specific to the patient, for example weight loss, make it unobservable in the available sample size.

The timing method incorporates non-imaging aspects such as the delivery of the first treatment beam and is therefore not a true representation of imaging session duration. However, this applies to both imaging systems and for the purpose of this comparison is not likely to be significant. This method provides the closest approximation of imaging time possible retrospectively.

Repeat imaging following an automated couch position correction on the kV OBI system was carried over from the MV EPI process, which required manual correction of the treatment couch position. The original intention was that this step would be abolished after establishing the reliability of the automated corrections, thereby reducing the duration of imaging sessions where a correction was required. A number of the longer kV imaging sessions included three sets of imaging. This was either due to patient movement during the analysis of the first set of images, or the detection of a positioning error that required manual adjustment, for example improving the alignment of the neck with the head as described in Court et al.Reference Court, Wolfsberger, Allen, James and Tishler¹¹ These stabilisation and reproducibility issues need to be addressed before abolishing the second image set, and further imaging during or after treatment delivery is being considered to determine the amount of intra-fraction motion present.

It was noted that a greater proportion of OBI sessions included a correction (31.8% vs. 20.8% on EPI). There are a number of possible explanations for this difference, most notably the fact that separate patient groups were imaged on each system, rather than each patient being imaged on both systems. As seen in Figure 2, patients on the OBI typically had tighter tolerances for correction, but no correlation between smaller tolerances and the proportion of corrections performed was seen in the studied sample. Other patient-specific factors such as weight loss over the course of treatment or difficulties with compliance may have contributed to this difference in correction frequency.

Other factors may be specific to the imaging system, for example staff confidence in their ability to detect a variation requiring correction. This study is unable to determine how much of the difference in correction frequency can be attributed to patient factors, the imaging system used or other factors.

As stated earlier, different patient groups were imaged on the two systems. For the purposes of comparison, it would be preferable to have both types of imaging performed on all patients. This would, however, entail additional imaging dose and longer treatment sessions without a benefit to the patients. We did not consider this ethically justifiable.

Accuracy and workflow should not be considered independently, and often one is traded off for the other.Reference Herman¹² A potential explanation of the findings of this study is in the actual matching processes used. As briefly described in the Methods and Materials, MV EPI matching was predominantly side by side image matching using electronic measuring tools to compare anatomy. OBI matching provides a larger range of options for image manipulation and a number of matching tools, including image overlay. Casual observation during the accuracy study found that although staff would match a number of points on an EPI image, with OBI images gross anatomy was matched using image overlay, prior to fine anatomy matching using a moveable window inset of the DRR over the kV image. As such the OBI matching involved a larger number of tasks, and effectively matched more anatomical ‘points’. This impacts on the time required to perform the matching, but should theoretically improve accuracy and the ability to detect deformations in patient set-up such as neck flexion and shoulder misalignment.