Discussion

Data regarding the impact of peer review on SBRT planning are limited to two smaller cohorts from time periods in which SBRT utilisation was less frequent, and another more recent cohort of 285 cases. ^{Reference Lo, Liu and Chan10–Reference Fitzgerald, Pryor and Aland12} Suggested plan changes from peer review are variably reported in the literature, but commonly are categorised as major changes which require repeat planning and minor changes which do not. ^{Reference Rouette, Gutierrez and O’Donnell13–Reference Brammer, Pettit and Allerton16} A previous review found that suggested plan changes occur in 10% of all cases, with higher rates of change observed for stereotactic radiotherapy and several specific cancer subsites. ^{Reference Huo, Gorayski, Poulsen, Thompson and Pinkham6} A systematic review from 2017 found similar outcomes, with suggested plan changes in 10.8% of all cases reviewed. ^{Reference Brunskill, Nguyen and Boldt7} These both appear to be lower than the rates of change for SBRT cases in the published literature to date. A Canadian report of 40 lung SBRT cases found major contouring changes recommended in 80% of plans, with 18% relating to target volumes and the rest relating to OARs. ^{Reference Lo, Liu and Chan10} A report of stereotactic radiotherapy cases in Australia found suggested plan changes in 22.3% of cases, though the rate of implementation of changes was unclear. ^{Reference Fitzgerald, Pryor and Aland12} A 2015 single-institution study of 513 SBRT cases found that changes were implemented in 22.6% of cases, with lower rates of change in higher-volume subsites including liver (18%) and lung (17%) cases. ^{Reference Matuszak, Hadley and Feng11} The proportion of suggested changes which were not implemented was not reported. These suggest that there is a higher rate of recommended plan changes for SBRT cases compared to non-SBRT cases, which is particularly relevant given the increasing utilisation of SBRT. We found a frequency of 17% for ‘learning moments’, while the rate of implemented plan changes was much lower at 3%.

The proportion of plan changes actually implemented following suggestion of a change was variably reported in the literature. For general radiotherapy cases, Walburn et al. found that radiation oncologist compliance with suggested changes was only 59%, with lower rates of compliance for more significant plan changes—suggesting that radiation oncologistss are reluctant to implement major changes. ^{Reference Walburn, Wang and Sud9} This may also imply that a substantial proportion of suggested changes were deemed to be of educational value rather than direct clinical significance. When considering the high rate of learning moments (28.7%) for our lower-volume treatment sites, it may be inferred that discussions of educational value occur more commonly when reviewing these cases. Time and resource limitations often mean that only selected cases can undergo peer review; thus, one of the factors considered when prioritising cases for review could be case volume. ^{Reference Huo, Gorayski, Poulsen, Thompson and Pinkham6}

We found a statistically significant difference in the rate of learning moments between higher- and lower-volume SBRT sites. This may be due to the presence of established institutional protocols for higher-volume sites and a larger body of evidence and guidelines. ^{Reference Shanker, Liu, Lee, Stuart, Powell, Wigg and Pryor17–Reference Vellayappan, Chao and Foote20} This is consistent with findings by Matuszak et al. for their cohort of SBRT cases, with lower rates of change noted for high-volume subsites. ^{Reference Matuszak, Hadley and Feng11} Compared with their series, our rate of suggested plan changes was lower at 3% compared with 22.6%. One reason for this may be the timeframe in which the analyses took place—our series is from a more recent time period (i.e., 2015–2018) compared to 2012–2015 in their series. During this intervening period, the evidence for SBRT has grown considerably, as well as utilisation rates and availability of SBRT-specific guidelines and protocols. ^{Reference Ma, Wang, Tseng and Sahgal21} Furthermore, the rate of changes from peer review tends to reduce over time due to the development of standardised protocols and the convergence of individual practices towards agreed standards. ^{Reference Brammer, Pettit and Allerton16,Reference Cox, Kapur, Sharma, Lee, Bloom, Sharma, Goode and Potters22} A second reason for the discrepancy may be that changes suggested at peer review are not always implemented.^{[Reference Walburn, Wang and Sud9]} Brundage et al. found that a high proportion of changes suggested from peer review were not actually implemented (51.2%), but instead served as guidance for future cases.^{[Reference Brundage, Dixon and Mackillop5]} This suggests there is a degree of subjectivity in radiotherapy planning, with the inference that there is a degree of acceptable plan variability for each case. Our data suggest that the rate of absolute critical errors is low; thus, there is a clear difference between what is considered to be outside of the accepted norm, and what is a critical safety issue. Finally, many cases in our department undergo individual one-on-one peer review before being formally presented at peer review rounds. As such, this process may have filtered out some of the changes that would otherwise be suggested at peer review rounds.

The overall rate of critical errors was low at 0.2% (2 cases). Critical errors in our institution were defined as issues which could compromise patient safety—in our series, these included an inadequate planning target volume margin and an incorrect image dataset used for co-registration from which volumes were defined. In both cases, a systematic root cause analysis was conducted to improve processes for subsequent cases. It is unclear from previous reports how frequently errors of this gravity were detected in peer review rounds.^{[Reference Matuszak, Hadley and Feng11,Reference Fitzgerald, Pryor and Aland12]} It should also be noted that there was a large difference between the rate of actually implemented plan changes and the rate of critical errors (3% versus 0.2%). It is likely that many suggested or implemented plan changes are presented as non-critical suggestions as opposed to compulsory changes deemed necessary for patient safety/quality of care.

As with any retrospective review, there are a number of limitations with our study. It was apparent that not all cases had a full set of peer review codes recorded: out of 1292 cases undergoing peer review, 934 had specific information regarding which plan aspects were reviewed and learning moments. It could be argued that our data may not be completely representative due to this discrepancy. However, these cases represented a minority for most treatment subsites. For prostate, soft tissue and kidney, they represented over 50% of cases—though these subsites comprised less than 5% of all cases. Furthermore, these were subsites which lacked significant evidence and standardised institutional guidelines during the timeframe of this study. Additionally, the time pressure inherent to peer review rounds and the manual nature of recording additional peer review elements in the electronic system may have contributed to incomplete recording of some elements as described. Indeed, time constraints are known to be a significant barrier to the facilitation of peer review.^{[Reference Caissie, Rouette and Jugpal23]} Furthermore, our institutional system relies on radiation therapists being present at meetings and recording as the radiation oncologists discuss each case. This results in a degree of familiarity required with the discussion and recording process; thus, the reliability of results may be affected if responsible staff are away or there is ambiguity in the discussion. Based on these findings, we are actively reviewing our peer review processes at our institution in accordance with recommendations by Marks et al.^{[Reference Marks, Adams and Pawlicki4]}

A further limitation to our study is that the proportion of ‘learning moments’ which were suggested changes versus educational discussion is unknown. Finally, there was no specific recording of implemented plan changes; the only way to determine which plans underwent changes was to search for plan amendments. It is possible that our result thus underestimates the number of actual plan changes. Through undertaking this study, a number of areas for future improvement in our processes have been identified, such as more standardised outcome measures, ‘learning moment’ differentiation between educational discussions and suggested changes, confirmation of implemented plan changes and a more streamlined system of data capture.