An interdisciplinary, case-based tool which simulates suboptimal treatment plans is a feasible and effective strategy to supplement radiation oncology resident education. Similar tools can be implemented in other skill-based specialties, such as surgical subspecialties, and other health-related disciplines, such as nursing and public health.
Abstract
Introduction: Radiation treatment plan evaluation is a critical component of radiation oncology practice. Despite its importance, prior studies have reported that most residents are interested in additional training and resources. We report our initial experience with a case-based educational curriculum to increase confidence and competence in plan evaluation.
Description of Innovation: In 2022, physicists, dosimetrists, and medical residents at a single institution developed cases that incorporated problems identified during plan evaluation. Three de-identified plans were selected and distributed to residents for voluntary independent review. Residents then participated in a teaching session to review cases. Following the session, participants were invited to complete anonymous surveys regarding the cases and didactic session.
Results: Cases incorporated problems encountered during plan evaluation. Half of eligible residents completed the case-based curriculum. Most correctly identified the major issues with each case. Only 57% and 42% correctly identified image misregistration leading to violated OAR constraints, respectively; 29% accurately listed the pros and cons of specific beam arrangements; and 14% identified techniques dosimetry could use to improve target coverage. On the follow-up survey, all residents reported that the cases and the review session improved their confidence. They indicated that they would like to see similar educational material incorporated into future training.
Conclusion: We describe an interdisciplinary exercise to supplement resident education in radiation treatment plan evaluation. This curriculum simulates suboptimal treatment plans and provides strategies to collaborate with team members to improve them. Similar strategies may prove useful in the education of other skill-based specialty trainees.
INTRODUCTION
Evaluating radiation treatment plans is an essential component of patient care for a radiation oncologist. Radiation treatment planning and evaluation is the process of using a CT simulation image, in conjunction with additional diagnostic imaging, to design high quality and safe radiation fields using treatment planning software. The goal of the planning process is to deliver high doses of radiation to the tumor (target volume) while limiting dose to normal tissues (organs at risk). As treatment planning techniques, including software automation, become increasingly complex, the importance of precise and accurate plan evaluation rises. Despite the critical importance of this skillset, many radiation oncology residents continue to feel uncomfortable with plan evaluation, with 60% reporting inadequate exposure to treatment planning during residency (Brower et al., 2021, Wu et al., 2020). Prior studies and surveys have suggested that this lack of comfort in plan evaluation results from lack of formal plan evaluation curriculum, lower numbers of cases per resident, and lack of independent plan review (Brower et al., 2021, Wu et al., 2020).
There is significant variation in resident education regarding treatment planning and plan evaluation across institutions, and efforts to standardize continue to be limited. Notably, Dean et al. proposed a systematic approach to plan evaluation using the acronym CB-CHOP, which stands for sequentially reviewing contours, beam arrangements/fields, coverage of targets, heterogeneity/hotspots, organs at risk (OARs), and prescription. This acronym is an example of a simple and easily distributed learning tool which helped standardize the plan evaluation process for many residents and providers (Dean et al., 2017). That said, many residency graduates are underprepared for independent practice of plan evaluation, with numerous studies demonstrating significant plan changes in peer-review (Brunskill et al., 2017, Martin-Garcia et al., 2020, Vijayakumar et al., 2019, Hoopes et al., 2015). Thus, there is an unmet need for additional resources which can simulate the real-world practice in a low stake setting. Here, we discuss an interactive case-based educational tool delivered to residents at a single institution in an effort to improve radiation treatment plan evaluation education. As many specialties struggle with parallel challenges in the training of residents, similar tools can be adapted to other contexts.
DESCRIPTION OF INNOVATION
Deidentified central nervous system (CNS) radiation treatment plans from previously treated patients were altered to create three test cases to highlight problems seen in suboptimal radiation plans. Faculty and resident radiation oncologists collaborated with dosimetrists and physicists to generate the flawed cases in the treatment planning software. Cases incorporated common problems encountered during radiation plan evaluation. The problems incorporated into the cases included inaccurate image registration, doses to OARs exceeding normal tissue constraints, multiple options for beam arrangements, and inadequate target volume coverage (Figures 1-4).
In May 2022 radiation oncology residents of all training levels (n = 14) at a single institution were invited to review each test case (Supplements A, B, & C) and complete anonymous surveys (Supplement D) designed to assess their evaluation of each plan. The answer key for case questions is available in Supplement D. Residents involved with case development were not eligible to participate in the educational tool or surveys. Residents were given four weeks to complete the activity. Participation was voluntary.
Case 1 was a radiation treatment plan for a test patient receiving adjuvant therapy for a high-grade glioma using standard CNS target volumes, doses, and OARs. Inaccurate image registration and critical OARs exceeding standard dose constraints were incorporated (Supplement A). The optic nerves and chiasm were contoured in the incorrect location due to misregistration of MRI to the planning CT (Figure 1). Trainees were asked to evaluate this plan and answer eight survey questions. Questions focused on the CB-CHOP method of plan evaluation. Trainees were asked if they would accept the plan, to justify their decision, and to explain any issues they may have identified.
Case 2 included two treatment plan options for a test patient receiving adjuvant radiation for a low-grade glioma (Supplement B). The two plans used different beam arrangements, one used non-coplanar arcs and one used coplanar arcs (Figure 3). Depending on the clinical scenario and plan objectives, one beam arrangement may be more advantageous. The survey asked trainees to describe the differences between the two plans and any pros or cons of each.
Left: An axial slice of a T1 weighted post contrast MRI scan of the brain that had been misregistered to the treatment planning CT scan is shown. Right: The treatment planning CT scan with the optic structures contoured using the misregistered MRI is shown. This image highlights that the contoured optic nerves are not in the correct location.
Treatment planning CT scan from Case 1 showing the correctly contoured optic structures in green, the incorrectly contoured optic structures in orange, and the 60 Gy isodose line in red. The 60 Gy isodose line overlaps with the true optic nerves, however it does not overlap with the incorrectly drawn optic nerves.
The plan on the left was made using non-coplanar partial arcs while the plan on the right was created using two coplanar arcs. Isodose lines and their corresponding dose levels are shown. This figure shows that the plan on the right has more low-dose spill to the non-target normal brain.
The dose volume histogram for two adjuvant radiation plans is shown. The planning target volumes (PTVs) for both plans are in red. The plan using the optimization structure is delineated using triangles and the plan generated without the optimization structure is delineated using squares. The plan generated using the optimization structure meets the coverage goal of at least 95% of the volume of the PTV receiving 100% of the prescription dose, while the other plan does not meet this objective.
Case 3 included two treatment plan options for a test patient receiving adjuvant radiation for a high-grade glioma (Supplement C). This case was designed to assess trainees’ abilities to detect inadequate target volume coverage and to troubleshoot ways to improve coverage while respecting normal tissue dose constraints. In this case, the target was close to, but not contacting the optic nerves and chiasm. Trainees were asked which plan they would approve and if they could identify techniques dosimetry might use to improve the plan they did not select.
Following completion of the cases, a one-hour long interactive session led by dosimetry reviewed the cases and concepts they were attempting to highlight. Each case was thoroughly discussed, including case survey questions. Trainees were encouraged to ask question related to the cases, as well as other difficult plan evaluation concepts. An anonymous follow-up survey to assess the teaching session was disseminated (Supplement E). This study and related surveys were submitted to the Institutional Review Board and found to be exempt from further review.
Following completion of the cases, a one-hour long interactive session led by dosimetry reviewed the cases and concepts they were attempting to highlight. Each case was thoroughly discussed, including case survey questions. Trainees were encouraged to ask question related to the cases, as well as other difficult plan evaluation concepts. An anonymous follow-up survey to assess the teaching session was disseminated (Supplement E). This study and related surveys were submitted to the Institutional Review Board and found to be exempt from further review.
RESULTS
Seven of 14 (50%) eligible trainees completed case review and accompanying surveys. Surveys took an average of 11.9 minutes to complete (range 8.7-13.8 minutes). Upon review of Case 1, fifty-seven percent of residents (n=4) correctly indicated that they would not approve the contours, 42% of residents (n=3) indicated that they would not approve the based upon OAR doses, and 71% of residents (n=5) indicated that they would not accept the plan overall (Figure 5). For Case 2 fifty-seven percent of residents (n=4) accurately identified that the difference between the two plans was the use of non-coplanar arcs vs coplanar arcs, and 29% of residents (n=2) accurately listed the pros and cons of these different beam arrangements (Figure 5). For Case 3, seventy-one percent of residents (n=5) accurately selected the superior plan; however, only 14% of residents (n=1) could describe
appropriate techniques they would discuss with dosimetry to improve this plan (Figure 5).
All residents (100%) participated in the interactive case review session held by dosimetry. On the follow-up survey, all residents (100%) responded that both the cases and the review session improved their confidence and understanding of treatment planning and plan evaluation (Figure 5). All respondents (100%) said they would like to see more educational material similar to this tool incorporated in residency training in the future (Figure 5).
DISCUSSION
Our single institution experience with an education initiative to improve plan evaluation training for radiation oncology residents was well-received by our participants. Four problems that occur in real-world practice were incorporated into three interactive cases. This case-based educational initiative fulfils an unmet need to improve treatment planning and plan evaluation curricula in radiation oncology residencies that has been identified in prior studies (Brower et al., 2021, Wu et al., 2020, Moideen et al., 2020). It appeals to multiple trainees, including those with visual, kinesthetic, analytical, and social learning styles. Similar case-based simulations can be developed in other specialties where residents may benefit from resources to improve confidence in specialty-specific skills.
It was encouraging that most residents correctly identified the issues with each case; 71% rejected a plan exceeding OARs in case 1, 57% identified the use of co-planar and non-coplanar beam arrangement in case 2, and 71% selected the plan with better target coverage in case 3. However, many residents struggled with the questions aimed to test their understanding of more intricate challenges with plan evaluation. In case 1, only 47% of respondents correctly identified an error in normal tissue dosing based on an image-fusion error. In Case 2, 29% accurately listed the pros and cons of co-planar vs non-co-planar beam arrangements. In Case 3, a small minority (14%) of respondents were able to provide methods by which dosimetry could improve target coverage when critical OARs were in close proximity. This highlights an area for potential improvement in resident education.
A rise in auto-contouring tools and templated plan goals and approval strategies may contribute to training deficits in certain more nuanced concepts of treatment plan evaluation. Moreover, image fusion is a critical component of segmentation that is rarely subject to peer review. Additionally, in most residency programs, faculty physicians provide guidance with contours and plan approvals. However, with the rise of remote work, radiation treatment plans often go through multiple iterations with dosimetry without resident involvement. This experience utilizes a face-to-face teaching session with medical physics and dosimetry instructors. Engaging the interdisciplinary educators within the radiation oncology department may prove invaluable for resident education as demands for faculty physician time increase.
While prior data supports contouring modules for residents, data continues to be limited regarding plan evaluation (Abugideiri et al., 2019, Gillespie et al., 2017). An international pilot study demonstrated that an interactive tool providing active feedback regarding head and neck contouring to residents at Addis Ababa University in Ethiopia improved trainees’ confidence significantly. Another study demonstrated the benefit of the “near-peer” teaching of radiation treatment planning for residents (Walls et al., 2022). Case-based plan evaluation initiatives more similar to ours have been completed in Canada, however literature remains limited in the United States (Walls et al., 2022, Roumeliotis et al., 2022, Winter et al., 2020, Nabavizadeh et al., 2016).. Notably, at Princess Margaret, a case bank of 70 cases for plan evaluation was developed with positive subjective feedback from residents (Winter et al., 2020).
This educational experience is limited by sample size both in the number of trainees assessed and the number of teaching cases available. The lower-than-expected participation of eligible residents was largely attributed to multiple residents being on vacation and studying for board exams during the timing of this activity. We sought to maximize participation in this study by making assessments anonymous, although this limited our ability to evaluate any differential in effectiveness of this teaching tool by residents’ training levels.
In the future, we plan to develop cases that correspond to each disease site covered in trainee didactics. Ultimately, this will result in a searchable case library that residents can refer to for additional practice. In addition, this data was collected from a single institution. However, we believe that it would be possible to expand the use of this tool to other residency programs and subspecialties. For example, a surgical subspecialty may choose to develop case-based simulations to help trainees identify common mistakes in a surgical procedure. Alternatively, a nursing student could use case-based simulations to work through challenges with providing emergent care for a patient with cardiac arrest. Further, public health trainees can use this approach to replicate real-world healthcare crises and possible solutions.
CONCLUSION
Our experience suggests that a case-based educational tool for plan evaluation is feasible and effective in improving trainee confidence and understanding of plan evaluation. This initiative meets a previously identified need for increased instruction in radiation plan evaluation for residents. This tool could be expanded to encompass additional disease sites and include a more comprehensive list of common treatment plan evaluation concepts. Further, it can be adapted and incorporated into other specialties. Additional investigation is required to understand the longitudinal effects of this tool on confidence and preparedness for independent clinical practice.
ACKNOWLEDGMENTS
The authors received no grant support for this work and declare they have no conflicts of interest in regard to this work. We would like to thank all residents who participated in this educational initiative.
Abugideiri, M., Schreibmann, E., Switchenko, J., McDonald, M. W., Beitler, J. J., Curran, W. J., Bruner, D., Patel, P., Tigeneh, W., Mijena, M., Tian, S., Dhabaan, A., Esiashvili, N., Liu, T., & Ali, A. N. (2019). Prospective International Pilot Study Evaluating the Efficacy of a Self-Guided Contouring Teaching Module With Integrated Feedback for Transitioning From 2D to 3D Treatment Planning. Journal of global oncology, 5, 1–16. https://doi.org/10.1200/JGO.18.00224
Brower, J. V., Chen, S., Ritter, A., Liauw, S. L., Rosenberg, S. A., Reddy, A. V., Golden, D. W., Gillespie, E. F., & Mattes, M. D. (2021). Comfort Level of US Radiation Oncology Graduates: Assessment of Transition to Independent Clinical Practice. Journal of cancer education : the official journal of the American Association for Cancer Education, 36(2), 278–283. https://doi.org/10.1007/s13187-019-01625-z
Brunskill, K., Nguyen, T. K., Boldt, R. G., Louie, A. V., Warner, A., Marks, L. B., & Palma, D. A. (2017). Does Peer Review of Radiation Plans Affect Clinical Care? A Systematic Review of the Literature. International journal of radiation oncology, biology, physics, 97(1), 27–34. https://doi.org/10.1016/j.ijrobp.2016.09.015
Dean, M., Jimenez, R., Mellon, E., Fields, E., Yechieli, R., Mak, R (2017). CB-CHOP: A simple acronym for evaluating a radiation treatment plan. Appl Rad Oncol. 6(4):28-30.
Gillespie, E. F., Panjwani, N., Golden, D. W., Gunther, J., Chapman, T. R., Brower, J. V., Kosztyla, R., Larson, G., Neppala, P., Moiseenko, V., Bykowski, J., Sanghvi, P., & Murphy, J. D. (2017). Multi-institutional Randomized Trial Testing the Utility of an Interactive Three-dimensional Contouring Atlas Among Radiation Oncology Residents. International journal of radiation oncology, biology, physics, 98(3), 547–554. https://doi.org/10.1016/j.ijrobp.2016.11.050
Hoopes, D. J., Johnstone, P. A., Chapin, P. S., Kabban, C. M., Lee, W. R., Chen, A. B., Fraass, B. A., Skinner, W. J., & Marks, L. B. (2015). Practice patterns for peer review in radiation oncology. Practical radiation oncology, 5(1), 32–38. https://doi.org/10.1016/j.prro.2014.04.004
Martin-Garcia, E., Celada-Álvarez, F., Pérez-Calatayud, M. J., Rodriguez-Pla, M., Prato-Carreño, O., Farga-Albiol, D., Pons-Llanas, O., Roldán-Ortega, S., Collado-Ballesteros, E., Martinez-Arcelus, F. J., Bernisz-Diaz, Y., Macias, V. A., Chimeno, J., Gimeno-Olmos, J., Lliso, F., Carmona, V., Ruiz, J. C., Pérez-Calatayud, J., Tormo-Micó, A., & Conde-Moreno, A. J. (2020). 100% peer review in radiation oncology: is it feasible?. Clinical & translational oncology : official publication of the Federation of Spanish Oncology Societies and of the National Cancer Institute of Mexico, 22(12), 2341–2349. https://doi.org/10.1007/s12094-020-02394-8
Moideen, N., de Metz, C., Kalyvas, M., Soleas, E., Egan, R., & Dalgarno, N. (2020). Aligning Requirements of Training and Assessment in Radiation Treatment Planning in the Era of Competency-Based Medical Education. International journal of radiation oncology, biology, physics, 106(1), 32–36. https://doi.org/10.1016/j.ijrobp.2019.10.005
Nabavizadeh, N., Burt, L. M., Mancini, B. R., Morris, Z. S., Walker, A. J., Miller, S. M., Bhavsar, S., Mohindra, P., Kim, M. B., Kharofa, J., & ARRO Executive Committee (2016). Results of the 2013-2015 Association of Residents in Radiation Oncology Survey of Chief Residents in the United States. International journal of radiation oncology, biology, physics, 94(2), 228–234. https://doi.org/10.1016/j.ijrobp.2015.10.014
Roumeliotis, M., Morrison, H., Conroy, L., Becker, N., Logie, N., Grendarova, P., Thind, K., McNiven, A., Hilts, M., & Quirk, S. (2022). Competency-Based Medical Education in Radiation Therapy Treatment Planning. Practical radiation oncology, 12(3), e232–e238. https://doi.org/10.1016/j.prro.2021.12.003
Vijayakumar, S., Duggar, W. N., Packianathan, S., Morris, B., & Yang, C. C. (2019). Chasing Zero Harm in Radiation Oncology: Using Pre-treatment Peer Review. Frontiers in oncology, 9, 302. https://doi.org/10.3389/fonc.2019.00302
Walls, G. M., Ellis, R., Lynch, S., Flynn, M. A., McCann, G., Jellett, L. J., & Harrison, C. (2022). Near-Peer Teaching in Radiation Oncology: a Proof of Principle Study for Learning Treatment Planning. Journal of cancer education : the official journal of the American Association for Cancer Education, 37(4), 1245–1250. https://doi.org/10.1007/s13187-022-02150-2
Winter, J. D., Adleman, J., Purdie, T. G., Heaton, J., McNiven, A., & Croke, J. (2020). An Innovative Learning Tool for Radiation Therapy Treatment Plan Evaluation: Implementation and Evaluation. International journal of radiation oncology, biology, physics, 107(4), 844–849. https://doi.org/10.1016/j.ijrobp.2020.03.018
Wu, S. Y., Sath, C., Schuster, J. M., Dominello, M. M., Burmeister, J. W., Golden, D. W., & Braunstein, S. E. (2020). Targeted Needs Assessment of Treatment Planning Education for United States Radiation Oncology Residents. International journal of radiation oncology, biology, physics, 106(4), 677–682. https://doi.org/10.1016/j.ijrobp.2019.11.023
Naba Ali, MD
Chief Resident, Radiation Oncologist, Department of Radiation Oncology, Winship Cancer Institute at Emory University, Atlanta, GA
Jonathan Wolf, CMD
Medical Dosimetrist, Northeast Georgia Health System Radiation Oncology, Braselton, GA
Pretesh R. Patel, MD
Associate Professor, Radiation Oncologist, Department of Radiation Oncology, Winship Cancer Institute at Emory University, Atlanta, GA
Jolinta Y. Lin, MD
Associate Professor, Radiation Oncologist, Department of Radiation Oncology, Winship Cancer Institute at Emory University, Atlanta, GA
Marian Axente, PhD, DABR
Assistant Professor, Medical Physicist, Department of Radiation Oncology, Winship Cancer Institute at Emory University, Atlanta, GA
James E. Bates, MD
Assistant Professor, Radiation Oncologist, Department of Radiation Oncology, Winship Cancer Institute at Emory University, Atlanta, GA
James R. Janopaul-Naylor, MD
Assistant Attending, Radiation Oncologist, Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, New York, NY
Ashley J. Schlafstein, MD
Radiation Oncologist, Lexington Medical Center Radiation Oncology, West Columbia, SC