Session Item

Organ at risk contouring is an important but time-consuming radiotherapy process that has become increasingly automated in recent years through computers and artificial intelligence. This has led to the development of automated contouring algorithms. However, to validate such contouring algorithms, guidelines are needed, and quantification of inter-expert observer variance can help to comprehend the quality of automated contours.

The project aimed to quantify the inter-observer variance of OAR in the head and neck region across DAHANCA oncology experts.

The DAHANCA OAR contouring guidelines were clarified and discussed through a two-day workshop. The discussions were based on a pre-workshop national contouring audit where 20 expert oncologists contoured 17 OARs of the same patient. Subsequently, 15 experts were asked to contour 26 patients at the workshop to collect multiple contours on the same patient.

For each patient, all contours were compared in pairs using the Dice index, mean surface distance (MSD), and 95% Hausdorff distance. To illustrate 3D spatial areas of larger contouring variance, all OAR were deformable registered to a reference patient, and the average contouring variance was displayed (fig 1).

In total, 3545 OAR were delineated in 26 patients, 17 per patient. The median number of observers per organ was 9 (IQR 7-9). The most familiar organs, like the brain stem, oral cavity, parotid and submandibular glands, esophagus and thyroid, showed high median dice, above 0.8. The most recent guidelines with new OAR, like pharynx constrictor muscles (PCM), lips, and buccal mucosa, had the lowest median dice, below 0.6.

The structures with the highest tissue contrast, like submandibular and thyroid glands, showed median MSD of just above 1 mm, while most other OAR had median MSD below 3 mm (fig 2). The spinal cord showed a very large variance linked to observers' inconsistency in following the guidelines of contouring the full length in the caudal direction.

Most information on inter-observer variation is found in the variance plots (fig 1), from which it is clear that the deep part of the parotid gland had the highest uncertainty. The transition between PCM low and esophagus was not well defined. Likewise, there is roughly one CT slice of variance for the transition between the glottic larynx and supraglottic larynx.  

A national auto-segmentation validation repository of 26 patients has been created. Areas of low image contrast present more contouring variance, and for OAR split into substructures, the transition causes a significantly increased variance.