Medical images are routinely used to evaluate treatment response in oncology and in clinical trials, typically by measuring tumor changes before and after treatment (RECIST). Since manual measurement of tumor is prone to errors and to inter- and intra-observer variability, ongoing research aims for automation of (volumetric) RECIST assessment. Amongst others, convolutional neural networks have been used to automatically evaluate RECIST scores. Whereas change in tumor or volume is a simple yet effective method to evaluate response to treatment, response patterns may be heterogeneous and complex. Therefore, methods to effectively extract dynamic information from medical images should be explored. Personalized medicine could be improved by capturing the longitudinal information in medical images for, amongst others, disease monitoring, treatment evaluation, or outcome prediction. Besides the use of longitudinal image analysis during or after treatment, accurate outcome prediction prior to treatment could allow for the choice of the right treatment or even a watch-and-wait approach.
Compared to 1D signal analysis, there are additional challenges when evaluating longitudinal 2D and 3D imaging, including registration, reliable segmentation, inconsistent imaging intervals, and sparse data. Longitudinal image analysis could be performed by modelling hand-crafted features, e.g. delta-radiomics or longitudinal radiomics. This requires accurate and consistent segmentation of the regions of interest, as well as standardization of image acquisition- and reconstruction protocols. Furthermore, the sparsity of the data is often a major challenge. While deep learning has been successfully used for multiple classification and prediction tasks, it hasn’t been widely explored for longitudinal medical image analysis. Nevertheless, deep learning is able to automatically learn complex and abstract representations from large amounts of data and identify subtle changes that are difficult for the human eye to detect. Besides modelling temporal relationships, deep learning could be used to fuse information extracted from different image modalities to make more accurate predictions. Ideally, longitudinal information from cross-modality images would be captured and integrated with information from complementary biomarkers. Some examples on longitudinal outcome prediction using deep learning can be found in literature. This ranges from the use of longitudinal magnetic resonance imaging to predict response after neoadjuvant chemoradiotherapy for rectal cancer patients, to the development of a deep learning-based deformable image registration algorithm proposed for longitudinal imaging studies and adaptive radiotherapy.