Optimal Pre-training for Vision Transformers in Medical Image Classification

Modality-adaptive transfer learning is crucial for advancing automated medical image analysis, particularly under data scarcity. In this work, we present a systematic study of modality-aligned pre-training for Vision Transformers (ViT) and Convolutional Neural Networks (CNN) on retinal optical coherence tomography (OCT) classification. Through controlled experiments across a broad range of data regimes (from 10 to 2000 labelled samples per class), we show that ViT models pre-trained on a physics-consistent OCT domain (breast tissue) achieve substantial performance gains in the few-shot setting, dramatically outperforming both ImageNet pre-training and random initialization. Conversely, transferring a retina-OCT-pre-trained ViT to a binary breast-OCT task lifts accuracy from 85.9% to 99.98% with only five training images per class, confirming bidirectional generalizability. Notably, this benefit does not extend to CNNs, which show little or no improvement from modality alignment. Visualization of self-attention maps reveals that modality-aligned ViTs more effectively focus on clinically relevant features when labelled data are limited, whereas all models converge as sample size increases. These findings highlight the critical interplay between network architecture, pre-training strategy, and data modality for medical imaging applications, and provide new insights into the unique transferability of self-attention-based models under real-world clinical constraints.