Comparing multimodal and unimodal representation learning for Rare cancer prediction

Multimodal fusion of Whole-Slide Images (WSIs) and RNA-Sequencing (RNASeq) is expected to improve clinical predictions by integrating complementary morphological and molecular information, yet this premise remains largely untested in rare cancers where sample scarcity limits model training. Here, we evaluate representations combined via multimodal fusion (TANGLE, CCA) against unimodal baselines (RNAseq, UNI, CONCH variants) on the tasks of survival analysis, histological type classification, and pathological stage prediction across three rare cancer cohorts (sarcoma, mesothelioma, and uterine carcinosarcoma). We find that the dominant unimodal embedding varies by task: survival performance is largely cohort-dependent, RNAseq surpasses other representations at subtyping, and WSI encoders perform best on staging. No fused representation provides a consistent advantage over these unimodal baselines. TANGLE approaches but does not exceed unimodal performance meaningfully, while CCA consistently underperforms, amplifying noise from the weaker modality. We observe low similarity between modalities (measured using Centered Kernel Alignment), but the distinctness of representations does not translate into complementary predictive value for downstream tasks. Our findings establish empirical boundaries on multimodal learning in rare cancer settings characterised by limited data.