Abstract
Glaucoma is the leading cause of irreversible vision loss worldwide. Most approaches rely on unimodal data, failing to fully leverage the complementarity between multimodal information. Furthermore, clinically acquired fundus photography often suffer from poor quality issues such as noise interference and low contrast. In addition, effectively integrating data from different modalities remains a significant challenge. To address these challenges, this paper proposes a glaucoma diagnosis and progression prediction model (MUGLDEM) based on deep multimodal fusion and collaborative attention mechanisms. This model integrates multimodal data from the UK Biobank including clinical text, fundus photography and optical coherence tomography (OCT). For low-quality fundus photography, MUGLDEM integrates non-local denoising and contrast-constrained histogram equalization algorithms to improve the quality of image. Then, MUGLDEM independently designs feature extraction models for different modal data to preserve the feature integrity of different modal. Further, MUGLDEM effectively fuses multimodal features by designing dual-gate residual attention module and a bidirectional cross-attention module. Finally, the fused multimodal features are utilized to diagnose glaucoma and predict its progression. The experimental results on the UK Biobank demonstrate that MUGLDEM achieves state-of-the-art performance in both glaucoma diagnosis and progression prediction tasks. The source code for this work is available on GitHub (https://github.com/lanbiolab/MUGLDEM).