Approved Research

Cross-modal medical analysis and reasoning

Lay summary

The cross-modal medical knowledge graph (a collection like a network that contained relationships between different types of medical data, e.g., images, text, etc.) is becoming critical for both academic research and clinical applications. Generating a representation that could reflect different modalities (organized by knowledge graph) remains challenging and underdeveloped. We will a) rely on the existing medical expert knowledge of the clinical disease to construct the basic structure for the knowledge graph, b) use statistic models to induce the features and relations from different data modalities in the UKBB and incorporate with the basic structure, c) use optimization algorithms to refine the structure of knowledge graph, d) use artificial intelligence methods to compute the similarity between two patients, and e) find the similar patient according to the clinical requirement. The UKBB data is essential to our knowledge graph construction because of its rich information from multiple modalities. Moreover, we will adjust the structure of the knowledge graph according to the updates in the field of medical science. Besides, we will also study the approaches to explain the process of computer reasoning. The project duration is 3 years. It can support the clinical decision, narrow the gap of medical resources between regions, improve the overall medical service level. In addition, UKBB data will also be used as a benchmark dataset for evaluating the related methods of the cross-modal medical knowledge graph.