AI model predicts cancer prognoses, responses to treatment for precision oncology as opposed to pure diagnosis

Good news! Cancer is history (soon)!

"The melding of visual information (microscopic and X-ray images, CT and MRI scans, for example) with text (exam notes, communications between physicians of varying specialties) is a key component of cancer care.  ...

Now researchers at Stanford Medicine have developed an AI model able to incorporate visual and language-based information. After training on 50 million medical images of standard pathology slides and more than 1 billion pathology-related texts, the model outperformed standard methods in its ability to predict the prognoses of thousands of people with diverse types of cancer, to identify which people with lung or gastroesophageal cancers are likely to benefit from immunotherapy, and to pinpoint people with melanoma who are most likely to experience a recurrence of their cancer. ...

Although artificial intelligence tools have been increasingly used in the clinic, they have been primarily for diagnostics (does this microscope image or scan show signs of cancer?) rather than for prognosis (what is this person’s likely clinical outcome, and which therapy is most effective for an individual?). ..."

From the abstract:

"Clinical decision-making is driven by multimodal data, including clinical notes and pathological characteristics. Artificial intelligence approaches that can effectively integrate multimodal data hold significant promise in advancing clinical care. However, the scarcity of well-annotated multimodal datasets in clinical settings has hindered the development of useful models.

In this study, we developed the Multimodal transformer with Unified maSKed modeling (MUSK), a vision–language foundation model designed to leverage large-scale, unlabelled, unpaired image and text data. MUSK was pretrained on 50 million pathology images from 11,577 patients and one billion pathology-related text tokens using unified masked modelling. It was further pretrained on one million pathology image–text pairs to efficiently align the vision and language features.

With minimal or no further training, MUSK was tested in a wide range of applications and demonstrated superior performance across 23 patch-level and slide-level benchmarks, including image-to-text and text-to-image retrieval, visual question answering, image classification and molecular biomarker prediction. Furthermore, MUSK showed strong performance in outcome prediction, including melanoma relapse prediction, pan-cancer prognosis prediction and immunotherapy response prediction in lung and gastro-oesophageal cancers. MUSK effectively combined complementary information from pathology images and clinical reports and could potentially improve diagnosis and precision in cancer therapy."

AI model predicts cancer prognoses, responses to treatment | Stanford Report "Researchers at Stanford Medicine have developed an artificial intelligence tool that is able to incorporate data from both medical images and text."

A vision–language foundation model for precision oncology (no public access)