Amazing stuff!
"... A “mind-reading” AI model called Brain-IT ... can reconstruct [a visual] scene based solely on the brain activity of a person viewing it ...
Brain-IT, by contrast, recognizes patterns of brain activity that are shared by all of us, and translates them into images. ...
“There exist nowadays models that translate brain activity into images, and they can even produce impressive reconstructions that preserve the semantic meaning of the image reasonably well,” ... “However, they tend to make mistakes in basic features such as composition and color.
The new model we developed outperforms them in reconstructing both the content of the image and its details. What’s more, while every other model requires dozens of hours of brain scans to learn to ‘read’ a new person, our model needs only one hour.” ..."
From the abstract:
"Reconstructing images seen by people from their fMRI brain recordings provides a non-invasive window into the human brain. Despite recent progress enabled by diffusion models, current methods often lack faithfulness to the actual seen images.
We present ``Brain-IT'', a brain-inspired approach that addresses this challenge through a Brain Interaction Transformer (BIT), allowing effective interactions between clusters of functionally-similar brain-voxels. These functional-clusters are shared by all subjects, serving as building blocks for integrating information both within and across brains.
All model components are shared by all clusters & subjects, allowing efficient training with limited amount of data.
To guide the image reconstruction, BIT predicts two complementary localized patch-level image features:
(i) high-level semantic features which steer the diffusion model toward the correct semantic content of the image; and
(ii) low-level structural features which help to initialize the diffusion process with the correct coarse layout of the image.
BIT's design enables direct flow of information from brain-voxel clusters to localized image features.
Through these principles, our method achieves image reconstructions from fMRI that faithfully reconstruct the seen images, and surpass current SotA approaches both visually and by standard objective metrics.
Moreover, with only 1-hour of fMRI data from a new subject, we achieve results comparable to current methods trained on full 40-hour recordings."
Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer (preprint, open access)
No comments:
Post a Comment