Abstract: PURPOSE: Recent explorations in neuroimaging studies have focused on using functional magnetic resonance imaging (fMRI) techniques to decode content information from human brain activity when viewing relatively simple background visual scenes. The response signals at multiple time points contain more decoding information than the peak or average response of the cerebral cortex induced by visual stimuli. In this study, we propose a decoding model with sequential relationship simulation capabilities that makes it possible to decode detailed category information about natural images with complex backgrounds from human brain activity signals measured by fMRI.
METHODS: This decoding model is constructed by the long short-term memory (LSTM) network in order to simulate the relationship of visual response signals at multiple time points induced by complex natural images to improve decoding accuracy.
RESULTS: The results show that the decoding accuracy of five types of natural images based on the LSTM-based decoding model can reach about 60%, better than six traditional algorithms and three deep learning models.
CONCLUSIONS: We prove that LSTM-based model can make better use of the spatiotemporal fMRI signals induced by the complex natural images. At the same time, we compared the confusion matrices of image classification using convolutional neural network (CNN) based on image features and LSTM-based decoding model based on the visual signal. The results show that the confusion matrices obtained by the two methods are highly similar. Finally, the decoding accuracies of different visual cortex were compared, and it was found that the higher visual cortex shows higher accuracy of classification than the lower areas in the visual cortex, suggesting that the higher visual cortex plays an important role in encoding high-level category information from complex natural images.

Key words: fMRI, Visual decoding, LSTM