Abstract: PURPOSE: A raw fMRI-based end-to-end deep learning model, MRGazerII, was proposed to recognize eye-movement state at the temporal interval of tens of millisecond.
METHODS: The movie-watching fMRI data we used were from the Human Connectome Project (HCP) 7T release, in which each subject watched four movies with their eye movements recorded. The binary morphology method was then used to segment the eye domains and the intermediate 6-layer slices of the eye balls were extracted and fed into the deep neural network for the eye movement prediction. A series of ResNet-CBAM, Transformer encoder and fully connected layer were assembled to give slice-level prediction. The dataset was categorized into training and validating group at cross-subject level.
RESULTS: The proposed model achieves an accuracy of 0.48 in the classification of eye movements with f1-scores of 0.56, 0.51 and 0.34 for the fixation, blink and saccade respectively. Correlation analysis show that the prediction of fixation (averaged r = 0.28) and blink (averaged r = 0.46) were highly correlated with the records of the eye tracker.
CONCLUSIONS: The proposed MRGazerII, a camera-free eye tracking method, is able to report typical eye movements at temporal interval of tens of milliseconds and would be helpful to future fMRI & eye-movement analysis.

Key words: fMRI, deep learning, eye movement