ISSN 1671-3710
CN 11-4766/R

心理科学进展 ›› 2022, Vol. 30 ›› Issue (3): 580-590.doi: 10.3724/SP.J.1042.2022.00580

• 研究前沿 • 上一篇    下一篇


万必成, 杨振, 李宏汀, 马舒()   

  1. 浙江理工大学心理学系, 杭州 310018
  • 收稿日期:2021-04-10 出版日期:2022-03-15 发布日期:2022-01-25
  • 通讯作者: 马舒
  • 基金资助:

The vivid tactile experience from vision and auditory: Clues from multisensory channel integration

WAN Bicheng, YANG Zheng, LI Hongting, MA Shu()   

  1. Psychology Department, Zhejiang Sci-Tech University, Hangzhou 310018, China
  • Received:2021-04-10 Online:2022-03-15 Published:2022-01-25
  • Contact: MA Shu


虚拟现实技术通过提供视觉、听觉和触觉等信息为用户创造身临其境的感知体验, 其中触觉反馈面临诸多技术瓶颈使得虚拟现实中的自然交互受限。基于多感官错觉的伪触觉技术可以借助其他通道的信息强化和丰富触觉感受, 是目前虚拟现实环境中优化触觉体验的有效途径。本文聚焦于触觉中最重要的维度之一——粗糙度, 试图为解决虚拟现实中触觉反馈的受限问题提供新思路。探讨了粗糙度感知中, 视、听、触多感觉通道整合的关系, 分析了视觉线索(表面纹理密度、表面光影、控制显示比)和听觉线索(音调/频率、响度)如何影响触觉粗糙度感知, 总结了当下调控这些因素来改变粗糙度感知的方法。最后, 探讨了使用伪触觉反馈技术时, 虚拟现实环境中视、听、触觉信息在呈现效果、感知整合等方面与真实世界相比可能存在的差异, 提出可借鉴的改善触觉体验的适用方法和未来待研究的方向。

关键词: 粗糙度知觉, 多感觉通道整合, 视觉线索, 听觉线索


Virtual reality creates an immersive experience for users by providing visual, auditory, and tactile information. However, tactile feedback still faces many technical bottlenecks, which limit natural interaction in the virtual reality environment. Pseudo-haptic technology based on multi-sensory illusion can enhance and enrich tactile perception with the help of information from other channels, which is one of the effective ways to optimize tactile perception in the virtual reality environment. In order to explore the problem in a more targeted way, the study focuses on roughness perception among different dimensions of tactile perception. We first discuss the relationship between visual, auditory, and tactile perception in roughness information integration and further analyze visual and auditory factors affecting roughness perception.

In the process of roughness perception, the channel with less noise and more reliable information will have a higher weight. By evaluating the roughness under every single channel (visual、auditory and tactile) condition,and each double-channel(visual-tactile and auditory-tactile) condition, previous studies have quantified the high weight of visual and auditory channels in roughness perception. It further verifies that roughness perception results from integrating tactile, visual, and auditory information. Therefore, it has excellent potential to manipulate visual and auditory factors to induce or enhance roughness perception in the virtual reality environment.

For visual factors, the texture of the surface, light or shadow, and the control display rate (CDR) may affect the perception of roughness. ①When the texture density of the object surface increase, the perceived roughness will increase accordingly; ②Since the angle of incidence influence the shadow of the surface texture, thus affecting the perception of roughness; ③ When the glossiness of surface decrease, the perceived roughness increase; ④Reducing the CDR of the virtual hand or presenting a vibratory virtual hand will increase the user's perception of the roughness.

For auditory factors, manipulating loudness and pitch/frequency can change the roughness perception: increasing loudness in the entire frequency of sound or increasing loudness in a high, medium, or low frequency band of sound can improve the roughness perception. There is a strong correlation between pitch/frequency and roughness perception. According to current research, we speculated that there might be an inverted U-shaped curve between pitch/frequency change and roughness perception.

In the future, in order to optimize the tactile roughness experience in virtual reality environment, studies can be carried out from the following aspects: Firstly, it is necessary to explore the differences between every single channel (visual, auditory, and tactile) information in the virtual reality environment and real environment respectively, since single channel information is the prerequisite for multi-sensory channel information in the virtual reality environment. Secondly, from the practical point of view, we can choose the channel more suitable to the current scene or task and provide sufficient information timely with emphasis. Thirdly, the difference of single-channel information in virtual reality will lead to information mismatch between different channels. Therefore, it is necessary to explore further how the brain integrates the originally different and mismatched multi-channel sensory information. Finally, most studies explored the usability of relevant sensory information on roughness perception from the perspective of single-channels or double-channels. Therefore, in the future, it is of significant importance to explore the information integration and processing rules of three channels in virtual reality.

Key words: roughness perception, multi-sensory channel integration, visual cues, auditory cues