Pay Self-Attention to Audio-Visual Navigation
Yinfeng Yu (Department of Computer Science and Technology, State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China;Xinjiang University),* Lele Cao (EQT Group), Fuchun Sun (Tsinghua University), Xiaohong Liu (Tsinghua University), Liejun Wang (Xinjiang University)
The 33rd British Machine Vision Conference
Abstract
Audio-visual embodied navigation, as a hot research topic, aims training a robot to reach an audio target using egocentric visual (from the sensors mounted on the robot) and audio (emitted from the target) input. The audio-visual information fusion strategy is naturally important to the navigation performance, but the state-of-the-art methods still simply concatenate the visual and audio features, potentially ignoring the direct impact of context. Moreover, the existing approaches requires either phase-wise training or additional aid (e.g. topology graph and sound semantics). Up till this date, the work that deals with the more challenging setup with moving target(s) is still rare. As a result, we propose an end-to-end framework FSAAVN (feature self-attention audio-visual navigation) to learn chasing after a moving audio target using a context-aware audio-visual fusion strategy implemented as a self-attention module. Our thorough experiments validate the superior performance (both quantitatively and qualitatively) of FSAAVN in comparison with the state-of-the-arts, and also provide unique insights about the choice of visual modalities, visual/audio encoder backbones and fusion patterns.Video
Citation
@inproceedings{Yu_2022_BMVC,author = {Yinfeng Yu andXinjiang University and Lele Cao and Fuchun Sun and Xiaohong Liu and Liejun Wang},title = {Pay Self-Attention to Audio-Visual Navigation},booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},publisher = {{BMVA} Press},year = {2022},url = {https://bmvc2022.mpi-inf.mpg.de/0046.pdf}}Copyright © 2022The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised byThe British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).