Acoustic Field Video for Multimodal Scene Understanding

Carnegie Mellon University

Abstract

Advanced vision-language models (VLMs) processing real-time video streams hold tremendous promise to unlock responsive and intelligent AI assistants. However, the video streams (RGB with stereo or mono audio) on which they rely are insufficient to interpret a bevy of commonplace states, contexts, and questions about everyday environments. In this work, we introduce and explore a new multimodal sensor input for AI agents: acoustic field video. Unlike conventional video, our video stream provides a spatially grounded visualization of sound intensity across a scene, offering a new and powerful dimension of perceptual understanding for AIs to understand a user's environment and context. Our real-time pipeline uses low-cost beamforming microphone arrays, already common in smart speakers and increasingly present in robots and XR headsets, yet this sensing capability remains unutilized for AI scene understanding. To assess the value of spatial-acoustic information for agentic understanding, we constructed an evaluation test set of 402 question–answer scenes, and compare three state-of-the-art VLMs given conventional video with and without paired acoustic field video. Our results show a consistent improvement when spatial-acoustic data is provided; the AI models we test jump from 31.2% correct to 54.8%. Our findings highlight that many everyday scene understanding tasks remain underconstrained when relying solely on video and audio, and that acoustic field data provides a promising and practical direction for multimodal reasoning.

VLM's improved scene understanding with acoustic field video

Red: Gemini 2.5 Pro's answer with conventional video that contains RGB video and stereo audio

Blue: Gemini 2.5 Pro's answer with our acoustic field video along with conventional video

Prompt: What is happening in the scene?

"The noise is the electronic beeping of a kitchen appliance, such as a microwave or oven."

"A gas stove burner is hissing."

Prompt: What is happening in the scene?

"An espresso machine is brewing a shot of coffee into a small white cup on a wooden countertop."

"An electric kettle is boiling water."

Prompt: You are a mobile robot. When the user asks "please check that", describe what action you would take.

"I would navigate toward the source of the sound to investigate the call for help."

"I will go to the room behind the sliding glass door on the left to investigate the alarm."

Prompt: You are a mobile robot. When the user asks "please stop that", describe what action you would take.

"Based on the audio and visual information, I will approach the table, identify the source of the beeping alarm, and press the appropriate button to silence it."

"I would go to the table and silence the alarm coming from the smartphone."

Prompt: What is happening in the scene?

"A bathroom exhaust fan is running."

"The toilet tank is filling with water."

Prompt: What is happening in the scene?

"An appliance in the kitchen is making a beeping sound."

"An alarm is sounding from the refrigerator's water and ice dispenser."

Result

The answers from VLM with our acoustic field video are not only more accurate but also preferred by human raters.

Citation

@misc{daehwa2026acousticfieldvideomultimodal, title={Acoustic Field Video for Multimodal Scene Understanding}, author={Daehwa Kim and Chris Harrison}, year={2026}, eprint={2601.17123}, archivePrefix={arXiv}, primaryClass={cs.HC}, url={https://arxiv.org/abs/2601.17123}, }