Acoustic Field Video for Multimodal Scene Understanding

Carnegie Mellon University

Abstract

We introduce and explore a new multimodal input representation for vision–language models: acoustic field video. Unlike conventional video (RGB with stereo/mono audio), our video stream provides a spatially grounded visualization of sound intensity across a scene, offering a new and powerful dimension of perceptual understanding. Our real-time pipeline uses low-cost beamforming microphone arrays - already common in smart speakers and increasingly present in robotics and XR headsets - yet this sensing capability remains unutilized for scene understanding. To assess the value of spatial acoustic information, we constructed an evaluation set of 402 question–answer scenes, comparing a SOTA VLM given conventional video with and without paired acoustic field video. Results show a clear and consistent improvement when incorporating spatial acoustic data; the VLM we test jumps from 38.3% correct to 67.4%. Our findings highlight that many everyday scene understanding tasks remain underconstrained when relying solely on visual and audio input, and that acoustic field data provides a promising and practical direction for multimodal reasoning.

Gemini's improved scene understanding with acoustic field video

Red: Gemini 2.5 Pro's answer with conventional video that contains RGB video and stereo audio

Blue: Gemini 2.5 Pro's answer with our acoustic field video along with conventional video

Prompt: What is happening in the scene?

"The noise is the electronic beeping of a kitchen appliance, such as a microwave or oven."

"A gas stove burner is hissing."

Prompt: What is happening in the scene?

"An espresso machine is brewing a shot of coffee into a small white cup on a wooden countertop."

"I will go to the room behind the sliding glass door on the left to investigate the alarm."

Prompt: You are a mobile robot. When the user asks "please check that", describe what action you would take.

"I would navigate toward the source of the sound to investigate the call for help."

"An electric kettle is boiling water."

Prompt: You are a mobile robot. When the user asks "please stop that", describe what action you would take.

"Based on the audio and visual information, I will approach the table, identify the source of the beeping alarm, and press the appropriate button to silence it."

"I would go to the table and silence the alarm coming from the smartphone."

Prompt: What is happening in the scene?

"A bathroom exhaust fan is running."

"The toilet tank is filling with water."

Prompt: What is happening in the scene?

"An appliance in the kitchen is making a beeping sound."

"An alarm is sounding from the refrigerator's water and ice dispenser."

Result

The answers from VLM with our acoustic field video are not only more accurate but also preferred by human raters.

Citation

@misc{daehwa2026acousticfieldvideomultimodal, title={Acoustic Field Video for Multimodal Scene Understanding}, author={Daehwa Kim and Chris Harrison}, year={2026}, eprint={2601.17123}, archivePrefix={arXiv}, primaryClass={cs.HC}, url={https://arxiv.org/abs/2601.17123}, }