Towards a Systematic Scene Analysis Framework for Audiovisual Data Representations

Abstract

This paper proposes the Audiovisual Scene Graph, a conceptual framework for structuring and analyzing audiovisual representations based on principles of visual and auditory scene analysis. The framework introduces three hierarchical levels, scene, object, and mark, to describe how sensory elements are organized and integrated into coherent perceptual units. It extends traditional scene graphs to support multimodal representations and can be used both to deconstruct existing scenes and to guide the design of new ones. The framework is further expanded to include data mappings through channels and marks, enabling its application in visualization and sonification. By clarifying how visual and auditory elements combine and convey information, the approach aims to improve the analysis, interpretation, and design of audiovisual data representations.