Understanding Modal Synergy for Exploration

Abstract

This paper investigates “modal synergy” through Photone, an interactive installation that sonifies image data to support exploratory audiovisual experiences. Rather than representing semantic image content, Photone maps syntactic properties such as color, brightness, and spatial features to sound, allowing users to “listen” to images via cursor interaction. Analysis of interaction logs from 233 users shows that those with longer engagement explore less visually salient regions, suggesting a shift toward discovering audiovisual relationships rather than recognizable objects. The authors interpret this as evidence of modal synergy, where visual, auditory, and interaction modalities merge into a unified experiential whole. To better understand this phenomenon, the paper proposes a qualitative study focusing on “connoisseurial exploration,” targeting experienced participants capable of articulating nuanced experiences, with the goal of developing a richer conceptual and descriptive framework for modal synergy.