Audio-Object Transcoding in Unity for 3D Music and XR

Abstract

Spatial audio is central to 3D music production, extended reality (XR) and immersive media, yet its authoring still largely takes place through conventional two-dimensional interfaces, creating a mismatch between how sound is produced and how it is experienced. This paper presents an experimental system that connects a native C++ audio plug-in, built with the JUCE framework, to a Unity-based three-dimensional environment via Unity's native plug-in interface. The core idea, audio-object transcoding, links the spatial parameters of an audio object to a corresponding three-dimensional visual representation, so that manipulating the visual object changes the sound and vice versa. The work targets two production contexts: 3D music production, for example Dolby Atmos, where spatial mixes are authored on a two-dimensional screen but heard in three dimensions, and XR sound design, where spatial audio tools can be technically complex. The prototype is an early technical exploration rather than a completed user study, and establishes a basis for future work using head-mounted displays and gesture-based interaction.