Puppeteering: Media control data as material for audiovisual creative innovation

Abstract

Millions of creative practitioners use advanced software to manipulate a large diversity of digital materials. Their work forms a significant part of the digital economy, where constant innovation is crucial. But existing tools can impede innovation by limiting digital materials to fixed media, e.g. musical notes, colours, coordinates or velocity. With existing practice, programming is necessary to transcend such restrictions, which limits the agency of most practitioners to shape their technologies. This research introduces the concept of elevating media control signals to be explicitly used as a material that end users directly manipulate in their tools without requiring that they program.

The work is grounded in the observation that tools across contexts and media share design patterns, and that those sharing many define a genre. “Puppeteering” involves two roles: first, a rich user interface application (Puppeteer) implementing a genre’s defining patterns, but no media-specific functionality. The second role is taken by media-specific tools (Puppets), remote-controlled by the puppeteer using standardised control data messages (Strings). The concept empowers craftworkers to manipulate and interrelate digital materials without programming. We demonstrate puppeteering through an implementation for time-based media software, with the purpose of opening up for very rich possibilities of future research.