Sketching Sound

Gestural Interaction in Expressive Music Programming

Spencer Salazar

A dissertation submitted to the Department of Music and the Committee on Graduate Studies of Stanford University in partial fulfillment of the requirements for the degree of Doctor of Philosophy.

Originally published May 2017 (permanent URL). Updated for the web and minor revisions November 2023.

Contents

  1. Introduction
  2. Background
  3. Coding and Musical Gesture
  4. Sketching and Musical Gesture
  5. Evaluation
  6. Conclusion

Additional Notes

David Kerr graciously produced a video recording of my oral defense of this doctoral thesis, held on March 9, 2017, hosted at https://www.youtube.com/watch?v=B4xZQxVVnJA. (Just prior to the defense, John Chowning, commenting on my choice of blazer, shirt, and tie, noted I looked ready to “go to a club.”)

Additional information about Auraglyph, an infinite modular music sketchpad discussed in this thesis, can be found at https://spencersalazar.com/auraglyph.

If referencing this dissertation in a research publication, please cite as follows:

@phdthesis{salazar2017sketching,
  title={Sketching Sound: Gestural Interaction in Expressive Music Programming},
  author={Salazar, Spencer},
  year={2017},
  school={Stanford University}
  address={Stanford, CA}
}

Abstract

New developments in technology lead to new types of interactions in computer music performance and composition. In the realm of mobile touchscreen devices such as phones and tablet computers, a variety of research efforts and software applications have explored these possibilities. These include musical experiences that utilize multitouch interaction, the physical properties of the devices themselves, the orientation and location sensing of the devices, and their persistent connection to the network. However, these interactions have been largely ignored in the space of music programming on the device itself.

We have developed two prototype systems to explore concepts related to the employment of these interactions and technologies in programming music on mobile touchscreen devices. The first of these, miniAudicle for iPad, is an environment for programming ChucK code on an iPad. In addition to a text editor and synthesis engine, miniAudicle for iPad incorporates features to ease the often-laborious process of typing on a touchscreen. The application includes additional functionality to greater leverage the intrinsic capabilities of the device, including multitouch interaction and location and orientation sensing.

The second prototype developed is a sound design and music composition system utilizing touch and handwritten stylus input. In this system, called “Auraglyph,’’ users draw a variety of audio synthesis structures, such as oscillators and filters, in an open canvas. Once created, these structures may be further parameterized by touch or in some cases with additional hand-drawn figures. Auraglyph also displays a variety of aspects of its program state to the user, conferring a deeper understanding of how each piece of a program affects the overall result. In addition to a novel interaction framework, standard synthesis functions, and system feedback display, Auraglyph is characterized by a unique visual design intended to draw users into its perspective of music computing.

These systems and the principles they embody have been evaluated through user studies, through the author’s experiences composing with them, and through the compositional experiences of other musicians. These evaluations comprise written and oral user feedback, quantitative analyses, and a number of performances of music utilizing these systems. Results from initial studies were used to re-examine these systems and rework them for later studies that were conducted. Together, these data have illuminated the advantages and drawbacks of the systems designed in this research and the principles underlying them.

Ultimately, we believe this research shows that the critical parameters for developing sophisticated software for new interaction technologies are consideration of the technology’s inherent affordances and mindful attention to design. To this end, we have proposed a set of principles for designing these systems stemming from this research and previous research in this field.