This annotated version by Adrian Freed borrows some ideas from medieval glosses. It is intented to provide context and readability unavailable in the original pdf.
This annotated version by Adrian Freed borrows some ideas from medieval glosses. It is intented to provide context and readability unavailable in the original pdf.
Matthew Wright
Adrian Freed
Ahm Lee
Tim Madden
Ali Momeni
email: {matt,adrian,ahm,tjmadden,ali}@cnmat.berkeley.edu
UC Berkeley
1750 Arch St.
Berkeley, CA 94709, USA
A better integration of computers, controllers and sound synthesizers will lead to lower costs, increased reliability, greater user convenience, and more reactive musical control. The prevailing technologies to interconnect these elements are bus (motherboard or PCI), operating system interface (software synthesis), or serial LAN (Firewire, USB, ethernet, fast ethernet). It is easy to adapt MIDI streams to this new communication substrate [11]. To do so needlessly discards new potential and perpetuates MIDI's well-documented flaws. Instead we have designed a new protocol optimized for modern transport technologies.
Open SoundControl is an open, efficient, transport-independent, message-based protocol developed for communication among computers, sound synthesizers, and other multimedia devices. Open SoundControl is a machine and operating system neutral protocol and readily implementable on constrained, embedded systems.
We begin by examining various networking technologies suitable for carrying Open SoundControl data and discuss shared features of these technologies that impact the design of our protocol. Next we discuss the encoding and formatting of data. We describe our novel addressing scheme based on a URL-style symbolic syntax. In our last section we specify a number of query messages that request information from an Open SoundControl system.
Open SoundControl is a transport-independent protocol, meaning that it is a format for data that can be carried across a variety of networking technologies. Networking technologies currently becoming widely available and economical include high speed busses such as PCI [8] and medium speed serial LANs such as USB [10], IEEE-1394 ("Firewire") [3], Ethernet, and Fast Ethernet [4]. Although Open SoundControl was not designed with a particular transport layer in mind, our design reflects features shared by modern networking technologies.
We assume that Open SoundControl will be transmitted on a system with a bandwidth in the 10+ megabit/sec range. MIDI's bandwidth, by contrast, is only 31.25 kilobit/sec, roughly 300 times slower. Therefore, our design is not preoccupied with squeezing musical information into the minimum number of bytes. We encode numeric data in 32-bit or 64-bit quantities, provide symbolic addressing, time-tag messages, and in general are much more liberal about using bandwidth for important features.
We assume that data will be delivered in packets (a.k.a. "datagrams") rather than as a stream of data traveling along an established connection. (Although many LANs transmit data serially, they deliver the data as a single block.) This leads to a protocol that is as stateless as possible: rather than assuming that the receiver holds some state from previous communications (e.g., MIDI's running status), we send information in larger, self-contained chunks that include all the relevant data in one place. This packet-based delivery model provides a mechanism for synchronicity: messages in the same packet (e.g., messages to start each of the notes in a chord) can be specified to have their effects occur at the same time as each other. We assume that the network services will tell us the length of each packet that we receive.
All modern networking technologies have the notion of multiple devices connected together in a LAN, with each device able to send a packet to any other device. So we assume that any number of clients might send Open SoundControl messages to a particular device. We also assume that the transport layer provides a return address mechanism that allows a device to send a response back to the device that sends it a message.
All Open SoundControl data is aligned on 4-byte boundaries. Numeric data are encoded using native machine representations of 32-bit or 64-bit big-endian twos-complement integers and IEEE floating point numbers. Strings are represented as a sequence of non-null ASCII characters followed by a null, padded with enough extra null characters to make the total length be a multiple of 4 bytes. These representations facilitate real-time performance by eliminating the need to reformat received data. (Except for the unavoidable big-endian/small-endian conversion. We note that most small-endian machines provide special instructions for this conversion.)
The basic unit of Open SoundControl data is a message, which consists of the following:
An Open SoundControl packet can contain either a single message or a bundle. A bundle consists of the following:
Note that bundles are recursively defined; bundles can contain other bundles.
Messages in the same bundle are atomic; their effects should be implemented concurrently by the receiver. This provides the important service in multimedia applications of specifying values that have to be set simultaneously.
Time tags allow receiving synthesizers to eliminate jitter introduced during packet transport by resynchronizing with bounded latency. They specify the desired time at which the messages in the bundle should take effect, a powerful mechanism for precise specification of rhythm and scheduling of events in the future. To implement time tags properly, message recipients need a real-time scheduler like those described by Dannenberg [1].
Time tags are represented by a 64 bit fixed point number. The first 32 bits specify the number of seconds since midnight on January 1, 1900, and the last 32 bits specify fractional parts of a second to a precision of about 200 picoseconds. This is the representation used by Internet NTP timestamps [6]. The Open SoundControl protocol does not provide a mechanism for clock synchronization; we assume either that the underlying network will provide a synchronization service or that the two systems will take advantage of a protocol such as NTP or SNTP [7].
The model for Open SoundControl message addressing is a hierarchical set of dynamic objects that include, for example, synthesis voices, output channels, filters, and a memory manager. Messages are addressed to a feature of a particular object or set of objects through a hierarchical namespace similar to URL notation, e.g., /voices/drone-b/resonators/3/set-Q. This protocol does not proscribe anything about the objects that should be in this hierarchy or how they should be organized; each system that can be controlled by Open SoundControl will define its own address hierarchy. This open-ended mechanism avoids the addressing limitations inherent in protocols such as MIDI and ZIPI [5] that rely on short fixed length bit fields.
To allow efficient addressing of multiple destination objects for parameter updates, we define a pattern-matching syntax similar to regular expressions. When a message's address is a pattern, the receiving device expands the pattern into a list of all the addresses in the current hierarchy that match the pattern, similar to the way UNIX shell "globbing" interprets special characters in filenames. Each address in the list then receives a message with the given arguments.
We reserve the following special characters for pattern matching and forbid their use in object addresses: ?, *, [, ], {, and }. The # character is also forbidden in object address names to allow the bundle mechanism. Here are the rules for pattern matching:
Our experience is that with modern transport technologies and careful programming, this addressing scheme incurs no significant performance penalty either in network bandwidth utilization or in message processing.
Some Open SoundControl messages are requests for information; the receiving device (which we'll call the Responder in this section) constructs a response to the request and sends it back to the requesting device (which we'll call the Questioner). We assume that the underlying transport layer will provide a mechanism for sending a reply to the sender of a message.
Return messages are formatted according to this same Open SoundControl protocol. There is always enough information in a return message to unambiguously identify what question is being answered; this allows Questioners to send multiple queries without waiting for a response to each before sending the next. The time tag in a return message indicates the time that the Responder processed the message, i.e., the time at which the information in the response was true.
Any message address that ends with a trailing slash is a query asking for the list of addresses underneath the given node. These messages do not take arguments. This kind of query allows the user of an Open SoundControl system to map out the entire address space of possible messages as the system is running.
Because Open SoundControl messages do not have any explicit type tags in their arguments, it is necessary for the sender of a message to know how to lay out the argument values so that they will be interpreted correctly.
An address that ends with the reserved name "/type-signature" is a query for the "type signature" of a message, i.e., the list of types of the arguments it expects. We have a simple syntax for representing type signatures as strings; see our WWW site for details.
An address that ends with the reserved name "/documentation" is a request for human-readable documentation about the object or feature specified by the prefix of the address. The return message will have the same address as the request, and a single string argument. The argument will either be the URL of a WWW page documenting the given object or feature, or a human-readable string giving the documentation directly. This allows for "hot-plugging" extensible synthesis resources and internetworked multimedia applications.
An address that ends with the reserved name "/current-value" is a query of the current value of the parameter specified by the prefix of the address. Presumably this parameter could be set to its current value by a message with the appropriate arguments; these arguments are returned as the arguments to the message from the Responder. The address of the return message should be the same as the address of the request.
We have had successful early results transmitting this protocol over UDP and Ethernet to control real-time sound synthesis on SGI workstations from MAX [9] programs running on Macintosh computers. Composers appreciate being freed from the limited addressing model and limited numeric precision of MIDI, and find it much easier to work with symbolic names of objects than to remember an arbitrary mapping involving channel numbers, program change numbers, and controller numbers. This protocol affords satisfying reactive real-time performance.
[1] Dannenberg, R, 1989. "Real-Time Scheduling and Computer Accompaniment," in M. Mathews and J. Pierce, Editors, Current Directions in Computer Music Research, Cambridge, Massachusetts: MIT Press, pp. 225-261.
[2] Freed, A. 1996. "Firewires, LANs and Buses," from SIGGRAPH Course Notes Creating and Manipulating Sound to Enhance Computer Graphics, New Orleans, Louisiana, pp. 84-87.
[3] IEEE 1995. "1394-1995: IEEE Standard for a High Performance Serial Bus," New York: The Institute of Electronical and Electronic Engineers.
[4] IEEE 1995b. "802.3u-1995 IEEE Standards for Local and Metropolitan Area Networks: Supplement to Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications: Media Access Control (MAC) Parameters, Physical Layer, Medium Attachment Units, and Repeater for 100 Mb/s Operation," New York: The Institute of Electronical and Electronic Engineers.
[5] McMillen, K, D. Wessel, and M. Wright, 1994. "The ZIPI Music Parameter Description Language," Computer Music Journal, Volume 18, Number 4, pp. 52-73.
[6] Mills, D., 1992. "Network Time Protocol (Version 3) Specification, Implementation, and Analysis," Internet RFC 1305. (http://sunsite.auc.dk/RFC/rfc1305.html)
[7] Mills, D., 1996. "Simple Network Time Protocol (SNTP) Version 4 for Ipv4, Ipv6 and OSI," Internet RFC 2030. (http://sunsite.auc.dk/RFC/rfc2030.html)
[8] PCI 1993. PCI Local Bus Specification, Revision 2.0, Hillsboro, Oregon: PCI Special Interest Group.
[9] Puckette, M., 1991. "Combining Event and Signal Processing in the MAX Graphical Programming Environment," Computer Music Journal, Volume 15, Number 3, pp. 58-67.
[10] The USB web site is www.usb.org
[11] Yamaha 1996. "Preliminary Proposal for Audio and Music Protocol, Draft Version 0.32," Tokyo, Japan: Yamaha Corporation.
Throughout history, people have adapted objects in their environment into musical instruments. The computer industry has developed the digitizing tablet (a.k.a. "artist's tablet") primarily for the purpose of drawing shapes in computer graphics illustration programs. These tablets are broadly available and low cost, and they sense many control dimensions such as pen or pointing device position, pressure, and tilt. We find that these controllers can be used for musical control in a variety of interesting and musically expressive ways.
We use a Wacom [9] ArtZ II 1212 digitizing tablet, a 12 inch by 12 inch model with both a stylus (pen) and a puck (mouse-like device) that allows two-handed input [7]. The stylus and puck are cordless, batteryless, and light weight. The tablet has a single DB-9 connector that carries both power and control information; on the other side of the cable is a mini-din connector suitable for the serial port on a Macintosh, PC, or SGI workstation. We've had no trouble running the cable at distances of about 30 feet.
The Wacom tablet accurately outputs the absolute X and Y dimensions of both the puck and stylus as integers in the range 0 to 32480. The stylus has a pressure-sensitive tip (and a pressure-sensitive "eraser" tip on the other end) that produces pressure readings in the range 0 to 255. When only the stylus is used, the Wacom tablet also outputs tilt values in two dimensions, in the range -60 to 60 degrees. Position and tilt are reported whenever the stylus or puck is in "proximity" of the tablet, within about a centimeter of the surface.
In addition to these continuous control variables, there are many ways to trigger events. The puck has 4 buttons, and the stylus has two buttons as well. Each tip of the stylus can be considered a button; pressing the stylus into the tablet will cause the same kind of button event. Another kind of event is generated when either device enters or leaves the proximity of the tablet. For the stylus, this event indicates which side (stylus tip side or eraser side) enters or leaves proximity.
It is important to evaluate the temporal behavior of computer systems for music [6]. We performed timing experiments on SGI machines using their fast UST clock, measuring the elapsed time between events reporting values of position, tilt, and pressure while continually moving the stylus so as to change each parameter constantly. The good news is that about 75% of parameter updates came within 1 ms of the previous update. The bad news is that the elapsed times greater than 1 ms averaged about 28 ms and were distributed more or less in a bell curve around 28 ms. The source of this erratic temporal performance seems to be context switches between the X server and our application program, a consequence of Wacom's decision to make tablet data available on Unix using X11 extension device valuator events. We expect that direct access to the serial stream from the tablet will address this difficulty.
We have developed a model for mapping tablet data into musical control information (either MIDI or Open SoundControl [11]) that allows performers to simply customize the musical behavior of the tablet. In our model, the two dimensional surface of the tablet is populated by any number of arbitrarily-shaped polygonal regions. These regions can overlap, and they have a vertical stacking order that determines which of two overlapping regions is above the other. Each region has a user-assigned symbolic name.
The following events may occur in a region:
· Puck or stylus enters (or leaves) the proximity of the tablet in the region
· Either end of the stylus makes (or breaks) contact with the tablet
· Puck or stylus moves into (or out of) the region
· Button press (or release) while pen or stylus is in the region
For each region the user can define a list of actions to take when any of these events occur, e.g., the stylus tip touching the tablet in a region might cause a pair of MIDI note-on events.
There are also several continuous parameters that are updated constantly by the tablet:
· X and Y coordinates for puck and stylus
· X and Y axis tilt for stylus (when not using puck)
· Stylus pressure
Each of these continuous parameters can also be mapped to a list of actions, e.g., the X axis tilt might correspond to pitch bend.
What makes our model dynamic is that actions can be added to and deleted from these lists in response to events that occur in a region. For example, pressing a button while the puck is in a region could add a mapping from puck Y position to overall volume, and having the puck enter a different region could remove this mapping. This features facilitates complex musical behaviors in response to various gestures.
To allow the mapping of data values from one range to another we provide two primitives. The first is a linear scaling operator that implements functions of the form f(x)=ax+b where a and b are constant. The desired output range is specified; the system knows the range of possible input values and can compute a and b automatically. The second is stored function evaluation. We compute user-specified memoryless nonlinear functions with an abstraction that subsumes table lookup, expression evaluation, neural network forward-pass, and other techniques. Of course the tablet itself is an excellent tool for drawing function curves.
One important feature of the tablet is that its position sensing is absolute rather than relative. This means that the performer does not need any visual feedback from a computer monitor to see which region the stylus or puck is in. The Wacom tablet has a clear plastic overlay under which paper and small objects may be placed. We can print a "map" of the user's defined regions to the scale of the tablet and place it under the overlay; this lets the user see exactly where each region is on the tablet. Removable adhesive strips for drafting may be used to create surface irregularities offereing tactile feedback.
Our first use of the digitizing tablet for musical control was to create an interface for a drone instrument similar in function to the Indian tambura. A real tambura has 4 strings with no fretboard or fingerboard; the player plucks each open string and lets it ring. Tambura players typically pluck the 4 strings one at a time and then start over at the first string, providing a continuous drone.
A tablet interface was designed that retains features of the playing style of a real tambura. We defined six regions on the tablet corresponding to six virtual strings. Each of these virtual strings had a corresponding monophonic synthesizer voice set to a particular drone timbre and pitch. The action of touching the stylus to the tablet in a particular region represented touching a finger to a real string: it caused the currently sounding note on that string to decay fairly rapidly. Releasing the stylus from the tablet represented the second half of the pluck, where the finger releases the string and sets it vibrating. This caused the voice to start a new note.
The loudness of each note was determined by the total amount of horizontal distance traveled between the time the stylus touched the region and the time the stylus left the tablet. Timbral control came from a mapping from the X axis position at the time the stylus left the tablet to the relative balance of even and odd harmonics in the synthesizer [5].
We use the tablet to emulate the gesture of strumming a stringed instrument. We define four to ten thin rectangular horizontal regions which represent virtual strings much as in the tambura example. Again, each of these regions has a corresponding synthesizer voice which is virtually plucked in two steps by the stylus entering and then leaving the region. The speed of the pluck (i.e., the reciprocal of the time between when the stylus enters and leaves the region) combines with the pen pressure to determine the loudness of each note.
When the upper stylus button is depressed the stylus' function changes from exciting the virtual strings to muting them. In this case, the note for each string still decays when the tablet enters the region, but a new note does not start when the string leaves the region. The middle button corresponds to a "half mute" mode where the synthesized sound is more inharmonic and each note decays faster, corresponding to the playing technique on real strummed instruments of partially muting strings with the palm.
On real stringed instruments, the position of the picking (or plucking or bowing) along the axis of the string is an important determinant of timbre. We emulate this control by mapping the X position at the time the stylus leaves the string region to a timbral control such as brightness.
We have several techniques for controlling pitch with the puck in the left hand. The first is modeled loosely on the autoharp. There are regions of the tablet that correspond to different chord roots, and each of the 4 puck buttons corresponds to a different chord quality (e.g., major, minor, dominant seventh, and diminished). Pressing a button with the puck in a particular region determines a chord, which in turn determines the pitches of each virtual string. The layout of chord regions and chord qualities, and the voicings of each chord are all easily configurable for different styles, pieces, or performers.
Another technique for pitch control is modeled on the slide guitar, with the puck taking the role of the slide. In this case the pitches of the virtual strings are in fixed intervals relative to each other, and the puck's horizontal position determines a continuous pitch offset that affects all the strings. When the puck is not in proximity of the tablet the strings are at their lowest pitches, corresponding to strumming open strings on a slide guitar. The buttons on the puck select from among a set of different tunings.
CNMAT's additive synthesis system provides timbral interpolation via the timbre space model [10]. The tablet is a natural way to control position in a two-dimensional timbre space.
The simplest application uses the tablet in conjunction with another controller like a MIDI keyboard. In this case the job of the tablet is just to determine position in the timbre space, so we map the two position dimensions to the two timbre space dimensions and use the other controller to determine pitch, loudness, and articulation.
Another approach is to use both hands on the tablet. The stylus position determines timbre space position. When the stylus touches the tablet a note is articulated, and when it leaves the tablet the note ends. Stylus pressure maps to volume. The puck, held in the left hand, determines pitch via a variety of possible mappings.
A third approach is to implement a process that continually produces a stream of notes according to parameterized models of rhythm, harmony, and melody. The left hand can manipulate the parameters of this model with the puck while the right hand navigates in timbre space with the stylus.
We would like to be able to recognize various kinds of strokes and gestures made with the stylus. Previous work on stroke recognition in the context of computer conducting [2] seems applicable, as does work on the integration of segmentation, recognition, and quantitative evaluation of expressive cursive gestures [8].
The tablet interface provides a musically potent and general two-dimensional interface for the control of sound synthesis and compositional algorithms. With the addition of stylus pressure and tilt, two additional dimensions are available for each hand. Our experiments have demonstrated that irregularities can be added to the surface providing tactile reference. The tablet interface is basically a spatial coordinate sensor system like the Mathews Radio Drum [1], the Buchla Lightning [3], the Theremin [4], joy sticks, and other sensor systems like ultra sound ranging. But, unlike some of these systems it offers the possibility of tactile reference, wide availability, flexible adaptation, precision, and reliability. The tablet interface and its spatial coordinate sensor cousins offer the possibility of long-lived alternative musical control structures that use mathematical abstractions in a reliable manner as the basis for a live-performance musical repertoire.
CNMAT gratefully acknowledges the support of Silicon Graphics, Inc., and Gibson Guitar Corporation for generous contributions that supported this work.
[1] Boie, B, M. Mathews, and A. Schloss. 1989. "The Radio Drum as a Synthesizer Controller," Proc. ICMC, Columbus, Ohio, pp. 42-45.
[2] Brecht, B., and G. Garnett. 1995. "Conductor Follower," Proc ICMC, Banff, Canada, pp. 185-186.
[3] The Buchla and Associates web site describes Lightning II: http://www.buchla.com/
[4] Chadabe, J. 1997. Electric Sound: The Past and Promise of Electronic Music, Englewood Cliffs, New Jersey: Prentice Hall.
[5] Freed, A. 1995. "Bring Your Own Control to Additive Synthesis," Proc ICMC, Banff, Canada, pp 303-306.
[6] Freed, A. 1997. "Operating Systems Latency Measurement and Analysis for Sound Synthesis and Processing Applications," Proc. ICMC, Thessaloniki.
[7] Kabbash, P, W. Buxton, and A. Sellen. 1995. "Two-handed Input in a Compound Task," Proc. SigCHI, Boston, Massachusetts, pp. 417-423. http://www.dgp.utoronto.ca/OTP/papers/bill.buxton/tg.html
[8] Keeler, J., D. Rumelhart, and W. Loew. 1991. "Integrated segmentation and recognition of hand-printed numerals," In R. Lippmann, J. Moody, and D. Touretzky, Editors, Neural Information Processing Systems Volume 3, San Mateo, CA: Morgan Kaufmann.
[9] The Wacom WWW site is www.wacom.com
[10] Wessel, D. 1979. "Timbre Space as a Musical Control Structure," Computer Music Journal, Volume 3, Number 2, pp 45-52.
[11] Wright, M., and A. Freed. 1997. "Open SoundControl Protocol," Proc ICMC, Thessaloniki. New Musical Control Structures from Standard Gestural Controllers, , International Computer Music Conference, 1997, Thessaloniki, Hellas, p.387-390, (1997)