|Full Text|| |
This note is a plea to the computer music community to aim higher
in the development of new graphical tools and better graphical user interfaces
(GUIs) for computer music applications.
It has been 10 years now since user interfaces such as that of Digidesign
Inc's Sound Designer and my own MacMix were designed. It is distressing to
see so how many recently built programs are simple permutations of the same
graphical elements. The interactive techniques used in MacMix (direct
manipulation with mouse of 2-D polygonal objects in overlapping rectangular
windows) were obsolete when I used them (by at least a decade). My defense
is that computing performance of the day did not afford more exciting techniques.
Now the situation is completely turned around. RISC computing performance
exceeds the requirements of the old bitmap 2-D graphics paradigm by a huge
First we should dismantle roadblocks hindering more interesting use of
graphical user interfaces: portability and compatibility. These requirements
have lead to very conservative choices. Of course, as researchers, we have
no excuse for favoring these requirements over innovation. Fortunately, the
personal computer and workstation industry has just successfully rallyed
around a strong, reliable and portable standard for 3D graphics: OpenGL
Features of OpenGL that are relevant to the computer music developer include:
- client/server architecture;
- strong integration with X and other windowing systems such as Microsoft
Windows and the Apple Macintosh Toolbox;
- broad range of geometric primitives including points, lines, polygons,
images and bitmaps;
- binding for C and C++; and
- only 120 basic functions in the library
As for device compatability, there is little point supporting small
screens or monochrome screens. The price-leading monitors are now color.
Most new computers have enough video RAM for 16-bit color standard. By the
time new software matures, computers will have 24-bit, million pixel graphics
support as standard features.
Here is a list of suggestions for future graphical user interfaces
for computer music.
- Thoughtful use of color - not just "pastel desktop and menus" or
color spectragrams. (Wolff and Yaeger 1993)--Tech
Note 3, "Visualization of floating-point data" pp. 237 illustrates some of
the pitfalls here. Color is a virtually unexplored and potentially powerful
element in "music visualization."
- 3-D object representations--with careful use of perspective, fog, depth
cuing and transparency--aid in the manipulation and processing of large numbers
of objects, see e.g., (Robertson et al 1991, 1993),
(Card et al. 1991), (Mackinlay et al. 1991), and (Bier
et al 1993). The multi-dimensionality of musical data begs for higher
dimensions of control and representation impossible with the current "paper
on a desktop" metaphor.
- Abandonment of the "data in files" model in favor of the mature database
and multidimensional search techniques used in geometric modelling, as described
in (Nielson et al. 1994) and
(Matousek 1994). Many of the interesting questions
we are asking are about properties of very large collections of different
objects, e.g. databases of analyzed sounds of musical instrument over many
pitches, loudness, and playing styles. We need to move beyond limited,
file-based, single-data-type applications towards models that support richer
data types, visualization paradigms and distributed storage such as the model
behind the rapidly evolving world-wide web.
- Support for input devices with higher control bandwidth and dimensionality
than the mouse, as in
1993). We need to integrate new kinds of keyboards, a broader range of
physical gestures, vocalization, and even non-human control sources.
- Integration of real-time sound synthesis throughout the graphical interface,
e.g. cursors on frequency axis that synthesize a continous reference tone
as they are moved (Gaver et al.1990-1993).
3-D Interface Paradigms
These references describe some of the recent work (at Xerox PARC
and elsewhere) on advanced software interfaces for the presentation and
manipulation of complex and very large information structures.
Robertson, G.G.; Card, S.K.; Mackinlay, J.D.
Information Visualization using 3-D Interactive Animation.
Communications of the ACM, April 1993, vol.36, (no.4):56-7.
Mackinlay, J.D.; Robertson, G.G.; Card, S.K.
The Perspective Wall: Detail and Context Smoothly Integrated.
IN: Human Factors in Computing Systems. Reaching Through Technology. CHI
'91. Conference Proceedings. (Human Factors in Computing Systems. Reaching
Through Technology. CHI '91. Conference Proceedings, New Orleans, LA, USA,
27 April-2 May 1991). Edited by: Robertson, S.P.; Olson, G.M.; Olson, J.S.
New York, NY, USA: ACM, 1991. p. 173-9.
Card, S.K.; Robertson, G.G.; Mackinlay, J.D. The Information
Visualizer, an Information Workspace. IN: Human Factors in Computing
Systems. Reaching Through Technology. CHI '91. Conference Proceedings. (Human
Factors in Computing Systems. Reaching Through Technology. CHI '91. Conference
Proceedings, New Orleans, LA, USA, 27 April-2 May 1991). Edited by: Robertson,
S.P.; Olson, G.M.; Olson, J.S. New York, NY, USA: ACM, 1991. p. 181-8.
Robertson, G.G.; Mackinlay, J.D.; Card, S.K.
Cone Trees: Animated 3-D Visualizations of Hierarchical
Information. IN: Human Factors in Computing Systems. Reaching Through
Technology. CHI '91. Conference Proceedings. (Human Factors in Computing
Systems. Reaching Through Technology. CHI '91. Conference Proceedings, New
Orleans, LA, USA, 27 April-2 May 1991). Edited by: Robertson, S.P.; Olson,
G.M.; Olson, J.S. New York, NY, USA: ACM, 1991. p. 189-94.
Bier, E.A.; Stone, M.C.; Pier, K.; Buxton, W.; and
others. Toolglass and Magic Lenses: The See-through Interface.
IN: Computer Graphics Proceedings. (Computer Graphics ProceedingsProceeding
of SIGGRAPH 20th Annual International Conference on Computer Graphics and
Interactive Techniques. The Eye of Technology, Anaheim, CA, USA, 1-6 Aug.
1993). New York, NY, USA: ACM, 1993. p. 73-80.
William Gaver has developed several software user interfaces to integrate
non-speach audio into the Macintosh Finder and the Xerox PARC Alternate Reality
Gaver, W. and Smith, R. (1990). Auditory Icons
in Large-scale Collaborative Environments. In D. Diaper et al. (Eds),
Human-Computer Interaction - INTERACT '90, Elsevier Science Publishers B.V.
Gaver, W. (1993). Synthesizing Auditory Icons, Proceedings
of INTERCHI'93, 228-235.
Gaver, W. and Smith, R. (1990). Auditory Icons in Large-scale
Collaborative Environments. In D. Diaper et al. (Eds), Human-Computer
Interaction - INTERACT '90, Elsevier Science Publishers B.V. (North-Holland),
Gaver, W., Smith, R. and O'Shea, T. (1991). Effective Sounds in
Complex Systems: The ARKola Simulation, Proceedings of CHI'91, 85-90.
The OpenGL graphics interface library is stable and well-documented
in the following references.
Neider, Jackie. OpenGL Programming Guide: The Official
Guide to Learning OpenGL, Release 1. OpenGL Architecture Review Board,
Jackie Neider, Tom Davis, Mason Woo. Reading, Mass. : Addison-Wesley,
OpenGL Reference Manual: The Official Reference Document for
OpenGL, Release 1 / OpenGL Architecture Review Board. Reading, Mass.
: Addison-Wesley, c1993.
Data Representation and Access
These problems, addressed in the scientific visualization community,
are the same ones we face in timbre representation:
Nielson, G.M.; Brunet, P.; Gross, M.; Hagen, H.; and
others. Research Issues in Data Modeling for Scientific
Visualization. IEEE Computer Graphics and Applications, March 1994,
Matousek, J. Geometric Range Searching.
ACM Computing Surveys, Dec. 1994, vol.26, (no.4):421-61.
Tips and Pitfalls
Graphics programming offers its own special challenges, but a muture
literature on effective and efficient graphical programming techniques is
Wolff and Yaeger
Visualization of Natural
Phenomena. Robert S. Wolff, Larry Yaeger. Santa Clara, Calif.
: TELOS, c1993.
Graphics Gems, Andrew Glassner (ed.), Academic Press 1990,
Graphics Gems II, James Arvo (ed.), Academic Press 1991,
Graphics Gems III, David Kirk (ed.), Academic Press 1992,
ISBN 0-12-409670-0 (with IBM disk) or 0-12-409671-9 (with Mac disk)
Graphics Gems IV, Paul Heckbert (ed.), Academic Press 1994,
ISBN 0-12-336156-7 with MAC floppy, ISBN 0-12-336155-9 with PC floppy
The following papers were part of a session at ICMC95 devoted to
GUI's for computer music:
Robin Bargar, Bryan Holloway, Xavier Rodet, Chris Hartman Defining
Heinrich Taube, Tobias Kunze
Capella: A Graphical Interface for
Richard Polfreman, John Sapsford-Francis A Human Factors Approach
to Computer Music Systems User-Interface Design
Insook Choi, Robin Bargar, Camille Goudeseune A Manifold Interface
for a High Dimensional Control Space