Introduction

Visual displays on deformable surfaces are compelling because they are easy to adapt to for the generation of users who have become heavily trained in ocularcentric, planar screen touch interactions since their invention in the mid 1980’s. Visual display is not the only form of output possible on deformable surfaces and for many applications will not turn out to be the best.

The two well-known problems with colocating deformation and image are:

Occlusion and visual distortions right at the points of interaction that are the most likely to be interesting to the user.

Difficulty mapping obvious sensed parameters (depth or pressure) to visual form with commensurate resolution.

We are exploring other sense modalities for deformable displays that avoid these problems and offer interesting affordances for display designers.

Sound is one of those modalities that is so obvious and pervasive that it may be easily overlooked. The state of the art in terms of human expressivity and interactivity in deformable displays is the Indian tabla drum. Good tabla players have sufficient control to be able play melodies on the surface using palm pressure to control pitch and their fingers to sound the drum with a wide variety of dynamics and timbral change. We are still very far from being able to sense or actuate a surface with sufficient precision in time and space to support the control musicians have over such surface interactions. The last musical instrument to be developed in the west with comparable control was the Clavichord. The keys of the clavichord couple the fingers to what become the movable bridges of multiple strings. Good clavichord players have a control of dynamic range that equal that of the piano in addition to pitch and timbral control – both impossible for the piano. The piano displaced use of the clavichord by being much louder and more suitable for the concert hall than the parlor.

HCI designers usually want more control over the mapping between input and output than a special purpose musical instrument provides. The insertion of digital computation and electrical transduction involves too many compromises to discuss fully here. We will instead focus on a few key challenges using the author’s work and examples from various collaborators.

Figure 1: the Tablo

The author’s “Tablo” is a controller designed to capture palm and finger gestures that a table player might use. Sound output is emitted from below the surface. An array of programmable lights provides visual feedback (analogous to the frets of a guitar) shining through the translucent conductive stretchable fabric. This fabric drapes over electrically resistive strips, lowering their resistance which then becomes a measure of displacement. Piezoresistive fabric segments around the annular base of the instrument are used to estimate pressure and provide a reference point to measure dynamic changes in displacement as an estimate of finger velocity.

Anna Flagg and Hannah-Perner Wilson have both adapted the two main piezoresistive pressure surface design patterns [2] to deformable applications using stretchable conductive and piezoresistive fabrics.

The basic construction is illustrated for Flagg’s “Cuddlebot” in figure 2.

Figure 2: Cuddlebot construction

The Cuddlebot is an affective robot that displays its “emotional” responses as sounds and vibration according to the nature of sensed interactions with its body surface.

 

 

 

 

 

 

Figure 3 shows Perner-Wilson’s robot skin. It uses a tubular adaptation of a basic fabric pressure multitouch design of Figure 4 [3]. Display in this case is retroaction in the form of motion of segments of the robot arm.

Figure 3: Robot Skin

Figure 4:Piezorestive Pressure Multitouch

Figure 5 shows the author’s most recent experiment towards capturing the responsiveness of hand drums [1]. From the display point of view it is interesting because the fabric sensing materials are built directly on the moving surface of loudspeaker drivers. This colocates sensing, audio and vibrotactile stimulation and leverages recent developments in control theory to make the coupled systems stable and also new sound transducers that have a flat and robust diaphragm.

Figure 5: Colocation

Discussion and Conclusion

Two major challenges persist in the engineering of the display systems presented: temporal and spatial resolution. The legacy from office automation applications of the 1980’s of low frame rates (30-60Hz) and low input data rates (<100Hz) persists and pervades in the implementation of desktop and mobile operating systems. New applications of these displays in gaming, music and other situations where tight multimodal integration is important require controlled latencies and sample rates better than 1kHz. We have shown that the sensing-data-rate problem can be finessed by translating the data into digital audio [4]. The spatial resolution issue is hard to solve currently without more reliable ways to connect the stretchable materials to the rigid materials supporting the electronics. One of the more promising approaches to this problem is to build the electronics with stretchable or bendable materials.

References

[1] Freed, A. Integration of Touch Pressure and Position Sensing with Speaker Diaphragms (Colocating Loudspeakers and Touch Interaction) Audio Engineering Society, AES, San Francisco, 2012.

[2] Freed, A. Novel and Forgotten Current-steering Techniques for Resistive Multitouch, Duotouch, and Polytouch Position Sensing with Pressure NIME, 2009.

[3] Schmeder, A. and Freed, A. Support Vector Machine Learning for Gesture Signal Estimation with a Piezo-Resistive Fabric Touch Surface NIME, Sydney, 2010.

[4] Wessel, D., Avizienis, R., Freed, A. and Wright, M. A Force Sensitive Multi-touch Array Supporting Multiple 2-D Musical Control Structures New Interfaces for Musical Expression, New York, 2007.

Sound, Vibration, and Retroaction in Deformable Displays, Freed, Adrian , SIGCHI Workshop: Organic experiences: (re)shaping interactions with deformable displays, 04/2013, Paris/France, (2013)

After a quick review of recent rapid developments in intrinsic fiber and textile sensing and actuation, I demonstrate pressure, stretch, position, acceleration, and multitouch sensors built with paper and textile structures. Their performance will be shown informally by integrating them into functional musical instruments. I close with a vision of how a design theory and practice may be developed for these materials, illustrated with new, folded, and pleated piezoresistive structures that have just begun to be explored.Ideas on Building Interactivity in Textiles and Related Fibers, Freed, Adrian , International Symposium on New Frontiers in Fiber Materials Science, 08/2011, Charlston, South Carolina, (2011)

Robust and Reliable Fabric, Piezoresistive Multitouch Sensing Surfaces for Musical Controllers, Roh, Jung-Sim, Freed Adrian, Mann Yotam, and Wessel David , NIME 2011, (2011)

Description:

Rectangular grids are dominant forms in electronics and textiles. Triaxial grids have not been explored in e-textile work so this breadboard is offered to begin these explorations. Triaxial grids sample the plane with higher density and the availability of whug connections as well as warp and woof simplifies circuits by providing a natural power/ground/signal triple.

Making:

The basic 3x3x3 configuration is made by sticking conductive copper tape on strips of basket-weaving reed. This is intended to evoke the thousands of year old traditions of basket weaving and invite 3-d explorations.

Card stock is easier to procure and stick the tape to. The idea is to use a substrate that won’t burn when you solder to the tape. Substitute fabric ribbons if you are going to sew to the breadboard instead.

I used pins to lock down two of parallel rows of 3 strips and then slid in the final row, guiding them over and under as required. The example has charlieplexed LED’s soldered on but this is just to start you thinking about how you might use such a dense array of available conductors.

Consider replacing a central strip with one with copper tape on both sides. Work out which conductors are then connected. Now consider sandwiching piezoresistive fabric between intersecting conductors. Can you use this to read an array of pressures sensors?

References/Inspirations:
core memory beading
triaxial textiles
basket weaving
multitouch

Materials: Cane, copper tape

Techniques: Weaving

The provided swatches will just be the breadboard. Illustrated are some LED’s controlled by Charlieplexing.

Circuit Diagram:
http://www.pcbheaven.com/wikipages/Charlieplexing/

Dates: July 24 + July 25, 2010 from 10am to 5pm FULL

Description   

During this hands-on workshop we will survey Arduino platforms, libraries, shields and programming techniques for a broad range of musical and sound applications. We will learn how to synthesize useful wave shapes including square, sine, triangle and pulses, how to manage polyphony and timing and how to playback and record sampled sounds.

// XYZ textile pad
// (procedural abstraction)
void analogPullup(int pin, boolean activate)
{  
  pinMode(14+pin, INPUT); // magic to active pullups 14 is the numbering of the analog pins
  digitalWrite(14+pin, activate ?   

This bracelet has an Arduino-powered ring of white LED's with a 3-axis accelerometer tilt sensor.

How many interactions/games can you think of with this platform?

There are 3 in the video:

Hand in the air: flashes (because at a party you want to signal that you want someone to talk to?).

Horizontal hand: always illuminates the top LED's whatever rotation your arm has ("smart flashlight")

Spins of the wrist: a blob spins around in the same direction and slows to a stop.

For a commercially produced inertial-sensing band keep an eye out on getymyo

This logs class syllabi and activities

Dates: Saturday June 26 and Sunday June 27, 2010 from 10am to 5pm

Description

Thanks to Angela Sheehan from Soft Circuit Saturdays for filming this in the challenging, busy, noisy environment of Maker Faire.