INTRODUCTION

The massification of portable touch screen devices in the last decade has precipitated a surge in investment in research and development of surface sensing techniques. One common theme has emerged in this work: the co-location of multiple sensing techniques.  Many patents have been issued [17] and filed around combining capacitive and resistive sensing in particular [6] [7] [1] [16] [15].

Unfortunately, the mass manufacturing techniques used in modern, touch-screens are not readily accessible to laboratory experimenters and vernacular engineers (a.k.a. DIY, makers, bricoleurs). Also vendors are often reticent to share low-level access to their sensor streams preferring instead to wrap their innovations in high level, black-boxed Application Programmer Interfaces (API). The platform described in this paper enables many interesting explorations of the interaction potential of hybrid surface sensing without requiring exotic materials or challenging construction techniques. "Off-the-shelf" materials are used for the physical aspects and all the software required is freely available.

Structure and Contribution

After a description of the platform, some prototypes are introduced with an elaboration of some experiments associated with them. Apart from the contribution of the platform itself, this paper points to the value of using sensor fusion to leverage the temporal aspects of sensor measurements and broaden the sphere of use of tangible sensing systems.

Co-located Sensor Platform

Sensor Components

Sensor components are assembled from readily available conductive materials or piezoresistive sensors from Interlink.  In the first device we present, a round sensor (model number 30-81794) is switched between resistive and capacitive modes. For the second device, the interdigitated conductors of a square FSR (30-73528) were separated from the piezoresistive backing material and stacked onto a piezoresistive track pad (54-00031). These configurations correspond to the two major design patterns seen in the literature and prior art: 1) shared, switched components and 2) stacked layers with fixed-function capacitive and resistive sensing.

Microcontrollor Data Acquisition

The Freescale MK20DX256 microcontroller was chosen for this work because it includes dedicated capacitance measurement hardware, two 16-bit ADC's, and it provides a dozen I/O pins that can be dynamically switched among many functions: digital output, digital input, input with pull-up resistors, analog voltage input and capacitance sensing.  This microcontroller is available in a compact board (Teensy 3.1) for which a free, multi-platform software development environment is available (Arduino and Teensyduino).

Sensor Fusion and Signal Conditioning

The platform includes:

·  A Freescale 96MHz ARM 32-bit microcontroller that can  perform sensor fusion computations and run gesture analysis and estimation algorithms. For development purposes, it is mostly used for data acquisition, time stamping and to transmit the measurands to a more powerful host computer via USB or Ethernet.

·  The o.io component [3] of the odot system running on the host computer in the freely-available PD or commercially-supported Max/MSP/Jitter programming environments. This is used to display, process and map the sensor data. Mappings to sound facilitate experiments on the temporal aspects of gesture leveraging the particular capabilities of the human auditory system.

·  The expression language of the “odot” system [4] provides arithmetic operations on high resolution time stamps allowing, for example, the computation of velocity estimates of touch and motion parameter vectors.

Case Studies

Switched Modality

This example employs a single sensing system and rapidly switches between resistive and capacitive measurement modes.

Figure 1. Microcontroller, Fault-simulating Switches and FSR.

The round Force Sensing Resistor (FSR) at the bottom of Figure 1 is made from an interdigitated array of printed silver ink conductors separated from a semiconducting, piezoresistive substrate by an air gap. Displacement (from the press of a finger) brings the conductive and semiconducting substrates into contact.  These FSR’s thereby combine a switching action and a compression-sensing action. Other researchers and suppliers use this common design pattern. The separating function has been implemented in various devices by exploiting the mechanical properties of the plastic layers involved, using trapped air [10], sparse arrays of elastomeric dots [2] or porous fabric nets [11].

While the conductive elements are separated from the resistive material, they can be used as plates for capacitive sensing. One of the interdigitated conductors is grounded. The other is connected to two different input pins of the Teensy microcontroller board. One of those pins is switched between capacitive and voltage measurement modes; the other provides a "pull-up" resistor to the 3.3v supply rail that is only enabled during voltage measurements.  This provides the current source needed to estimate electrical resistance values from the measured input voltages.

Switching between sense modalities is determined by the actual values obtained in each mode: sudden large increases in measured capacitance signal the closures of the switching action of the sensor. This precipitates the transition to resistive measurements. Jumps to high voltage values, measured during resistive mode, signal the separations from the resistive material, prompting a return to capacitive measurements.

The simple prototype of Figure 1 was developed to learn whether capacitance sensing could serve as a degraded-performance backup of a resistive sensor system subject to wear and failures [8]. The dark slide switches allow simulation of broken leads to the sensor, shorts to ground and shorts between the leads. We have confirmed that a finger press on the sensor can be detected in all the fault scenarios except (as we expected) for the case of both leads shorted to ground or both leads broken. We were, in those cases, able to establish and report the nature of the fault.

We learned from this prototype that the proximity of the resistive surface to the conductors produces a significant shunt capacitance that shapes the e-field and reduces sensitivity of the capacitive sensing to a usable range of 4mm from finger to surface.

Stacking a special-purpose sensor on top of a resistive sensor allows for a reduction of capacitive coupling and increased sensitivity and dynamic range for proximity estimation. This is explored in the following section.

Stacked Concurrent Sensors

The sensor system of Figure 2 includes a piezoresistive track pad that senses position and pressure of a single finger touch or stylus point. It has a thick protective upper layer to resist wear from pens in “point of sale” applications. The interdigitated conductors glued on top were taken from a square FSR from Interlink. Its original resistive layer was peeled off and put aside leaving just the conductors that are used primarily as a proximity sensor, complementing the position and pressure sensing of the track pad underneath.

Figure 2. Stacked Track Pad and Interdigitated Surface Sensor

It is well known that the accuracy of position and pressure measurements in piezoresistive materials decreases at low contact pressures–a problem partially addressed using complex machine learning techniques that require calibration tasks [14].  Capacitive proximity measurements provide an alternative approach to ameliorate this situation by giving more accurate temporal information about the points of arrival and departure of touch gestures. This is especially useful in musical-instrument applications, for example, where moderate latencies of a few milliseconds can be introduced to make time to extrapolate pressure information to transition events and thereby absorb jitter in the overall gesture/sound synthesis signal path.

The key to this non-linear sensor fusion is the availability of accurate time stamps for transition events and acquired pressure events. These are provided in the “OSC for Arduino and Embedded Devices” library used to encode measured data and send it as SLIP-wrapped, OSC-encoded bundles to the host application.

As well as exploiting increased temporal fidelity for surface touch events, capacitive sense data can be used to estimate arrival and departure velocity profiles of objects interacting with the surface. This technique was used in the Mathews/Boie Radio Drum [9], a music controller that measures proximity by coupling radio waves from transmitting batons to a specially structured receiving antenna array. On the Radio Drum a reference plane is established based on signal strength and velocity is estimated by sampling the rate of change of proximity as a baton passes through this reference plane. This method is susceptible to changes in environment conditions. Using the surface interaction timing data available in our devices more precisely establishes a reference plane to which velocity profiles can be anchored.

If reliability is the primary concern it makes most sense to glue the capacitive sensor array (of Figure 2) conductive-ink side down. This would protect the ink from the corrosive effects of skin contact. However, for experimentation we found it more interesting to glue it conductor-side up allowing us to explore the use of resistive sensing through the skin of an impinging finger to estimate very light pressure values before the FSR tablet underneath is able to. This increases the temporal precision of touch and release time estimates and produces an affordance for gentle stroke gestures that don’t involve the track pad at all.

We found a usable proximity measurement range of 0-16mm with the stacked arrangement of Figure 2.

From Prototypes to Platform

The prototypes discussed so far are accessible because they are constructed from commercially available sensors that require so little preparation. Unfortunately, such screen-printed sensors are only available in a small number of different sizes and shapes. Custom sized, flexible printed circuits boards are now within reach for well-funded projects such as the multi-touch resistive arrays from Sensitronics, but it is the recently availability of conductive and resistive ink pens, ink jet cartridges, paper, textile threads and yarns that creates the opportunity for the accessible, affordable and scalable designs researchers need.  Our platform includes a selection of all of these materials because they synergistically work together. For example, silver-based conductive ink pens can be used to quickly create conductive patterns but conductors will break across folds in plastic or paper substrates. We solve this by using conductive thread at these hinge points attached to conductive ink pen traces using conductive epoxy.

Textile processes can be slower and more technically demanding than sketching with conductive ink, but textiles offer degrees of freedom of movement unavailable to the developable surfaces provided by sheet materials, e.g. sheer and stretch.

Figure 3 shows a prototype that combines custom-built e-textile position and pressure sensors with commercial sensors. The substrate surfaces in this case are the diaphragms of loudspeakers, an example of co-located sensing and actuation typical of haptic feedback scenarios. Fabric was particularly useful in the case of the large, round speaker because off-the-shelf devices can accommodate neither that shape nor size.

Figure 3. Haptic Speakers, fabric (top left and bottom) and commercial FSR (top right).

Future Work and Conclusion

An obvious avenue for future work is to address the challenge of translucency so that sensing can be co-located with displays. The requisite materials and processes are becoming more accessible for desktop use as printing techniques for organic conductors and semiconductors improve and the performance of the devices produced also grows. Meanwhile we use top projection as a reasonable approximation – one that is especially convenient as we explore scaling to large-scale surfaces such as floors and walls.

As suggested in recent publications on specific projects [12] [5] [13], the platform components presented here can be composed into a wide variety of interesting experimental prototypes for tangible interaction design. We look forward to enabling new projects as we further share our platform and techniques–especially as they move from our core focus of musical instrument controllers to other application spaces where collocated sensing is valuable.

Dedication

In memory of Professor David Wessel, his generous mentorship and enumerable contributions to tangible and embodied interaction.

Support

This work was supported in part by TerraSwarm, one of six centers of STARnet, a Semiconductor Research Corporation program sponsored by MARCO and DARPA.

REFERENCES

[1] Antila, M., Raemoe, T. and Karhiniemi, M., Touch sensor and method for operating a touch sensor, WO 2008155409 A2, 2008.

[2] Eventoff, F.N. and Christiansen, T., Digitizer pad, US4739299, 1988.

[3] Freed, A., DeFilippo, D., Gottfried, R., MacCallum, J., Lubow, J., Razo, D., Rostovtsev, I. and Wessel, D. o.io: a Unified Communications Framework for Music, Intermedia and Cloud Interaction ICMC, IMCA, Athens, Greece, 2014.

[4] Freed, A., MacCallum, J. and Schmeder, A. Composability for Musical Gesture Signal Processing using new OSC-based Object and Functional Programming Extensions to Max/MSP. NIME 2011 (June). 308-311, 2011.

[5] Freed, A., Uitti, F.-M., Mansfield, S. and MacCallum, J. "Old" is the new "New": a Fingerboard Case Study in Recrudescence as a NIME Development Strategy NIME, Daejeon Korea, 2013.

[6] Fricker, J.P., Alou, M. and Kasser, B., Resistive and capacitive touchpad, US20010013855 A1, 2001.

[7] Griffin, J.T., Touch-sensitive display with capacitive and resistive touch sensors and method of control, US8179376,2012.

[8] Kent, J. and Wilson, G.D., Dual sensor touchscreen utilizing projective-capacitive and force touch sensors, US6492979, 2002.

[9] Mathews, M.V. and Oberheim, T.E., Baton and X, Y, Z, position sensor, US6222522,2001.

[10] Murphy, J., Kapur, A. and Burgin, C., The helio: A study of membrane potentiometers and long force sensing resistors for musical interfaces. New Interfaces for Musical Expression, (2010), 459-462.

[11] Roh, J., Freed, A., Mann, Y. and Wessel, D., Robust and reliable fabric, piezoresistive multitouch sensing surfaces for musical controllers. in New Interfaces for Musical Expression, (2011), 11.

[12] Rowland, J. Flexible Audio Speakers for Composition and Art Practice. Leonardo Music Journal, 23. 33-36, 2013.

[13] Rowland, J. and Freed, A. Colocated Surface Sound Interaction SigCHI, Paris, France, 2013.

[14] Schmeder, A. and Freed, A. Support Vector Machine Learning for Gesture Signal Estimation with a Piezo-Resistive Fabric Touch Surface NIME, Sydney, 2010.

[15] Serban, B., Boyer, P. and Hoyer, D., Method of generating input data, WO 2009021837 A1, 2009.

[16] Serban, B., Boyer, P. and Schoos, A., Input device, WO 2008009687 A2, 2008.

[17] Wilson, G.D., Borgnis, V.E., Kent, J., Lewis, M., Loucks, D., Roney, J. and Patti, M.B., Touch confirming touchscreen utilizing plural touch sensors, US6504530, 2003.