The massification of portable touch screen devices in the
last decade has precipitated a surge in investment in research and development
of surface sensing techniques. One common theme has emerged in this work: the co-location
of multiple sensing techniques. Many patents have been issued  and
filed around combining capacitive and resistive sensing in particular   
Unfortunately, the mass manufacturing techniques used in
modern, touch-screens are not readily accessible to laboratory experimenters
and vernacular engineers (a.k.a. DIY, makers, bricoleurs). Also vendors are often
reticent to share low-level access to their sensor streams preferring instead to
wrap their innovations in high level, black-boxed Application Programmer Interfaces
(API). The platform described in this paper enables many interesting
explorations of the interaction potential of hybrid surface sensing without
requiring exotic materials or challenging construction techniques.
"Off-the-shelf" materials are used for the physical aspects and all
the software required is freely available.
Structure and Contribution
After a description of the platform, some prototypes are
introduced with an elaboration of some experiments associated with them. Apart
from the contribution of the platform itself, this paper points to the value of
using sensor fusion to leverage the temporal aspects of sensor measurements and
broaden the sphere of use of tangible sensing systems.
Co-located Sensor Platform
Sensor components are assembled from readily available
conductive materials or piezoresistive sensors from Interlink. In the
first device we present, a round sensor (model number 30-81794) is switched
between resistive and capacitive modes. For the second device, the interdigitated
conductors of a square FSR (30-73528) were separated from the piezoresistive backing
material and stacked onto a piezoresistive track pad (54-00031). These
configurations correspond to the two major design patterns seen in the
literature and prior art: 1) shared, switched components and 2) stacked layers
with fixed-function capacitive and resistive sensing.
Microcontrollor Data Acquisition
The Freescale MK20DX256 microcontroller was chosen for this work
because it includes dedicated capacitance measurement hardware, two 16-bit
ADC's, and it provides a dozen I/O pins that can be dynamically switched among
many functions: digital output, digital input, input with pull-up resistors,
analog voltage input and capacitance sensing. This microcontroller is
available in a compact board (Teensy 3.1) for which a free, multi-platform software
development environment is available (Arduino and Teensyduino).
Sensor Fusion and Signal Conditioning
The platform includes:
A Freescale 96MHz ARM 32-bit microcontroller that can
perform sensor fusion computations and run gesture analysis and estimation
algorithms. For development purposes, it is mostly used for data acquisition,
time stamping and to transmit the measurands to a more powerful host computer via
USB or Ethernet.
The o.io component  of the odot system running on the host
computer in the freely-available PD or commercially-supported Max/MSP/Jitter programming
environments. This is used to display, process and map the sensor data.
Mappings to sound facilitate experiments on the temporal aspects of gesture
leveraging the particular capabilities of the human auditory system.
The expression language of the “odot” system  provides
arithmetic operations on high resolution time stamps allowing, for example, the
computation of velocity estimates of touch and motion parameter vectors.
This example employs a single sensing system and rapidly
switches between resistive and capacitive measurement modes.
Figure 1. Microcontroller, Fault-simulating Switches and FSR.
The round Force Sensing Resistor (FSR) at the bottom of
Figure 1 is made from an interdigitated array of printed silver ink conductors
separated from a semiconducting, piezoresistive substrate by an air gap.
Displacement (from the press of a finger) brings the conductive and
semiconducting substrates into contact. These FSR’s thereby combine a switching
action and a compression-sensing action. Other researchers and suppliers use
this common design pattern. The separating function has been implemented in
various devices by exploiting the mechanical properties of the plastic layers
involved, using trapped air , sparse arrays of elastomeric dots  or
porous fabric nets .
While the conductive elements are separated from the resistive
material, they can be used as plates for capacitive sensing. One of the
interdigitated conductors is grounded. The other is connected to two different input
pins of the Teensy microcontroller board. One of those pins is switched between
capacitive and voltage measurement modes; the other provides a
"pull-up" resistor to the 3.3v supply rail that is only enabled
during voltage measurements. This provides the current source needed to
estimate electrical resistance values from the measured input voltages.
Switching between sense modalities is determined by the
actual values obtained in each mode: sudden large increases in measured
capacitance signal the closures of the switching action of the sensor. This precipitates
the transition to resistive measurements. Jumps to high voltage values,
measured during resistive mode, signal the separations from the resistive
material, prompting a return to capacitive measurements.
The simple prototype of Figure 1 was developed to learn
whether capacitance sensing could serve as a degraded-performance backup of a
resistive sensor system subject to wear and failures . The dark slide
switches allow simulation of broken leads to the sensor, shorts to ground and
shorts between the leads. We have confirmed that a finger press on the sensor
can be detected in all the fault scenarios except (as we expected) for the case
of both leads shorted to ground or both leads broken. We were, in those cases,
able to establish and report the nature of the fault.
We learned from this prototype that the proximity of the
resistive surface to the conductors produces a significant shunt capacitance
that shapes the e-field and reduces sensitivity of the capacitive sensing to a
usable range of 4mm from finger to surface.
Stacking a special-purpose sensor on top of a resistive
sensor allows for a reduction of capacitive coupling and increased sensitivity
and dynamic range for proximity estimation. This is explored in the following
Stacked Concurrent Sensors
The sensor system of Figure 2 includes a piezoresistive
track pad that senses position and pressure of a single finger touch or stylus
point. It has a thick protective upper layer to resist wear from pens in “point
of sale” applications. The interdigitated conductors glued on top were taken
from a square FSR from Interlink. Its original resistive layer was peeled off
and put aside leaving just the conductors that are used primarily as a
proximity sensor, complementing the position and pressure sensing of the track
Figure 2. Stacked Track Pad and Interdigitated Surface Sensor
It is well known that the accuracy of position and pressure
measurements in piezoresistive materials decreases at low contact pressures–a
problem partially addressed using complex machine learning techniques that require
calibration tasks . Capacitive proximity measurements provide an
alternative approach to ameliorate this situation by giving more accurate
temporal information about the points of arrival and departure of touch
gestures. This is especially useful in musical-instrument applications, for
example, where moderate latencies of a few milliseconds can be introduced to make
time to extrapolate pressure information to transition events and thereby
absorb jitter in the overall gesture/sound synthesis signal path.
The key to this non-linear sensor fusion is the availability
of accurate time stamps for transition events and acquired pressure events.
These are provided in the “OSC for Arduino and Embedded Devices” library used
to encode measured data and send it as SLIP-wrapped, OSC-encoded bundles to the
As well as exploiting increased temporal fidelity for
surface touch events, capacitive sense data can be used to estimate arrival and
departure velocity profiles of objects interacting with the surface. This
technique was used in the Mathews/Boie Radio Drum , a music controller that
measures proximity by coupling radio waves from transmitting batons to a
specially structured receiving antenna array. On the Radio Drum a reference
plane is established based on signal strength and velocity is estimated by
sampling the rate of change of proximity as a baton passes through this
reference plane. This method is susceptible to changes in environment
conditions. Using the surface interaction timing data available in our devices
more precisely establishes a reference plane to which velocity profiles can be
If reliability is the primary concern it makes most sense to
glue the capacitive sensor array (of Figure 2) conductive-ink side down. This
would protect the ink from the corrosive effects of skin contact. However, for
experimentation we found it more interesting to glue it conductor-side up allowing
us to explore the use of resistive sensing through the skin of an impinging
finger to estimate very light pressure values before the FSR tablet underneath
is able to. This increases the temporal precision of touch and release time
estimates and produces an affordance for gentle stroke gestures that don’t
involve the track pad at all.
We found a usable proximity measurement range of 0-16mm with
the stacked arrangement of Figure 2.
From Prototypes to Platform
The prototypes discussed so far are accessible because they
are constructed from commercially available sensors that require so little
preparation. Unfortunately, such screen-printed sensors are only available in a
small number of different sizes and shapes. Custom sized, flexible printed
circuits boards are now within reach for well-funded projects such as the
multi-touch resistive arrays from Sensitronics, but it is the recently
availability of conductive and resistive ink pens, ink jet cartridges, paper,
textile threads and yarns that creates the opportunity for the accessible,
affordable and scalable designs researchers need. Our platform includes a
selection of all of these materials because they synergistically work together.
For example, silver-based conductive ink pens can be used to quickly create
conductive patterns but conductors will break across folds in plastic or paper
substrates. We solve this by using conductive thread at these hinge points
attached to conductive ink pen traces using conductive epoxy.
Textile processes can be slower and more technically
demanding than sketching with conductive ink, but textiles offer degrees of
freedom of movement unavailable to the developable surfaces provided by sheet
materials, e.g. sheer and stretch.
Figure 3 shows a prototype that combines custom-built
e-textile position and pressure sensors with commercial sensors. The substrate
surfaces in this case are the diaphragms of loudspeakers, an example of
co-located sensing and actuation typical of haptic feedback scenarios. Fabric
was particularly useful in the case of the large, round speaker because
off-the-shelf devices can accommodate neither that shape nor size.
Figure 3. Haptic Speakers, fabric
(top left and bottom) and commercial FSR (top right).
Future Work and Conclusion
An obvious avenue for future work is to address the
challenge of translucency so that sensing can be co-located with displays. The
requisite materials and processes are becoming more accessible for desktop use
as printing techniques for organic conductors and semiconductors improve and
the performance of the devices produced also grows. Meanwhile we use top
projection as a reasonable approximation – one that is especially convenient as
we explore scaling to large-scale surfaces such as floors and walls.
As suggested in recent publications on specific projects 
 , the platform components presented here can be composed into a wide
variety of interesting experimental prototypes for tangible interaction design.
We look forward to enabling new projects as we further share our platform and
techniques–especially as they move from our core focus of musical instrument
controllers to other application spaces where collocated sensing is valuable.
In memory of Professor David Wessel, his generous mentorship
and enumerable contributions to tangible and embodied interaction.
This work was supported in part by TerraSwarm, one of six
centers of STARnet, a Semiconductor Research Corporation program sponsored by
MARCO and DARPA.
 Antila, M., Raemoe, T. and Karhiniemi, M., Touch sensor
and method for operating a touch sensor, WO 2008155409 A2, 2008.
 Eventoff, F.N. and Christiansen, T., Digitizer pad, US4739299,
 Freed, A., DeFilippo, D., Gottfried, R., MacCallum, J.,
Lubow, J., Razo, D., Rostovtsev, I. and Wessel, D. o.io: a Unified
Communications Framework for Music, Intermedia and Cloud Interaction ICMC,
IMCA, Athens, Greece, 2014.
 Freed, A., MacCallum, J. and Schmeder, A. Composability
for Musical Gesture Signal Processing using new OSC-based Object and Functional
Programming Extensions to Max/MSP. NIME 2011 (June). 308-311, 2011.
 Freed, A., Uitti, F.-M., Mansfield, S. and MacCallum, J.
"Old" is the new "New": a Fingerboard Case Study in
Recrudescence as a NIME Development Strategy NIME, Daejeon Korea, 2013.
 Fricker, J.P., Alou, M. and Kasser, B., Resistive and
capacitive touchpad, US20010013855 A1, 2001.
 Griffin, J.T., Touch-sensitive display with capacitive and
resistive touch sensors and method of control, US8179376,2012.
 Kent, J. and Wilson, G.D., Dual sensor touchscreen
utilizing projective-capacitive and force touch sensors, US6492979, 2002.
 Mathews, M.V. and Oberheim, T.E., Baton and X, Y, Z,
position sensor, US6222522,2001.
 Murphy, J., Kapur, A. and Burgin, C., The helio: A
study of membrane potentiometers and long force sensing resistors for musical
interfaces. New Interfaces for Musical Expression, (2010), 459-462.
 Roh, J., Freed, A., Mann, Y. and Wessel, D., Robust and
reliable fabric, piezoresistive multitouch sensing surfaces for musical
controllers. in New Interfaces for Musical Expression, (2011), 11.
 Rowland, J. Flexible Audio Speakers for Composition and
Art Practice. Leonardo Music Journal, 23. 33-36, 2013.
 Rowland, J. and Freed, A. Colocated Surface Sound
Interaction SigCHI, Paris, France, 2013.
 Schmeder, A. and Freed, A. Support Vector Machine
Learning for Gesture Signal Estimation with a Piezo-Resistive Fabric Touch
Surface NIME, Sydney, 2010.
 Serban, B., Boyer, P. and Hoyer, D., Method of
generating input data, WO 2009021837 A1, 2009.
 Serban, B., Boyer, P. and Schoos, A., Input device, WO
2008009687 A2, 2008.
 Wilson, G.D.,
Borgnis, V.E., Kent, J., Lewis, M., Loucks, D., Roney, J. and Patti, M.B.,
Touch confirming touchscreen utilizing plural touch sensors, US6504530, 2003.