The Augmented Hand Series

source: flong

The “Augmented Hand Series” (by Golan Levin, Chris Sugrue, and Kyle McDonald) is a real-time interactive software system that presents playful, dreamlike, and uncanny transformations of its visitors’ hands. It consists of a box into which the visitor inserts their hand, and a screen which displays their ‘reimagined’ hand—for example, with an extra finger, or with fingers that move autonomously. Critically, the project’s transformations operate within the logical space of the hand itself, which is to say: the artwork performs “hand-aware” visualizations that alter the deep structure of how the hand appears.

The Augmented Hand Series is a real-time interactive software system that presents playful, dreamlike, and uncanny transformations of its visitors’ hands. Originally conceived in 2004, the project was developed at the Frank-Ratchye STUDIO for Creative Inquiry in 2013-2014 through a commission from the Cinekid Festival of children’s media.

The installation consists of a box into which the visitor inserts their hand, and a touchscreen interface which displays their ‘reimagined’ hand, altered by various dynamic and structural transformations. In the version shown here, in its premiere at the 2014 Cinekid Festival in Amsterdam, the kiosk is accompanied by a large rear-projection. The touchscreen allows participants to select among the different transformations.

Critically, the project’s morphological transformations operate within the logical space of the hand itself. That is to say: the artwork performs “hand-aware” visualizations that alter the deep structure of how the hand appears—unlike, say, a funhouse mirror, which simply distorts the entire field of view.

The system uses the real-time posture of the participant’s real hand as the moment-to-moment baseline for its transformations. Participants are free to use either of their hands, and the system works properly even with visitors who wiggle their fingers, or who move and turn their hand—within certain limits. The software may produce unpredictable glitches if the visitor’s hand differs significantly from a flat palm-down or palm-up pose. Currently, the system’s behavior for postures like fists (in which many of the fingers are occluded) is undefined, as is multi-hand interaction.

Developing an interaction that can work with a very diverse public is always a challenge. The Augmented Hand Series accommodates a wide range of hand sizes, from children (of about 5 years old) through adults, as well as a very broad range of skin colors. The system also performs robustly with hands that have jewelry, nail polish, tattoos, birthmarks, wrinkles, arthritic swelling, and/or unusually long, short, wide or slender digits. Nevertheless, the system’s behavior for individuals with more or fewer than five fingers is presently undefined. There are many more kinds of hands to support, and doing so remains an area of active research for the project.

Visitor interactions at the Cinekid Festival (continued), 1’26” [Vimeo • YouTube]
About twenty different transformations or scenes have been developed. Some of these perform structural edits to the hand’s archetypal form; others endow the hand with new dimensions of plasticity; and others imbue the hand with a kind of autonomy, whose resulting behavior is a dynamic negotiation between visitor and algorithm. The videos here present live demonstrations of ten of these scenes:

Many more transformations are planned. The sketches below depict some of the current scenes, as well as some as-yet-unfinished scenes conceived during the project’s proposal phase.

The hand is a critical interface to the world, allowing the use of tools, the intimate sense of touch, and a vast range of communicative gestures. Yet we frequently take our hands for granted, thinking with them, or through them, but hardly ever about them. Our investigation takes a position of exploration and wonder. Can real-time alterations of the hand’s appearance bring about a new perception of the body as a plastic, variable, unstable medium? Can such an interaction instill feelings of defamiliarization, prompt a heightened awareness of our own bodies, or incite a reexamination of our physical identities? Can we provoke simple wonder about the fact that we have any control at all over such a complex structure as the hand?

We know that the interrelations of hand, mind and identity are far from simple. Persons with alien hand syndrome, for example, have hands which move independently of their conscious will, as if they belonged to another person. By contrast, amputees suffering from phantom limb syndrome continue to feel their missing hand as if it were still there; their discomfort is sometimes relieved with a mirror box, which uses the virtual image of their intact hand to trick the mind and retrain the brain. Within this framework, the Augmented Hand Series can be understood as an instrument for probing or muddling embodied cognition, using a ‘direct manipulation’ interface and the suspension of disbelief to further problematize the mind-body problem. We see evidence of our instrument’s powers in the actions of young visitors who, uncertain whether to believe their eyes, peek into the box to double-check what is (not) really happening to their hand.

The Augmented Hand Series is positioned to prompt, like Dr. Seuss (“I wish I had eleven, too!”) an empathetic acceptance of difference, and a recognition that there are many ways to be. Or could have been. In his essay “Eight Little Piggies”, the eminent biologist Stephen Jay Gould considers why we are shaped the way we are, and concludes—based on fascinating fossil evidence from some of the first land animals, some 400 million years ago—that there is nothing special or inevitable, nor even necessarily evolutionarily optimal about having five fingers on each hand. As Walt Disney discovered nearly a century ago, five fingers may not even be ideal for expressive gestural communication. From this perspective, the Augmented Hand Series is a work of participatory, transhuman biology-fiction that allows for the first-person exploration of these concepts. Enraptured visitors
A wish to elicit and address wonder—our own, and others’—has been the deep common bond between Kyle, Chris and myself as we developed this project. Ultimately, as collaborators, our underlying personal motivations for developing the project are each quite different, but Kyle’s, which arise from his experiences as a lucid dreamer, are perhaps the most poetic:

Artist Biographies
Golan Levin explores the intersection of abstract communication and interactivity. Blending equal measures of the whimsical, the provocative, and the sublime in a wide variety of media, Levin applies creative twists to digital technologies that highlight our relationship with machines, expand the vocabulary of human action, and awaken participants to their own potential as creative actors. At Carnegie Mellon University, he is Associate Professor of Electronic Art and serves as Director of the Frank-Ratchye STUDIO for Creative Inquiry, a laboratory dedicated to supporting atypical, anti-disciplinary and inter-institutional research projects across the arts, science, technology and culture.

Chris Sugrue is an artist and engineer who develops interactive installations, audio-visual performances and experimental interfaces. Her works experiment with technology in playful and curious ways and investigate topics such as artificial life, gestural performance and optical illusions. She has exhibited internationally in such festivals and galleries as Ars Electronica, Sónar Festival, Pixel Gallery, Medialab-Prado, Matadero Madrid, and La Noche En Blanco Madrid. She teaches new media arts at The Parsons School of Design in Paris.

Kyle McDonald works with sounds and codes, exploring translation, contextualization, and similarity. With a background in philosophy and computer science, he strives to integrate intricate processes and structures with accessible, playful realizations that often have a do-it-yourself, open-source aesthetic. He enjoys creatively subverting networked communication and computation, exploring glitch and embedded biases, and extending these concepts to reversal of everything from personal identity to work habits. Kyle is a member of F.A.T. Lab, community manager for openFrameworks, and an adjunct professor at the NYU ITP.