It’s the day before the first performance. And amid the bustle of one final rehearsal, soloist Marguerite Witvoet is warming up her voice. Her other voice.
Rather than trilling through the scales, she raises and lowers her gloved right hand in steps. As she does, each movement produces a vowel sound from the computer she wears on her back. She spends a few moments perfecting a soft “e.” Then she works on her consonants, flicking the fingers of her left hand against her thumb to create “p” and “t.”
Gloved and wired, Witvoet is preparing for the world’s first entirely self-contained gesture-realized digital operatic duet.
The mini-opera What Does a Body Know? at Open Ears, an alternative music festival in Kitchener, Ont., this past May, marked the official unveiling of the University of British Columbia’s Visual Voice project. The computer strapped to Witvoet’s back translates the movements of her two gloved hands into sounds. Shape the sounds together and intelligible words come out. Witvoet sings in her own voice. And her hands sing back. It’s both solo and duet.
On stage, the inaugural performance appears as more of a curiosity piece than polished art. The gloves’ voice can be rather murky and, befitting their age, temperamental.
The script calls for Witvoet to begin singing in her own voice with the gloves mimicking. Then the gloves lead and Witvoet follows. Finally, singer and gloves sing together in two-part harmony. It’s something of a robotic My Fair Lady. Unfortunately, the system crashed midway through the second act and Witvoet was forced to sing alone for the remainder.
Such is the risk when you’re on the cutting edge of music and technology. “Sometimes you break every string on your violin,” shrugs Witvoet afterwards. “Besides, I was too far into the performance to stop and reboot.” A saving grace: since it was the premiere for the entire concept, the audience had no way of knowing they’d actually missed the climax. In time, perhaps listeners will come to expect more from the gloves they hear.
The technology to convert hand movements into sound is not particularly new. Sidney Fels, director of UBC’s Media and Graphics Interdisciplinary Centre (MAGIC) and the technical brains behind Visual Voice (the project has more than $500,000 in federal funding), was one of the early pioneers. In the 1990s he created Glove-Talk, a system that allowed a wired glove wearer to read Dr. Seuss books in reasonably intelligible form. Other researchers made similar breakthroughs with different systems.
“We are a gesturing people,” Fels observes. “We use our hands or bodies to say lots of things, like pointing or shrugging. But with computers, the only gestures we currently use are keyboard strokes and a mouse. I don’t see it as too far a leap to say we should be able to do much more with our hands.”
Since that early establishing work, the path for glove-gesture technology has branched in two distinct directions.
Speaking gloves have obvious, if limited, applications for sign language and speech therapy. And they may find a place in hospitals as well. Recent trials have proven the efficacy of using gloves to call up and manipulate onscreen medical records during surgery, in the style of Tom Cruise’s futuristic thriller Minority Report. This marks an improvement over existing technology, such as touch screens, which present problems of contamination.
The other route has explored entertainment possibilities. Nintendo’s Wii console uses a simple form of hand-held gesture technology. Fels, however, is interested in something more expressive. At MAGIC he assembled an artistic team of a composer, librettist and costume designer, plus performers such as Witvoet, and set about creating a series of mini-operas performed via glove.
“I picked music [for this technology] because of its cultural significance,” Fels says. “Innovations such as the telephone or the electric guitar have the power to transform our world.” He bristles at suggestions that creating a new musical instrument is less important than a sterile filing system for medical records, and looks for the day when teens will tote his glove to the beach for campfire singalongs.
In fact it’s possible to foresee wired gloves in everyone’s future. Marcelo Wanderley, a professor of music technology at the Schulich School of Music at McGill University in Montreal, figures the intuitive nature of hand movements make gloves the perfect method for controlling everything from apartment heating systems to high-speed trains.
And while we wait for that day to arrive, using gloves to sing seems perfectly sensible and practical. “What’s nice about music is that we are dealing with extremely skilled people making precise, accurate movements on very sensitive instruments,” says Wanderley. “Music is an ideal test bed for this technology. I think everyone would agree it’s better to perfect it with musicians than to start off with surgeons learning long-distance brain surgery.”