Home / Autumn 2011 / Brave New World

Brave New World

By Orli Belman

At USC’s Institute for Creative Technologies, virtual reality is the new reality.

ICT's Light Stage 5 captures the shape, shine, color and texture of an actor's face, creating a realistic digital character. Photo: USC Institute for Creative TechnologiesSlideshow IconICT's Light Stage 5 captures the shape, shine, color and texture of an actor's face, creating a realistic digital character. Photo: USC Institute for Creative Technologies
ICT video games transform textbook teachings and real-world wisdom into interactive learning experiences. Research teams have collaborated with experts from the USC Rossier School of Education and the USC School of Cinematic Arts to develop applications that combine advanced artificial intelligence, evidence-based educational designs and engaging story-based lessons. In the negotiation trainer ELECT BiLAT, the student assumes the role of a U.S. Army officer who needs to conduct a series of meetings with local leaders to achieve mission objectives. Photo by Chuck Bowman To be successful, players must establish relationships with these virtual characters and be sensitive to their cultural conventions. Declining an offer to drink tea or skipping small talk to discuss business can set the negotiations back or end them completely. The game incorporates ICT research on advanced virtual humans who display believable behaviors and computational models of social interaction that emulate individual and group responses. And it features intelligent tutoring to provide students with real-time guidance and in-depth feedback. Serious games like these are used by thousands of American servicemen. West Point cadet Eric Zastoupil recently tested the game with ICT scientist H. Chad Lane. Photo by Chuck Bowman Paul Debevec, ICT’s associate director for graphics research and a research professor of computer science at the USC Viterbi School of Engineering, was inspired to become a visual-effects innovator after watching a DeLorean fly in the movie <em>Back to the Future</em>. Debevec is seen in Light Stage 6, one of a series of LED light-filled spheres he developed to capture and simulate how people and objects appear under real-world lighting conditions. Debevec’s technologies enable virtual worlds and characters to look convincing. They have been employed to create detailed digital faces that mirror their human counterparts down to skin pores and wrinkles. Work on these systems earned Debevec and his collaborators a 2010 Scientific and Engineering Academy Award. ICT’s Graphics Lab also developed a 3-D video teleconferencing system that beams hologram-like images capable of maintaining eye contact and conversations with people in other locations. Photo by Al Seib/Los Angeles Times ICT’s Mixed Reality Lab (MxR) studies and creates immersive systems that incorporate both real and virtual elements. Led by Mark Bolas, who also is an associate professor in the Interactive Media Division of the USC School of Cinematic Arts, MxR creates simulated environments in which participants can speak, move and gesture as readily as they would in the real world. Photo by Al Seib/Los Angeles Times A head-mounted projector generates individualized perspectives providing each wearer a different image on the same screen. Photo by Al Seib/Los Angeles Times The system allows a user to perceive whether a virtual character is establishing eye contact, gesturing or pointing a weapon at them. Photo by Al Seib/Los Angeles Times Bolas demonstrates “stretching space,” an effort led by researcher Evan Suma that uses imperceptible redirection techniques to transform a limited physical space into a boundless virtual world. For example, a gravel path inside the lab provides the base for a winding journey down roadways and through buildings. Photo by Al Seib/Los Angeles Times Meet Ada and Grace, two bright and bubbly educators who arrived at Boston’s Museum of Science in 2009. Science and technology are literally part of their being. That’s because they aren’t real people – but virtual ones. Designed to advance the public’s awareness of, and engagement in, computer science and emerging learning technologies, the virtual guides make a museum visit richer by answering questions, suggesting exhibitions and explaining the technology that makes them work. Named after two inspirational female computer science pioneers, Ada Lovelace and Grace Hopper, these digital docents are trailblazers in their own right. As part of an exhibition called <em>InterFaces</em>, they are among the first and most advanced virtual humans ever created to speak face-to-face with museum visitors. As both examples and explainers of technical scientific concepts, Ada and Grace represent a new medium for engaging the public in science. Photo: USC Institute for Creative Technologies Built on the same platform as Ada and Grace, ICT’s Sgt. Star is a life-sized virtual human who can talk about Army life and careers. Other ICT virtual characters help develop skills in leadership, negotiation and cultural awareness. Photo courtesy of USC Institute for Creative Technologies In another project called Gunslinger, virtual human technologies combine with Hollywood storytelling and set building to transport users to the Wild West. Players speak with virtual characters, who also speak to one another. By combining improvised conversation with carefully crafted narrative, Gunslinger pushes the frontiers of virtual human research and interactive storytelling. Photo by Steve Cohn Petty Officer Samuel Sarax is a combat veteran with emotional scars that won’t heal. A collaboration between the USC School of Social Work and ICT, this virtual patient is helping prepare future clinicians to address mental health needs of soldiers, veterans and their families. Student therapists can practice their skills in conducting interviews and making diagnoses before meeting real patients. Other ICT medical virtual-reality projects provide therapy for post-traumatic stress disorder and rehabilitation for stroke and traumatic brain injury. Photo courtesy of USC Institute for Creative Technologies ICT scientist Jacquelyn Ford Morie uses controlled breathing into a microphone to cause her avatar to jog in the virtual world Second Life. Morie’s research is part of a larger effort that explores using social networks to develop novel relaxation techniques that people can access from home or any remote location. Photo by Branimir Kvartuc A patient with spinal cord injuries helps physical therapist Belinda Lange test out Jewel Mine, a research prototype created with her team using Microsoft Kinect depth-sensing technology, along with software from OpenNI, that allows for individualized balance training and upper limb exercises. At ICT, Lange, who also is a research assistant professor in the USC Davis School of Gerontology, leads efforts to incorporate virtual reality and gaming technologies to improve physical rehabilitation, therapy and exercise. Photo by Branimir Kvartuc Computer science graduate student James Reinebold demonstrates a simulated Humvee application developed at ICT that can be used to determine an individual’s cognitive abilities and responses to threatening situations. By presenting such simulation challenges, performance can be measured in ways that may help assess and treat brain injury. Photo by Branimir Kvartuc It is all about “face time” at the ICT Multicomp Lab, headed by computer scientist Louis-Philippe Morency, who is also a research assistant professor at the USC Viterbi School of Engineering. Morency, pictured here with research programmer Giota Stratou, studies how to get computers to understand and anticipate non-verbal conversational cues such as eye gaze, smiling and nodding, which can be used to grasp the subtleties of conversation and make virtual character’s behavior more human and believable. Photo by Branimir Kvartuc Motion capture technician John Brennan makes some adjustments as Randy Nolta, a technical artist at ICT, readies himself for a motion capture photo shoot on the IR stage, a motion capture facility that is part of ICT’s Mixed Reality Lab. Photo courtesy of USC Institute for Creative Technologies

FOR APRIL FOOLS’ THIS YEAR, Google introduced Gmail Motion, an email feature that claimed to replace simple key strokes and mouse clicks with exaggerated body movements like pantomiming opening an envelope to read a message or licking a stamp to send one.

People got a laugh watching Google’s spoof video. But Evan Suma, a virtual-reality researcher at the USC Institute for Creative Technologies (ICT), got an idea.

Using skeleton tracking data from Microsoft Kinect, a system that allows users to interact through voice and body gestures without the need for a controller, Suma proved the prank was possible. His demonstration video, posted that same day, went viral, earning applause from tech blogs, The New York Times and Google itself.

Suma’s video was lighthearted. But it is a prime example of the serious work taking place at ICT, a U.S. Army-sponsored research and development lab in Playa Vista, Calif., where academics, artists, scientists and storytellers take computer-based toys and dream up futuristic tools that train soldiers, treat patients, teach students and more.

Think play with a purpose.


ICT is changing the science of animation. Courtesy of LabTV

Suma’s gesture-translating toolkit has led to the development of more effective physical rehabilitation systems. His USC colleagues have already transformed off-the-shelf headsets and joysticks into successful virtual-reality therapy for treating post-traumatic stress disorder. ICT- created games teach U.S. armed forces skills ranging from negotiating with people of other cultures to detecting improvised explosive devices. Additional applications build social skills for children with autism and educate parents about juvenile cancers.

The institute specializes in developing virtual humans, computer-animated characters that appear, speak, understand, express emotions and display body language in ever-more realistic ways. So similar are the virtual models to humans that ICT’s graphics guru Paul Debevec received an Academy Award in 2010 for his advances in creating believable digital doubles in movies like Avatar and Spider-Man 2.

In research settings, these human facsimiles advance social scientists’ understanding of how people think, feel and behave. Outside the lab, they live in laptops and large installations across the country, employed as digital docents explaining science to museum visitors, online coaches providing guidance to soldiers and families seeking mental health resources, and virtual role players replacing live actors for training mental health workers or teaching troops to better conduct field interviews.

“It is not enough to use technologies to create a cool experience,” says Randall W. Hill Jr., ICT’s executive director who oversees an interdisciplinary team of nearly 200 experts, including computer scientists, digital artists, script writers, game designers, physical therapists and psychologists. “We are creating a whole new way for people to engage with computers so that they can practice, learn and perform better.” 

And that is a gesture anyone can appreciate.